build: port copy-trilium.sh cleanup functionality to cross-platform TS #1436

pano9000 · 2025-03-16T01:43:33Z

WIP that partially addresses #1174

-> moves functionality from copy-trilium.sh into cross-platform copy-dist.ts, so that we can get rid of one extra build script AND have other build processes also benefit from the copy-trilium.sh features (mainly "node_modules" cleanup -> which results in smaller file sizes for releases) -> this is currently only utilized for the "bare metal" Server build, but will now be available for all other builds as well

~~on a very first test, I got a reduction of 200+ MB of the node_modules folder this way~~

Edit 2025-03-27: Updated PR Description

This PR essentially ports the existing node_modules cleanup functionality of the copy-trilium.sh script to a cross-platform standalone TS script -> cleanupNodeModules.ts.
Its purpose is to "prune" the node_modules folder by removing unnecessary files not needed for production (e.g. like tests, demos, examples, TS files, etc.), which cuts down on the node_modules folder size -> I currently get a reduction of ~270 MB of the node_modules folder by running the script.

Porting the script to a standalone TS script allows us to run the cleanup on all platforms and for all build targets.
Previously the cleanup was only used with the "bare metal" Server build, the other targets (Docker and electron-forge) were not benefiting from it at all.

Due to the build target' peculiarities, we need to be able to run the cleanup script at different points in their respective build stages.

Since it usually runs during a stage where we might not have any devDeps available (e.g. in electron-forge's "afterPrune" hook), I've opted to just run it straight with node and its "--experimental-strip-types" flag – since we require node >= 22, that is no issue.

Two things that are still open/missing here:

excalidraw fonts copying – however that will currently not cause issues anyways, as excalidraw is still part of the dependencies and not the devDependencies — once the work on the branch mentioned in the link is ready, we can move it to devDep
the copy-trilium.sh script was also pruning ".map" files from our "libraries" subfolder -> I am not sure if we want that or not though (it only affects a single ckeditor.map file of 8MB) — I would leave this one out for this PR though

pano9000 · 2025-03-25T08:11:55Z

will run some tests this evening with the CI, locally this now builds nicely.

only thing that will be broken after this:
copying of the excalidraw fonts:
"node_modules/@excalidraw/excalidraw/dist/prod/fonts/",

however for this I am already working on a branch, that moves this step to the webpack build phase, where it actually fits better

pano9000 · 2025-03-25T08:23:59Z

CI currently fails with:
✖ Running package command [FAILED: Command failed: node --experimental-strip-types /tmp/electron-packager/tmp-JQI6jy/resources/app/bin/cleanupNodeModules.ts '/tmp/electron-packager/tmp-JQI6jy/resources/app'
node: bad option: --experimental-strip-types
]

need to check, which node version we actually run in the CI - with node v22 --experimental-strip-types definitely works

eliandoran · 2025-03-25T08:25:20Z

CI currently fails with: ✖ Running package command [FAILED: Command failed: node --experimental-strip-types /tmp/electron-packager/tmp-JQI6jy/resources/app/bin/cleanupNodeModules.ts '/tmp/electron-packager/tmp-JQI6jy/resources/app' node: bad option: --experimental-strip-types ]

need to check, which node version we actually run in the CI - with node v22 --experimental-strip-types definitely works

I believe we run 20, but feel free to bump it up in the YAML workflows. Look for setup-node.

pano9000 · 2025-03-25T08:32:24Z

yeah, makes sense

we currently v22 inside Docker builds anyways and only
Github CI and Server Build as using v20.

both are LTS versions and I don't see any reason, why we shouldn't go with v22 for all builds.

I'll bump the versions in the YAML files and the server build script this evening, when I am back from work

this is useless at the moment, as all build processes are running "npm ci --omit=dev" anyways, i.e. they will just install everything remaining again

…rom copy-trilium in preparation to get rid of the file altogether - rest of the functionality will be merged into copy-dist.ts

helps avoiding downloading the packages a 2nd time during our prune stage

since we our own more "sophisticated" pruning during copy-dist, we need to set this to false, as otherwise build will fail

this is not needed anymore since commit 6b9d8f0

adapted from copy-trilium.sh

same as before, but some of the modules use singular/plural, so just check for both

deleting of ts and map files from node_modules folder, adapted from copy-trilium.sh.

* image-q/demo -> doesn't exist anymore (and even if it did - previous cleanup step, would've removed this anyways) * @excalidraw/excalidraw/dist/* -> updated to point to the dev folder * boxicons -> only fonts and css folders are used, so remove the other ones * jimp -> updated paths and removed non-existent path fixing here for historical reasons, next step is to move these over to copy-dist.ts

this is necessary, since for Docker and electron-forge, we need to run this as an extra step after copy-dist for electron-forge: after it is done with its own "pruning", as we otherwise would need to also take care of certain electron related pruning for Docker: as a last step in the build stage

all of its functionality has been "absorbed" by the cross-platform copy-dist and cleanupNodeModules scripts, that can be used for all of our builds now

extra note on mermaid: that should be packed by webpack now, so we just need to move it to devDeps and and we should be good

…sDelete

this is only needed for executing package scripts -> which we don't do in production fixes #1499

also added a logging function for debugging purposes

pano9000 · 2025-03-26T08:59:17Z

Node version in CI has been bumped to v22 in the meantime, now just need to add the running of the cleanup script in the Docker build process and we should be ready here :-)

* moved the running of npm ci from copy-dist to cleanupNodeModules * added flag to disable it (necessary for electron-forge)

previously forgot to add this here as well -> had it only in the Dockerfile.alpine

pano9000 · 2025-03-27T08:23:22Z

Seems like the Windows runner is behaving strange :-(
https://github.com/TriliumNext/Notes/actions/runs/14101381731/job/39498152263#step:6:1671

Error: Command failed: node --experimental-strip-types C:\Windows\SERVIC~1\NETWOR~1\AppData\Local\Temp\electron-packager\tmp-a02ejC\resources\app\bin\cleanupNodeModules.ts 'C:\Windows\SERVIC~1\NETWOR~1\AppData\Local\Temp\electron-packager\tmp-a02ejC\resources\app' --skip-prune-dev-deps
(node:12428) ExperimentalWarning: Type Stripping is an experimental feature and might change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
Supplied path ''C:\Windows\SERVIC~1\NETWOR~1\AppData\Local\Temp\electron-packager\tmp-a02ejC\resources\app'' does not exist. Aborting.

Not sure if node is maybe getting tripped up by the "~1" thing in the windows path? -> will need to investigate.

Edit: Likely caused by the single apostrophes around the path — need to check if that is potentially some "cmd" shell issue?

MacOS build also fails, but that is due to #1531 – edit: but it does correctly build, when the "flakiness" does not happen :-)
https://github.com/TriliumNext/Notes/actions/runs/14102223495/job/39500707877

attempt to fix failing Windows build in CI, which uses cmd shell, which apparently treats single apostrophes differently

pano9000 · 2025-03-27T08:57:40Z

started another CI run here after an attempt to fix Windows Runner build:
https://github.com/TriliumNext/Notes/actions/runs/14102223495/job/39500707864

still running, so let's see when and if it finishes :-)

Edit: Build Works :-)
Comparison:

pano9000 force-pushed the build_copy-dist-trilium-merge branch 2 times, most recently from 666f9c3 to 812806b Compare March 25, 2025 08:10

pano9000 added 22 commits March 26, 2025 08:12

build(copy-dist): get rid of manual node_module copying

17d5fdb

this is useless at the moment, as all build processes are running "npm ci --omit=dev" anyways, i.e. they will just install everything remaining again

build(build-server): move "build:prepare-dist" call to build-server f…

1640000

…rom copy-trilium in preparation to get rid of the file altogether - rest of the functionality will be merged into copy-dist.ts

build(copy-dist): move "pruning" to copy-dist from copy-trilium.sh

8977492

build(copy-dist): copy over existing node_modules

6fb270e

helps avoiding downloading the packages a 2nd time during our prune stage

build(electron-forge): set prune to false

8feb201

since we our own more "sophisticated" pruning during copy-dist, we need to set this to false, as otherwise build will fail

build(copy-trilium): remove now unnecessary package.json patching

bee7793

this is not needed anymore since commit 6b9d8f0

build(copy-dist): add initial cleanupNodeModules functionality

6260ea1

adapted from copy-trilium.sh

build(copy-dist): add further folders to list of filterableDirs

a964317

same as before, but some of the modules use singular/plural, so just check for both

build(copy-dist): execute filterableDirs cleanup in one chain

8275f3c

build(copy-dist): add further cleanupNodeModules functionality

6749d80

deleting of ts and map files from node_modules folder, adapted from copy-trilium.sh.

chore(scripts): remove now unneeded electron-forge:prepare

3e3344b

build(server): use cleanupNodeModules script

f79b925

build(copy-trilium): delete now unused script

57ee619

all of its functionality has been "absorbed" by the cross-platform copy-dist and cleanupNodeModules scripts, that can be used for all of our builds now

build(dockerignore): add cleanupNodeModules as exception

83da24b

chore(cleanupNodeModules): rename nodeDir to nodeModulesContent

b457fa2

chore(cleanupNodeModules): remove commented out paths

6fe23f9

extra note on mermaid: that should be packed by webpack now, so we just need to move it to devDeps and and we should be good

build(cleanupNodeModules): use path.join and basePath for extraFolder…

1150f78

…sDelete

build(cleanupNodeModules): add some minimalistic logging

1528703

build(cleanupNodeModules): delete .bin folder

51f2e23

this is only needed for executing package scripts -> which we don't do in production fixes #1499

build(cleanupNodeModules): move removal of elements to its own function

0b42803

also added a logging function for debugging purposes

pano9000 force-pushed the build_copy-dist-trilium-merge branch from 4deb496 to 0b42803 Compare March 26, 2025 08:49

build: make running of npm ci in build scripts configurable

75431ca

* moved the running of npm ci from copy-dist to cleanupNodeModules * added flag to disable it (necessary for electron-forge)

pano9000 changed the title ~~build: merge copy-trilium.sh functionality into copy-dist.ts~~ build: port copy-trilium.sh cleanup functionality to cross-platform TS Mar 27, 2025

pano9000 added 4 commits March 27, 2025 09:04

chore(docker): remove TODO

e9fa37c

build(docker): use absolute path to stay consistent

d5bc984

build(docker): add missing cleanupNodeModules removal to Dockerfile

fa03586

previously forgot to add this here as well -> had it only in the Dockerfile.alpine

chore(docker): move exception below TODO comment

6218ae6

pano9000 marked this pull request as ready for review March 27, 2025 08:12

pano9000 and others added 2 commits March 27, 2025 09:35

build(electron-forge): use double quotes for the buildPath in afterPrune

9fe3746

attempt to fix failing Windows build in CI, which uses cmd shell, which apparently treats single apostrophes differently

Merge branch 'develop' into build_copy-dist-trilium-merge

5a2c5d8

eliandoran approved these changes Mar 27, 2025

View reviewed changes

eliandoran merged commit 10f044a into develop Mar 27, 2025
10 of 11 checks passed

eliandoran deleted the build_copy-dist-trilium-merge branch March 27, 2025 17:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

build: port copy-trilium.sh cleanup functionality to cross-platform TS #1436

build: port copy-trilium.sh cleanup functionality to cross-platform TS #1436

pano9000 commented Mar 16, 2025 •

edited

Loading

pano9000 commented Mar 25, 2025

pano9000 commented Mar 25, 2025

eliandoran commented Mar 25, 2025

pano9000 commented Mar 25, 2025

pano9000 commented Mar 26, 2025

pano9000 commented Mar 27, 2025 •

edited

Loading

pano9000 commented Mar 27, 2025 •

edited

Loading

build: port copy-trilium.sh cleanup functionality to cross-platform TS #1436

build: port copy-trilium.sh cleanup functionality to cross-platform TS #1436

Conversation

pano9000 commented Mar 16, 2025 • edited Loading

pano9000 commented Mar 25, 2025

pano9000 commented Mar 25, 2025

eliandoran commented Mar 25, 2025

pano9000 commented Mar 25, 2025

pano9000 commented Mar 26, 2025

pano9000 commented Mar 27, 2025 • edited Loading

pano9000 commented Mar 27, 2025 • edited Loading

pano9000 commented Mar 16, 2025 •

edited

Loading

pano9000 commented Mar 27, 2025 •

edited

Loading

pano9000 commented Mar 27, 2025 •

edited

Loading