Ci: improve install time iteration 2

[belgattitude]

What does it do?

Improve ci time by tackling install, part of

• Ci improve ci time iteration 3 #16694
• Ci: improve install time iteration 2 #16638
• ci(perf): use alpine docker based images for postgresql + no fsync (wip) #16709
• Remove @vscode/sqlite3 from example and CI matrix #16693
• CI: speed up CI time by improving yarn install and caches (iteration 1 / >30%) #16581

See #16581 and the vision: #16581 (comment)

Context

In the first iteration:

• The safest way. I just fixed a cache and provided a custom install action. Measured two possible alternatives (only yarn cache vs yarn cache + install_state + node_modules). Based on that yarn cache only was choosen (very close to each others)

This second iteration will be more open to discussion and re-assessing the previous choices based on this new situation

Why ?

Cause in the current state it brings modification to the yarn.lock and have an impact on local disk usage.

But there's two ways I can do it

• Basically the safest way: I change globally the compressionLevel
• Advanced: I override those settings only on ci. (that's something I actually do when building docker images (less than 3 minutes): https://github.com/belgattitude/artist/blob/512bc55735a32daf4a3fe11c337d04cfa8a11c12/apps/strapi/Dockerfile#L12-L16). So I can have both low size on local and faster install on ci. But for now it affects the lock. For the curious https://github.com/belgattitude/artist/blob/512bc55735a32daf4a3fe11c337d04cfa8a11c12/.github/workflows/release-strapi-docker.yml

Note that while for the docker example above is fine, it's not yet on CI with some hacks cause it produce a new yarn.lock. I'll coordinate with upstream yarn. Or ... I bring a solution but it will look a bit hacky.

I'll try to document in the files for you to get a clear idea.

But I'm waiting first for iteration 1 to be merged (will be clearer to follow).

And as in the previous iteration, I'll need

• To measure / benchmark the options I have, I'll ask for workflow approval :)

Next steps

• If we choose to change the yarn.lock we'll have to provide a small doc/warning to all open PR and/or maintainers. Cause they'll have to refresh the lock (easy: yarn install then push the lock - it won't change versions, just checksums)
• When we align on the solution, I'll bring a 3rd iteration that will try to extract 5-10% more speedup. But it's related to how we decide to merge this one

There's even more to do... but after those I'll have to request/coordinate with the upstream yarn codebase and if it works the cache of install-state/node_modules cache will make sense to be enabled. But that's a longer road ;) I like 🌳

Why is it needed?

🌳

[belgattitude]

Some might wonder, why it's suddently needed. Cause afterall it's overridden in the install action.

Context:

• By removing compression... local archives won't be zipped. That's what we want on CI... on local it's okay (more disk space, let's say it's fine). But if enableGlobalCache is true, downloaded archives won't be pruned... and over time the local installs will gros fast. See my answer in Performance opportunity for yarn 3 cache actions/setup-node#325 (comment).

PS: enableGlobalCache might be default in upcoming yarn version, so we align for the future as well)

[belgattitude]

the zip isn't compressed. More work for action/cache restore / network / decompress (zstd detects when it can compress or not so... now the *.zip files will be compressable by action/cache -> slower but more compressed too)

For the curious:

My previous benchmarks +/- based on https://github.com/belgattitude/nextjs-monorepo-example

Speed

CI Scenario	Install	CI fetch cache	CI persist cache	Setup
yarn4 mixed-compression	±42s	±3s	(±9s)	0s
yarn4 no compression	±34s	±4s	(±6s)	0s
pnpm8	±19s	±8s	(±29s)	2s

Cache budget

CI Scenario	Fetch cache	Persist cache	Size
yarn4 mixed-compression	±5s	±6s	220MB
yarn4 no compression	±8s	±9s	180MB
pnpm7	±14s	±16s	273MB

More metrics in https://github.com/belgattitude/compare-package-managers

[belgattitude]

This one is not totally required. We'll see the ultimate choice made by upcoming yarn 4. But I'd say... keep it there explicitly (in yarn 3 it's the default).

I'm open to remove this too (anyway the install action explcitly overrides it to 'hardlinks-local')

[gu-stav]

@belgattitude I've merged your other PR - could you update this one with main? 🙏🏼

[belgattitude]

@gu-stav done, but I'll have to double check how I've impacted the lock during the merge. So consider this more a wip till I'm sure nothing have been silently updated. BTW github action seems down... so let's move slowly

[belgattitude]

@gu-stav I keep this note as a reminder (about something unrelated to this P/R)

one thing that I realized also is that the deps vscode/sqlite3 is consuming time and does not seem to complie in CI anyway. I'll do a cost/benefit P/R later on. Not sure of the value in the getting-started example

yarn why @vscode/sqlite3
└─ getstarted@workspace:examples/getstarted
   └─ @vscode/sqlite3@npm:5.1.2 (via npm:5.1.2)

See the full run https://github.com/strapi/strapi/actions/runs/4946607251/jobs/8845313813

Impact is important and does not work very well in ci's (the build will fail)

Myself I would remove this dep in the example. Value is limited if you're installing better-sqlite3 in the example

https://github.com/strapi/strapi/blob/main/packages/core/database/lib/connection.js#L38

  // NOTE: this tries to find the best sqlite module possible to use
  // while keeping retro compatibility
  return (
    trySqlitePackage('better-sqlite3') ||
    trySqlitePackage('@vscode/sqlite3') ||
    trySqlitePackage('sqlite3')
  );

Or keep the examples outide of the main workspace (even better). Many options possibles.

The suggested P/R is there #16693

[belgattitude]

Would be awesome to have something labelling the status to run the full test suite ❤️

[belgattitude]

Next iterations will be

• Remove @vscode/sqlite3 from example and CI matrix #16693
• Ci improve ci time iteration 3 #16694

But I'm confident that we can reach a full test on < 45 minutes (vs +/-1h30 before iteration 1) soon. And more ideas to come.

[belgattitude]

Got the measurements now.

setting the compressionLevel to 0 only saves 5-20s per install. With CE there's around 10 steps. (the saving is 1min - 3minutes only per push/run)

So I'm not sure. To keep it short,

Trade-offs:

• For dev local install zip files aren't compressed: 1gb on disk vs 250Mb
• For CI only +/-2 minutes saving per run (tests/ce).
• For CI as the zip appears now to be compressible: saves 100Mb on cache budget (loose speed on restore as action/cache/zstd strats to compress the zips. so the overhead is moved to decompression).
• If using docker at some point, compressionLevel:0 keep more optims possibles.

So it all depends... Wdyt ?

For me this PR can be merged safely... and in my experience compressionLevel:0 is a good thing, cause more deps you add, more gain you get (till this lands nodejs/node#45651 but it's a long way).

PS: next iterations: #16694 and/or #16721 are part of the serie too.

[gu-stav]

@belgattitude it seems possible to use env variables in the yarnrc file. Could we set the compression level only in a CI environment?

yarnpkg/berry#1341

[belgattitude]

Not really... as the lock contains checksums. Whenever you change compression they are recalculated.

It's possible to hack of course... That's actually how I speed up docker images build:

RUN YARN_ENABLE_GLOBAL_CACHE="false" \
    YARN_CACHE_FOLDER=/var/cache/yarn-artist-strapi \
    YARN_COMPRESSION_LEVEL=0 \
    YARN_CHECKSUM_BEHAVIOR=ignore \
    yarn install --immutable --inline-builds

but I don't recommend cause right now it changes the yarn.lock as well. so you'll need to rollback the change after install. for docker it's fine (divide install by 2)

But that's something I'll propose to change or clarify in upstream yarn. Will PR next week and see how it's received (cause I'm not sure)

[gu-stav]

@belgattitude thanks. I'll do some local testing tomorrow, but I think the size difference wouldn't be a huge problem.

@joshuaellis @Convly @innerdvations what do you think?

[innerdvations]

Compression level I'm not so sure about. It seems like a fairly minimal improvement compared to the tradeoffs. Are the next iterations dependent on it or can it be left alone for now?

[belgattitude]

@innerdvations seems okay to me. I'll move it to the later iteration and when I can measure the impact based on final round about installs.

So I propose this order

• chore: align swc usage for test:unit and fix swc/helpers hazards #16731 (changes the lock) - I'll also provide the same for test:front if convincing.
• Ci improve ci time iteration 3 #16694 (changes the lock - some risk so we'll have to be extra careful and is unblocked by the previous one. I feel it will prevent weird issues on the long term, always good)

Then I'll measure the new baseline and adapt

That's the easiest way for me

Thanks