-
-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Parallelize @astrojs/image transforms #4626
Parallelize @astrojs/image transforms #4626
Conversation
🦋 Changeset detectedLatest commit: 8156382 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
@@ -40,6 +40,7 @@ | |||
"test": "mocha --exit --timeout 20000 test" | |||
}, | |||
"dependencies": { | |||
"@altano/tiny-async-pool": "^1.0.2", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In case anyone is hesitant to take on @altano/tiny-async-pool
, a new dependency:
- The used export is super tiny: 315B gzipped
- The package is a fork of async-pool by Rafael Xavier de Souza. I left it MIT licensed and barely modified the code other than to modernize it (typescript, esm export, vitest, etc). The original package gets ~900K weekly downloads so it's mature code and should be fairly bug free.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great work! Want to leave it to @tony-sull to take a look at, but it lgtm.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Very nice!
Just ran it locally against the astro.build site and was impressed to see that sharp
didn't blow up Node's memory usage when processing 8 images in parallel 🚀
Changes
This change parallelizes
@astrojs/image
transforms to get a big performance boost.@astrojs/image
currently processes images serially, one after the other. If it processes 10 images and takes 1 second to process each, the total time will be 10 seconds.This change makes it process images in batches. The batch size is the # of CPUs on the machine, e.g. 8 on my dev machine.
Testing
I manually put the newly built code in dist/build/ssg.js into my current project (I can't figure out how to get
pnpm
andnpm
to let me link across projects). I then did a before and after build.I then edited
astro.config.mts
and enableddebug
logging:BEFORE - 92 second build
AFTER - 39 second build
NOTE: Yes I know a 92s->39s build time is still abysmal.
@astrojs/image
is very slow at processing animated gifs. I will investigate that separately.Docs
No docs needed as this is a performance improvement