Bundle WordPress translation files for faster site creation#2558
Bundle WordPress translation files for faster site creation#2558ivan-ottinger merged 10 commits intotrunkfrom
Conversation
c1003a4 to
9cbbf55
Compare
…ation Instead of downloading translation files from WordPress.org at runtime via the setSiteLanguage blueprint step, bundle core language packs for all 19 supported non-English locales at build time. During site creation for the latest WP version, files are copied locally and WPLANG is set via defineWpConfigConsts + setSiteOptions. Falls back to setSiteLanguage for non-latest versions or when bundled packs are unavailable. Also makes setupWPServerFiles resilient so individual step failures don't prevent subsequent steps from running.
WordPress 6.5+ uses .l10n.php files, making .po (source) and .mo (legacy binary) files unnecessary at runtime. Removing them reduces the language packs from 80MB to 27MB. Also fixes ts-node module resolution by inlining the locale list in the download script instead of importing from common/lib/locale.ts.
Download and bundle translation files for akismet, twentytwentyfive, twentytwentyfour, and twentytwentythree alongside core translations. Plugin/theme translations are stored in languages/plugins/ and languages/themes/ subdirectories and copied to the site during creation. Adds ~2MB to the bundle.
Replace slashes in the label used for temp zip filenames to avoid creating subdirectories in the temp folder.
Move the list of WordPress locale codes to common/lib/wp-locales.ts so it can be imported by both the download script (ts-node) and app code (Vite) without module resolution issues. Remove the unused studioToWpLocaleMap from common/lib/locale.ts.
Read the wp-content/plugins and wp-content/themes directories from the extracted WordPress installation instead of hardcoding the list. Handles single-file plugins like hello.php via a slug mapping.
66ae173 to
dbe9753
Compare
| } | ||
| } | ||
|
|
||
| async function removePoAndMoFiles( dirPath: string ): Promise< void > { |
There was a problem hiding this comment.
I decided to drop the .po and .mo translation files here - in favor of performant .l10n.php (more context: https://make.wordpress.org/core/2024/02/27/i18n-improvements-6-5-performant-translations/).
This approach takes the additional bundle size down to about 29 MB - compared to around 80 MB if we included .po and .mo files.
One drawback I see in this approach is that hello-dolly plugin has the new .l10n.php translation files for French and Korean only so far. This means that this plugin won't be translated in other locales when a new site is created.
Curious to hear your point of view.
There was a problem hiding this comment.
Overall I think this tradeoff would be worth it to save on the bundle size.
I also saw that Akismet also only has 19 translations in the .l10n.php format out of the 75 shown on WP.org.
Do you know if there's a plan to migrate these to the more performant version in the future maybe?
There was a problem hiding this comment.
I also saw that Akismet also only has 19 translations in the .l10n.php format out of the 75 shown on WP.org.
We only keep the files of locales we need for Studio (19 locales) and drop the rest. 🙂
Do you know if there's a plan to migrate these to the more performant version in the future maybe?
I haven't found any info on that, but for what we need, only hello-dolly seems to be affected. Moving forward into the future, I expect all new plugins and themes will rely on the new translation format.
There was a problem hiding this comment.
We only keep the files of locales we need for Studio (19 locales) and drop the rest. 🙂
Aaah, that makes sense, thanks! 😄
I haven't found any info on that, but for what we need, only hello-dolly seems to be affected. Moving forward into the future, I expect all new plugins and themes will rely on the new translation format.
Yeah, I think so too.
| ); | ||
|
|
||
| // Plugin translations | ||
| const pluginSlugs = getBundledSlugs( path.join( wpContentPath, 'plugins' ) ); |
There was a problem hiding this comment.
Initially I was just listing specific plugins and themes to bundle their translations, but then decided to automatically get the list based on what's included in the current WordPress.
This approach will ensure we won't need to update the list of plugins and themes when they change in the WordPress (e.g. when a new core default theme is added).
There was a problem hiding this comment.
I think this is a great approach to avoid the maintenance burden. 👍
gcsecsey
left a comment
There was a problem hiding this comment.
Thanks for working on this @ivan-ottinger! 🙌 Functionally, this is working well!
I confirm the language packs are now downloaded when starting the app:
The wp-files/latest/languages directory and its subdirectories are populated:
When creating a site using the newest WordPress version, translations are present in the site's wp-content/languages directory, and omitted for English sites:
| German site | English site |
|---|---|
![]() |
![]() |
| } | ||
| } | ||
|
|
||
| async function removePoAndMoFiles( dirPath: string ): Promise< void > { |
There was a problem hiding this comment.
Overall I think this tradeoff would be worth it to save on the bundle size.
I also saw that Akismet also only has 19 translations in the .l10n.php format out of the 75 shown on WP.org.
Do you know if there's a plan to migrate these to the more performant version in the future maybe?
| ); | ||
|
|
||
| // Plugin translations | ||
| const pluginSlugs = getBundledSlugs( path.join( wpContentPath, 'plugins' ) ); |
There was a problem hiding this comment.
I think this is a great approach to avoid the maintenance burden. 👍
sejas
left a comment
There was a problem hiding this comment.
This is a super cool feature! and the best of all it helps when creating a new site and the computer is offline.
Creating a WordPress site in Spanish in offline mode:
| Before | After |
|---|---|
![]() |
![]() |
It increases the time of installing dependencies by one minute, but I think we can improve it as a follow-up by downloading the wp-files in parallel. Happy to own that task after you merge this PR.
BEFORE:
> time npm install
npm install 11.26s user 3.56s system 15% cpu 1:36.16 total
AFTER:
> time npm install
npm install 6.05s user 1.85s system 50% cpu 15.705 total
Also, I would recommend skipping the languages for older themes like 2024 and 2023 themes.
|
Thank you for your reviews and testing, Gergely and Antonio!
Great idea! Looks like you already have the code locally so feel free to create the PR for it. 🙂 Happy to review it.
At the moment it downloads translations for all bundled themes - but true is that only one is active. I agree it is a good idea to save a bit more storage and time. We could indeed skip 2023 and 2024. That should be a quick change. If you like, feel free to include it in your wp-files parallel copy PR Antonio. Otherwise I will add a new PR for it. 🙂 |
* Bundle WordPress core language packs to speed up non-English site creation Instead of downloading translation files from WordPress.org at runtime via the setSiteLanguage blueprint step, bundle core language packs for all 19 supported non-English locales at build time. During site creation for the latest WP version, files are copied locally and WPLANG is set via defineWpConfigConsts + setSiteOptions. Falls back to setSiteLanguage for non-latest versions or when bundled packs are unavailable. Also makes setupWPServerFiles resilient so individual step failures don't prevent subsequent steps from running. * Drop .po and .mo files from bundled language packs WordPress 6.5+ uses .l10n.php files, making .po (source) and .mo (legacy binary) files unnecessary at runtime. Removing them reduces the language packs from 80MB to 27MB. Also fixes ts-node module resolution by inlining the locale list in the download script instead of importing from common/lib/locale.ts. * Bundle translations for default plugins and themes Download and bundle translation files for akismet, twentytwentyfive, twentytwentyfour, and twentytwentythree alongside core translations. Plugin/theme translations are stored in languages/plugins/ and languages/themes/ subdirectories and copied to the site during creation. Adds ~2MB to the bundle. * Fix temp file path for plugin/theme language pack downloads Replace slashes in the label used for temp zip filenames to avoid creating subdirectories in the temp folder. * Extract WordPress locale list to shared module Move the list of WordPress locale codes to common/lib/wp-locales.ts so it can be imported by both the download script (ts-node) and app code (Vite) without module resolution issues. Remove the unused studioToWpLocaleMap from common/lib/locale.ts. * Add hello-dolly to bundled plugin translations * Remove as const from WP_LOCALES to fix ts-node compilation * Align language pack download messages with existing status format * Auto-detect bundled plugins and themes for translation downloads Read the wp-content/plugins and wp-content/themes directories from the extracted WordPress installation instead of hardcoding the list. Handles single-file plugins like hello.php via a slug mapping. * Remove redundant comment in site creation language setup
* Bundle WordPress core language packs to speed up non-English site creation Instead of downloading translation files from WordPress.org at runtime via the setSiteLanguage blueprint step, bundle core language packs for all 19 supported non-English locales at build time. During site creation for the latest WP version, files are copied locally and WPLANG is set via defineWpConfigConsts + setSiteOptions. Falls back to setSiteLanguage for non-latest versions or when bundled packs are unavailable. Also makes setupWPServerFiles resilient so individual step failures don't prevent subsequent steps from running. * Drop .po and .mo files from bundled language packs WordPress 6.5+ uses .l10n.php files, making .po (source) and .mo (legacy binary) files unnecessary at runtime. Removing them reduces the language packs from 80MB to 27MB. Also fixes ts-node module resolution by inlining the locale list in the download script instead of importing from common/lib/locale.ts. * Bundle translations for default plugins and themes Download and bundle translation files for akismet, twentytwentyfive, twentytwentyfour, and twentytwentythree alongside core translations. Plugin/theme translations are stored in languages/plugins/ and languages/themes/ subdirectories and copied to the site during creation. Adds ~2MB to the bundle. * Fix temp file path for plugin/theme language pack downloads Replace slashes in the label used for temp zip filenames to avoid creating subdirectories in the temp folder. * Extract WordPress locale list to shared module Move the list of WordPress locale codes to common/lib/wp-locales.ts so it can be imported by both the download script (ts-node) and app code (Vite) without module resolution issues. Remove the unused studioToWpLocaleMap from common/lib/locale.ts. * Add hello-dolly to bundled plugin translations * Remove as const from WP_LOCALES to fix ts-node compilation * Align language pack download messages with existing status format * Auto-detect bundled plugins and themes for translation downloads Read the wp-content/plugins and wp-content/themes directories from the extracted WordPress installation instead of hardcoding the list. Handles single-file plugins like hello.php via a slug mapping. * Remove redundant comment in site creation language setup
* Bundle WordPress core language packs to speed up non-English site creation Instead of downloading translation files from WordPress.org at runtime via the setSiteLanguage blueprint step, bundle core language packs for all 19 supported non-English locales at build time. During site creation for the latest WP version, files are copied locally and WPLANG is set via defineWpConfigConsts + setSiteOptions. Falls back to setSiteLanguage for non-latest versions or when bundled packs are unavailable. Also makes setupWPServerFiles resilient so individual step failures don't prevent subsequent steps from running. * Drop .po and .mo files from bundled language packs WordPress 6.5+ uses .l10n.php files, making .po (source) and .mo (legacy binary) files unnecessary at runtime. Removing them reduces the language packs from 80MB to 27MB. Also fixes ts-node module resolution by inlining the locale list in the download script instead of importing from common/lib/locale.ts. * Bundle translations for default plugins and themes Download and bundle translation files for akismet, twentytwentyfive, twentytwentyfour, and twentytwentythree alongside core translations. Plugin/theme translations are stored in languages/plugins/ and languages/themes/ subdirectories and copied to the site during creation. Adds ~2MB to the bundle. * Fix temp file path for plugin/theme language pack downloads Replace slashes in the label used for temp zip filenames to avoid creating subdirectories in the temp folder. * Extract WordPress locale list to shared module Move the list of WordPress locale codes to common/lib/wp-locales.ts so it can be imported by both the download script (ts-node) and app code (Vite) without module resolution issues. Remove the unused studioToWpLocaleMap from common/lib/locale.ts. * Add hello-dolly to bundled plugin translations * Remove as const from WP_LOCALES to fix ts-node compilation * Align language pack download messages with existing status format * Auto-detect bundled plugins and themes for translation downloads Read the wp-content/plugins and wp-content/themes directories from the extracted WordPress installation instead of hardcoding the list. Handles single-file plugins like hello.php via a slug mapping. * Remove redundant comment in site creation language setup
* Bundle WordPress core language packs to speed up non-English site creation Instead of downloading translation files from WordPress.org at runtime via the setSiteLanguage blueprint step, bundle core language packs for all 19 supported non-English locales at build time. During site creation for the latest WP version, files are copied locally and WPLANG is set via defineWpConfigConsts + setSiteOptions. Falls back to setSiteLanguage for non-latest versions or when bundled packs are unavailable. Also makes setupWPServerFiles resilient so individual step failures don't prevent subsequent steps from running. * Drop .po and .mo files from bundled language packs WordPress 6.5+ uses .l10n.php files, making .po (source) and .mo (legacy binary) files unnecessary at runtime. Removing them reduces the language packs from 80MB to 27MB. Also fixes ts-node module resolution by inlining the locale list in the download script instead of importing from common/lib/locale.ts. * Bundle translations for default plugins and themes Download and bundle translation files for akismet, twentytwentyfive, twentytwentyfour, and twentytwentythree alongside core translations. Plugin/theme translations are stored in languages/plugins/ and languages/themes/ subdirectories and copied to the site during creation. Adds ~2MB to the bundle. * Fix temp file path for plugin/theme language pack downloads Replace slashes in the label used for temp zip filenames to avoid creating subdirectories in the temp folder. * Extract WordPress locale list to shared module Move the list of WordPress locale codes to common/lib/wp-locales.ts so it can be imported by both the download script (ts-node) and app code (Vite) without module resolution issues. Remove the unused studioToWpLocaleMap from common/lib/locale.ts. * Add hello-dolly to bundled plugin translations * Remove as const from WP_LOCALES to fix ts-node compilation * Align language pack download messages with existing status format * Auto-detect bundled plugins and themes for translation downloads Read the wp-content/plugins and wp-content/themes directories from the extracted WordPress installation instead of hardcoding the list. Handles single-file plugins like hello.php via a slug mapping. * Remove redundant comment in site creation language setup
…ndling (#2575) * Bundle WordPress translation files for faster site creation (#2558) * Bundle WordPress core language packs to speed up non-English site creation Instead of downloading translation files from WordPress.org at runtime via the setSiteLanguage blueprint step, bundle core language packs for all 19 supported non-English locales at build time. During site creation for the latest WP version, files are copied locally and WPLANG is set via defineWpConfigConsts + setSiteOptions. Falls back to setSiteLanguage for non-latest versions or when bundled packs are unavailable. Also makes setupWPServerFiles resilient so individual step failures don't prevent subsequent steps from running. * Drop .po and .mo files from bundled language packs WordPress 6.5+ uses .l10n.php files, making .po (source) and .mo (legacy binary) files unnecessary at runtime. Removing them reduces the language packs from 80MB to 27MB. Also fixes ts-node module resolution by inlining the locale list in the download script instead of importing from common/lib/locale.ts. * Bundle translations for default plugins and themes Download and bundle translation files for akismet, twentytwentyfive, twentytwentyfour, and twentytwentythree alongside core translations. Plugin/theme translations are stored in languages/plugins/ and languages/themes/ subdirectories and copied to the site during creation. Adds ~2MB to the bundle. * Fix temp file path for plugin/theme language pack downloads Replace slashes in the label used for temp zip filenames to avoid creating subdirectories in the temp folder. * Extract WordPress locale list to shared module Move the list of WordPress locale codes to common/lib/wp-locales.ts so it can be imported by both the download script (ts-node) and app code (Vite) without module resolution issues. Remove the unused studioToWpLocaleMap from common/lib/locale.ts. * Add hello-dolly to bundled plugin translations * Remove as const from WP_LOCALES to fix ts-node compilation * Align language pack download messages with existing status format * Auto-detect bundled plugins and themes for translation downloads Read the wp-content/plugins and wp-content/themes directories from the extracted WordPress installation instead of hardcoding the list. Handles single-file plugins like hello.php via a slug mapping. * Remove redundant comment in site creation language setup * Move language pack downloads out of postinstall into npm run make * Skip translation downloads for older bundled themes * Add retry with exponential backoff to language pack downloads Prevents flaky CI builds by retrying failed requests up to 3 times with exponential backoff (1s, 2s, 4s). Failed downloads now fail the build instead of silently skipping, ensuring all translations are included. * Add Hungarian locale to language pack downloads * Refactor fetchWithRetry to use recursion instead of for loop * Remove redundant 'Files extracted' log message * Rename download:language-packs script to download-language-packs * Rename usedBundledPacks to isUsingBundledLanguagePacks for clarity




Related issues
Closes STU-1250
Related to STU-980
Proposed Changes
The PR proposes to bundle WordPress core, plugin and theme translation files. This saves about 3 - 4 seconds in localized site creation time while adding only about 29 MB to the app bundle and additional time when running
npm install.Feel free to check the comments with the background in the related task (STU-1250).
scripts/download-wp-server-files.tsfor all 19 supported non-English locales.server-files/language-packs/directory..l10n.phpand.jsonfiles directly into the site'swp-content/languages/directory and setWPLANGvia blueprint steps (defineWpConfigConsts+setSiteOptions) — no network round-trip needed.setSiteLanguageblueprint step for non-latest WP versions or when bundled packs are unavailable..l10n.phpand.jsonfiles (WordPress 6.5+ format), skipping.poand.moto reduce size (~29MB vs ~82MB).setupWPServerFilesresilient so individual step failures don't prevent subsequent steps from running.Testing Instructions
npm installand verify language packs are downloaded without errors. You can find them in the project root:wp-files/latest/languages,wp-files/latest/languages/pluginsandwp-files/latest/languages/themes.npm startand set your system locale to a non-English language (e.g. Swedish).wp-content.npm test -- cli/commands/site/tests/create.test.tsand verify all tests pass.Pre-merge Checklist