feat(scripts): convert PO→JSON translation pipeline po2json to cross-platform Python version#39724
feat(scripts): convert PO→JSON translation pipeline po2json to cross-platform Python version#39724FrancescoCastaldi wants to merge 19 commits into
po2json to cross-platform Python version#39724Conversation
Adds scripts/compile_po.py, a standalone utility to compile and convert Apache Superset translation files. The script: - Checks Python (Babel) and Node/npm/npx dependencies automatically - Compiles .po files with pybabel - Syncs compiled PO files to superset-frontend/src/translations - Installs po2json and prettier locally (avoids repeated npx downloads) - Converts all .po files to JED 1.x JSON format in parallel (ThreadPoolExecutor) - Runs prettier on all generated JSON files in a single batch Usage: python scripts/compile_po.py
FrancescoCastaldi
requested review from
betodealmeida,
craig-rueda,
dpgaspar,
eschutho,
geido,
hainenber,
kgabryje,
mistercrunch,
nytai,
rusackas,
sadpandajoe and
villebro
as code owners
github-actions
Bot
added
the
risk:ci-script
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #39724 +/- ##
==========================================
+ Coverage 64.35% 64.37% +0.01%
==========================================
Files 2569 2569
Lines 134680 134684 +4
Branches 31254 31255 +1
==========================================
+ Hits 86679 86707 +28
+ Misses 46505 46479 -26
- Partials 1496 1498 +2
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
![codeant-ai-for-open-source[bot]](https://avatars.githubusercontent.com/u/151810221?s=60&v=4){kind=small}
✅ Deploy Preview for superset-docs-preview ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
Added Apache License information to the top of the script.
![bito-code-review[bot]](https://avatars.githubusercontent.com/in/1061978?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review Agent Run #1072ef
Actionable Suggestions - 2
-
scripts/compile_po.py - 2
- Incorrect Fallback Condition · Line 74-74
- Broken npx Fallback · Line 85-89
Additional Suggestions - 1
-
scripts/compile_po.py - 1
-
Missing License Header · Line 1-1New files in this repository must include the Apache license header. Please add the standard ASF header above the shebang line.
-
Review Details
-
Files reviewed - 1 · Commit Range:
e6e8e36..e6e8e36- scripts/compile_po.py
-
Files skipped - 0
-
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
- MyPy (Static Code Analysis) - ✔︎ Successful
- Astral Ruff (Static Code Analysis) - ✔︎ Successful
Bito Usage Guide
Commands
Type the following command in the pull request comment and save the comment.
-
/review- Manually triggers a full AI review. -
/pause- Pauses automatic reviews on this pull request. -
/resume- Resumes automatic reviews. -
/resolve- Marks all Bito-posted review comments as resolved. -
/abort- Cancels all in-progress reviews.
Refer to the documentation for additional commands.
Configuration
This repository uses Superset You can customize the agent settings here or contact your Bito workspace admin at evan@preset.io.
Documentation & Help
AI Code Review powered by
✅ Deploy Preview for superset-docs-preview ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
✅ Deploy Preview for superset-docs-preview ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
Code Review Agent Run #f5ee6bActionable Suggestions - 0Review Details
Bito Usage GuideCommands Type the following command in the pull request comment and save the comment.
Refer to the documentation for additional commands. Configuration This repository uses Documentation & Help AI Code Review powered by |
Fix all pre-commit check failures: - I001: sort imports alphabetically (glob, importlib.util, os, shutil, subprocess, sys) - S603: add noqa comment on subprocess.run call - C901: reduce compile_translations complexity by extracting _run_pybabel, _sync_po_to_frontend, _convert_po_files_parallel, _run_prettier helpers - E501: break long lines at 88 chars - mypy: add full type annotations to all functions (from __future__ import annotations)
Refactor compile_po.py to improve npm package installation and conversion logic.
|
Thanks for putting this together. A few questions before I can sign off:
No objection to the parallelization idea — just want to make sure we're not adding a second translation pipeline indefinitely. |
ruff-format reformatted 1 file (scripts/compile_po.py)
Code Review Agent Run #ee5bf9Actionable Suggestions - 0Review Details
Bito Usage GuideCommands Type the following command in the pull request comment and save the comment.
Refer to the documentation for additional commands. Configuration This repository uses Documentation & Help AI Code Review powered by |
…k in compile_po.py - fix 'Incorrect Fallback Condition': remove early return None, None after npm install warning, allowing the code to proceed to check if local binaries exist (fixes unreachable code at lines 103-110) - fix 'Broken npx Fallback': add -y flag to npx calls for po2json and prettier to avoid interactive prompts in non-TTY environments
|
Thanks for the feedback @rusackas and @hainenber! Re: overlap with
The intent is not to remove Re: parallelization bottleneck (@hainenber) Re: Bito's findings (fallback condition + npx |
Code Review Agent Run #f6ea96Actionable Suggestions - 0Review Details
Bito Usage GuideCommands Type the following command in the pull request comment and save the comment.
Refer to the documentation for additional commands. Configuration This repository uses Documentation & Help AI Code Review powered by |
|
@FrancescoCastaldi thanks for sharing your perspective there. I wholly agree on making Having said that, the original script is extremely simple and IMO, if anything, it'd be better to have a 1-to-1 Python/JS equivalence. The more line/branch of code, the more maintenance effort in the long run. Re: possible benefit for parallelization in local machines, I don't think there are hardware out there getting bogged down by this lightweight translation, unless you're on Arduino or similar. I'd need a realistic stats on that. |
…lent of po2json.sh Remove parallelization (ThreadPoolExecutor, MAX_WORKERS) and all associated complexity in response to reviewer feedback. The new script is a minimal, sequential Python equivalent of po2json.sh: it iterates over .po files, calls po2json and prettier one at a time, and exits with a non-zero code on failure. This keeps the cross-platform benefit while minimizing maintenance overhead and line/branch count.
|
@hainenber Thanks for the thorough feedback you're right on both counts. I've just pushed a complete rewrite of
The only thing kept over the shell script is cross-platform compatibility (Windows path handling, no Let me know if this looks closer to what you had in mind! |
On Windows, npm global binaries (po2json, prettier) are installed as .cmd batch scripts and cannot be resolved by subprocess.run() with a list argument. Adding shell=True when os.name == 'nt' lets the Windows shell (cmd.exe) find and execute .cmd files transparently, while keeping shell=False on Unix for safety.
There was a problem hiding this comment.
Code Review Agent Run #7ed816
Actionable Suggestions - 1
-
scripts/compile_po.py - 1
- subprocess.run without explicit check argument · Line 33-33
Review Details
-
Files reviewed - 1 · Commit Range:
0ce3ebd..1e620e3- scripts/compile_po.py
-
Files skipped - 0
-
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
- MyPy (Static Code Analysis) - ✔︎ Successful
- Astral Ruff (Static Code Analysis) - ✔︎ Successful
Bito Usage Guide
Commands
Type the following command in the pull request comment and save the comment.
-
/review- Manually triggers a full AI review. -
/pause- Pauses automatic reviews on this pull request. -
/resume- Resumes automatic reviews. -
/resolve- Marks all Bito-posted review comments as resolved. -
/abort- Cancels all in-progress reviews.
Refer to the documentation for additional commands.
Configuration
This repository uses Superset You can customize the agent settings here or contact your Bito workspace admin at evan@preset.io.
Documentation & Help
AI Code Review powered by
hainenber
changed the title
po2json to cross-platform Python version
Co-authored-by: bito-code-review[bot] <188872107+bito-code-review[bot]@users.noreply.github.com>
Code Review Agent Run #262c32Actionable Suggestions - 0Review Details
Bito Usage GuideCommands Type the following command in the pull request comment and save the comment.
Refer to the documentation for additional commands. Configuration This repository uses Documentation & Help AI Code Review powered by |
3 participants
SUMMARY
Adds
scripts/compile_po.py, a standalone CLI utility to compile and convert Apache Superset translation files in a single automated pipeline. The script:.pofiles withpybabel compilesuperset-frontend/src/translationspo2jsonandprettierlocally once (avoids repeatednpx -ydownloads).pofiles to JED 1.x JSON format in parallel usingThreadPoolExecutor(up tomin(8, cpu_count*2)workers)prettieron all generated JSON files in a single batch callUsage:
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
TESTING INSTRUCTIONS
python scripts/compile_po.pyfrom the repository root..mofiles are generated undersuperset/translations/and.jsonfiles appear undersuperset-frontend/src/translations/.ADDITIONAL INFORMATION