chore(i18n): harden backfill_po — full language-name map + resilient batch translation - #41644
Merged
Conversation
The backfill prompt falls back to the raw ISO code when a language is absent from LANGUAGE_NAMES (e.g. "Translate ... into lv (lv)"), which gives Claude a weaker signal than a real language name. Add the six present catalogs that were missing: cs (Czech), fi (Finnish), lv (Latvian), ro (Romanian), sr_Latn (Serbian, Latin script), and th (Thai). LANGUAGE_NAMES now covers every locale directory under superset/translations. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A source msgid containing a literal double-quote (e.g. `"Exact match"`) can make the model echo unescaped quotes into its JSON response, so `json.loads` fails and the ENTIRE batch is discarded — losing up to `--batch-size` good translations per poison entry. `_resilient_translate` wraps `translate_batch`: on a JSON parse failure (ValueError) it bisects the batch and recurses, so only the offending entries are isolated; a lone entry that still fails falls back to a plain-text prompt (`_translate_single_plaintext`) that avoids the JSON envelope entirely. CLI failures (RuntimeError) still propagate to the existing per-batch handler. Adds four unit tests covering passthrough, bisect-and-fallback with correct index remapping, dropped-entry-on-fallback-failure, and RuntimeError propagation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
rusackas
requested review from
betodealmeida,
dpgaspar,
eschutho,
geido,
hainenber,
kgabryje,
mistercrunch,
nytai,
sadpandajoe,
sha174n and
villebro
as code owners
July 1, 2026 23:10
Contributor
Code Review Agent Run #e1ea99Actionable Suggestions - 0Review Details
Bito Usage GuideCommands Type the following command in the pull request comment and save the comment.
Refer to the documentation for additional commands. Configuration This repository uses Documentation & Help AI Code Review powered by |
github-actions
Bot
added
the
risk:ci-script
✅ Deploy Preview for superset-docs-preview ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #41644 +/- ##
========================================
Coverage 64.42% 64.42%
========================================
Files 2668 2670 +2
Lines 147181 147395 +214
Branches 33946 33985 +39
========================================
+ Hits 94815 94956 +141
- Misses 50649 50707 +58
- Partials 1717 1732 +15
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
sadpandajoe
approved these changes
Jul 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
SUMMARY
Improvements to
scripts/translations/backfill_po.py(the AI translation backfill tool), split into two focused commits. This is the tooling companion to the per-language backfill PRs (#41608 de, #41609 es, #41612 lv, #41613 fi, #41640 sk, #41641 th, …); it will accumulate further hardening as the sweep surfaces issues.1. Complete the
LANGUAGE_NAMESmapThe prompt falls back to the raw ISO code when a language is absent from
LANGUAGE_NAMES(e.g. "Translate … into lv (lv)"), a weaker signal than a real language name. Adds the six present catalogs that were missing:cs(Czech),fi(Finnish),lv(Latvian),ro(Romanian),sr_Latn(Serbian, Latin script),th(Thai). The map now covers every locale directory undersuperset/translations.2. Resilient batch translation (bisect + plain-text fallback)
A source
msgidcontaining a literal double-quote (e.g."Exact match") can make the model echo unescaped quotes into its JSON response, sojson.loadsfails and the entire batch is discarded — losing up to--batch-sizegood translations per offending entry. (Observed live on the Slovak catalog: 3 such strings sank ~100 translations across batches.)_resilient_translatewrapstranslate_batch: on a JSON parse failure it bisects the batch and recurses, isolating the offenders; a lone entry that still fails falls back to a plain-text prompt (_translate_single_plaintext) that avoids the JSON envelope entirely. CLI failures (RuntimeError) still propagate to the existing per-batch handler. Verified live on the Thai catalog — the same 3 quote-bearing entries were recovered automatically with zero batch loss.Adds 4 unit tests: passthrough, bisect-and-fallback with correct index remapping, dropped-entry-on-fallback-failure, and RuntimeError propagation.
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
N/A — internal tooling.
TESTING INSTRUCTIONS
(31 tests pass.)
ADDITIONAL INFORMATION