Japanese localization pipeline: glossary generation, list-driven do-not-translate masking, verified translation on PRs - #191
Merged
matenadasdi merged 20 commits intoAug 19, 2026
Conversation
…n/-prefix infra
Rebuilds ja-localization-full's tooling on top of main's now-merged
strip-en-prefix-urls PR, instead of merging the two histories directly:
that PR and this branch's NT-tagging commit both touched almost every
file under docs/ and src/partials/ for unrelated reasons (link-prefix
stripping vs. do-not-translate tagging), so a real merge produced 280+
conflicts. Content-bearing commits are cherry-picked as targeted patches
onto main's versions instead; superseded fixes (this branch's own navbar
/en/ regex, GlossTerm path, DocSidebar/DocBreadcrumbs hrefs) are dropped
in favor of main's better versions of the same fix.
Brings over:
- ja translation pipeline: .github/scripts/{build_ui_library,translate_docs}.py,
the ja-style-check/refresh-ui-library/translate-ja-docs workflows,
.textlintrc.yaml, localization/ (glossary, preferred translations,
translation guide, README).
- Locale-switch UI: navbar localeDropdown item in docusaurus.config.ts,
white-text CSS for it against the purple header (it would otherwise
render dark-on-dark), i18n/ja/'s UI-string scaffold.
- scripts/patch-api-info.js: disambiguates duplicate OpenAPI sidebar
labels, which broke `docusaurus write-translations` outright once a
second locale existed.
- scripts/check-links-source.js: strips JSX/MDX tags before slugifying
heading text, so a heading like "The <NT>Dashboards</NT> page" slugifies
the same as Docusaurus's real build-time slugger.
- src/components/NT + 'NT' added to docusaurus.config.ts's tag-escaping
preprocessor allowlist: the do-not-translate marker component itself,
not yet applied to any doc content (that's a separate change).
- scripts/add_notranslate_tags.py, the term-wrapping script this pipeline
runs — included here with a fix for a bug never caught before because it
was never rebased onto other real content changes: it wrapped terms
inside a heading's explicit anchor ID too (`{#workspace}` ->
`{#<NT>workspace</NT>}`), producing an invalid anchor. Fixed by excluding
`{#...}` spans the same way frontmatter/code/JSX attributes already are.
Still not run against docs content — that's the next commit.
- .claude/launch.json: a production-build-all-locales entry, since dev
mode only ever serves one locale at a time.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Bulk, mechanical rename of the 83 remaining plain-Markdown doc pages to .mdx — no content changes. Needed so scripts/add_notranslate_tags.py (reintroduced in the previous commit but not yet run) can wrap terms in these pages too: the <NT> marker it inserts is a JSX component, which .md files can't carry. Frontmatter `slug` values are untouched, so no URLs change and no redirects.json entries are needed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Plain Markdown tolerates an unclosed <br>, but MDX parses it as JSX,
which requires a matching closing tag or a self-closing slash. Renaming
these two pages to .mdx in the previous commit surfaced the mismatch
("Unexpected closing tag </details>, expected corresponding closing tag
for <br>") and failed the build. Verified: a full build of both en and
ja locales now succeeds.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…generated docs scripts/patch-api-info.js gained duplicate-sidebar-label disambiguation in an earlier commit, but that only takes effect by actually running the script — it doesn't retroactively touch the already-generated API reference files committed to the repo. Without this, building the ja locale fails outright: "Download files" duplicated across codespaces-service-session-download and codespaces-service-session-download-file broke `docusaurus write-translations`'s sidebar translation keys. Ran `node scripts/patch-api-info.js` and committed the result; the next `npm run gen-api-docs` will reproduce the same output automatically. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This translated page predates main's strip-en-prefix-urls merge, so its internal links still carried the old hardcoded /en/ prefix. Under the new per-locale baseUrl, that resolved to broken /ja/en/... URLs — the same class of bug that PR fixed everywhere else. Fixed with the same tool: `python3 scripts/strip_en_prefix.py` (i18n/ is already in its scope). Verified: building the ja locale no longer logs any /ja/en/ link warnings. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Left over from before the src_root/dest_root path-mapping logic it's a fallback for existed. Every real call passes a docs/-relative path (from git diff output, --src-root defaults to "docs"), so this branch was unreachable and, if it ever were reached, would guess wrong under the current per-locale routing anyway (paths here are file paths, not URLs, but the string was still a leftover assumption worth not carrying forward). Raises loudly instead of guessing, matching this repo's convention elsewhere (see promote-root-static-files.js) of failing hard rather than drifting silently. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Ran `docusaurus write-translations --locale ja` to sync current.json with main's sidebar changes since this scaffold was last generated: two new RDE API sidebar entries (git provider integration get/update) and a stale /en/-prefixed description string for the API reference link, now bare. No other i18n JSON file changed — navbar, footer, and code.json scaffolds are already in sync. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Brings in the infrastructure that was split out of this PR and has since merged — #188 (the <NT> component, the 83 .md -> .mdx renames, the check-links-source.js anchor fix, the patch-api-info.js duplicate-label fix) and #189 (the MCP sync /en/ fix) — plus #187's docusaurus-plugin-llms bump. The overlapping work is identical on both sides, so it resolves without conflict. What remains unique to this branch is the translation pipeline, the localization glossary and guides, the i18n/ja content, and the locale switcher. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ote NT tagging to an optional escape hatch
The glossary term tiers were only the tagger's input: protection required
materializing ~17k <NT> wrappers across 499 source files, and every glossary
change meant a mass re-tag diff. Now translate_docs.py compiles the same
tiers into masking regexes at translation time via a new shared matcher
(scripts/nt_terms.py, extracted verbatim from the tagger — full-corpus
report output is identical before/after: 17,141 wraps across 499 files),
so the docs source stays free of translation markup and a glossary edit is
a one-line change effective everywhere on the next translation.
- scripts/nt_terms.py: shared TermMatcher — generic tier loading (a renamed
tier can't silently drop out of protection, fixing the step_names /
step_names_and_field_labels regen mismatch), canonical steplib Step
titles, and all disambiguation rules (Title-Case gating, UI-context
gating, exact-case acronyms/code literals, suffix splitting).
- translate_docs.py: masks every term match after the structural pass
(acronyms included — masking them costs nothing, unlike tagging);
deterministic post-check that every masked token survives the model
output exactly once plus a stop_reason check (also closes the silent
8k-token truncation hole; max_tokens raised to 32k), with retries and a
non-zero exit instead of ever writing an unverified page.
- add_notranslate_tags.py: now an optional tool — reports what the matcher
protects by default, --write places manual <NT> escape-hatch tags for
page-specific cases (masked before term matching, so they always win).
- refresh-ui-library.yml: no docs re-tagging; the weekly PR touches only
localization/, a reviewable one-file-per-artifact diff.
- build_ui_library.py: preserves code_literals across regens; two new
protect patterns (whole import/export lines, {#heading-anchor} IDs).
- Glossary header, README, and JA-TRANSLATION-GUIDE updated to the new
design; dead preferred_translations stub removed.
Verified: mask -> identity-translate -> verify -> unmask roundtrips
byte-exact on 60 real pages (3,710 tokens); dropped/invented-token
detection confirmed; tagger --write idempotent.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
matenadasdi
requested review from
aorcsik,
ilanazholobovsky and
zoltan-baba
as code owners
August 18, 2026 17:42
…lation-nt-tagging-f787d1
matenadasdi
changed the title
…rnative to ANTHROPIC_API_KEY Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…equests at 32k max_tokens Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… crashing Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- env_var pattern now requires a digit/underscore (or $ prefix): a bare
ALL-CAPS run is as often English emphasis ("NOT", "MUST") as an
identifier, and masking it splices untranslated English into the Japanese
sentence (real acronyms stay protected exact-case via the acronyms tier).
The old flow had the same bug but hid it — without token verification the
model silently dropped the masked emphasis word.
- bold promotion generalized: every **...** span in the JA output becomes
<strong>...</strong>, run before unmasking so code can't be touched. The
old version was gated on <NT> spans, which no longer exist in output, so
bold-against-particle CommonMark breakage (**X**を...) had returned.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Produced by a real translate_docs.py run on docs/bitrise-ci/getting-started/ getting-started.mdx — replaces the pilot translation from the old tagged-source flow. No <NT> markup or NT import in the output; protected terms restored as plain text and token-verified; bold spans emitted as <strong>; code blocks byte-identical to the EN source. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Auto-translation on docs PRs ships disabled until enough pages are migrated to make lock-step /ja/ worthwhile. The job skips silently (no failing check, no noise); enabling is flipping one repo variable, no code change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…f a repo variable Adding the secret is the single ON switch; while it doesn't exist the job skips silently on every docs PR. (secrets isn't available in a job-level if, so a tiny check job exports its presence as an output.) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every other class of do-not-translate content (code, URLs, product terms, Step names) is masked out of the text before the model ever sees it. Internal relative links (](/bitrise-ci/foo), the site's own convention since scripts/strip_en_prefix.py) were the one exception, relying instead on the system prompt's "preserve links" instruction — the same class of risk this pipeline was specifically built to eliminate everywhere else. There's no legitimate translated form of a route, so mask it the same way the existing `url` pattern already masks https://... targets. Added to build_ui_library.py's PROTECT_PATTERNS (the canonical source — ja-do-not-translate-glossary.yaml's protect_patterns section is regenerated from it weekly) and mirrored directly into the checked-in glossary so it's effective immediately rather than waiting on a refresh run. Also incidentally protects markdown image targets (), which have the same shape and the same reasoning. Verified: mask/unmask round-trips byte-exact on a real translated page, and masking runs clean with zero errors across all 466 real docs + partials files in the repo. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The workflow's changed-file detection used `git diff --name-only --diff-filter=AM`, which discards D (deleted) and R (renamed/moved) entries outright — neither case was ever visible to translate_docs.py. Deleting an English page left its Japanese translation orphaned forever; renaming one left the old JA file behind at its stale path with no translation triggered for the new one (a pure rename shows as neither A nor M). - translate-ja-docs.yml now emits full `git diff --name-status -M` instead of the filtered --name-only, scoped to docs/ via pathspec. - translate_docs.py's new parse_changes() reads the status codes: A/M -> translate (unchanged), D -> delete the JA counterpart, R -> move the JA counterpart to match (dest_path() on both the old and new git-reported path — a move is just a rename where the changed path segment happens to be the directory instead of the filename, so it's the same operation). A plain path with no status prefix still means "translate" unchanged, preserving manual CLI usage. - Edge case: a rename whose JA counterpart was never translated in the first place (nothing to move) falls through to queuing the new path for translation instead, so a renamed-but-not-yet-translated page doesn't silently stay untranslated. - Edge case, left unhandled deliberately: if a rename is bundled with a content rewrite large enough that git's similarity threshold can't recognize it as the same file, git reports it as plain D + A instead of R. That already falls through correctly to the D and A handling above with no special-casing needed — arguably correct anyway, since content changed enough that git can't tell it's the same file probably warrants a fresh translation, not just a relocation. - Delete/rename dispatch runs before the ANTHROPIC_API_KEY check, so a PR that only deletes or moves pages (no content changes) doesn't require one. Verified: parse_changes() unit-tested against realistic --name-status output (add/modify/delete/two renames/excluded api-reference/excluded non-.mdx); delete and rename dispatch integration-tested against a scratch directory tree, including the no-existing-counterpart fallback. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A fast-follow push to the same PR — pushing, noticing something, pushing again — starts a new run while the previous one may still be mid- translation. The old run's own `git push` at the end is the only step that runs after translation, and with no concurrency control it can either fail outright as a non-fast-forward, or (if the timing lines up) succeed at pushing a Japanese translation of English content that's already outdated by the second push. Cancelling the superseded run outright, before it ever reaches that final push, is cleaner than teaching the push step to rebase and retry: a cancelled run can't push stale content by definition, and it doesn't burn API calls translating something about to be superseded anyway. Same pattern deploy.yml already uses for the identical class of problem, keyed to the PR branch instead of the ref since this workflow only ever triggers on pull_request. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the Japanese localization pipeline: docs are machine-translated to
jaon every docs PR, with do-not-translate terms enforced structurally from two maintained lists — no translation markup in the docs source.What it does
build_ui_library.py+refresh-ui-library.yml— weekly scan of the product frontends + steplib + docs regeneratesui_copy_library.jsonand the tiered do-not-translate glossary (localization/ja-do-not-translate-glossary.yaml). The refresh PR touches onlylocalization/.translate_docs.py+translate-ja-docs.yml— on every docs PR, changed pages are translated to Japanese and committed back to the same PR. Ships disabled: the job is gated on the existence of theANTHROPIC_API_KEYsecret and skips silently while it's absent — adding the secret is the ON switch once enough pages are migrated (only adding files underi18n/ja/, never touching the author's files), so EN and JA can't drift apart. Front matter is never sent to the model, soslugcan't be touched.scripts/nt_terms.py— shared term matcher that compiles the glossary tiers (plus canonical steplib Step titles) into masking rules, with disambiguation: Title-Case gating for ambiguous single words, UI-context gating for labels, exact-case acronyms, inflectional suffixes left translatable.scripts/add_notranslate_tags.py— optional tool: reports what the matcher would protect on a page;--writeplaces a manual<NT>escape-hatch tag for the rare case the lists can't decide (a manual tag always wins).i18n/ja/locale content, navbar locale switcher, JTF textlint check,localization/README.md+JA-TRANSLATION-GUIDE.md.What contributing looks like
Nothing changes for docs authors:
docs/and open a PR, exactly as today.i18n/ja/, never touching your own files. (While auto-translation is disabled, this step simply shows as skipped.)/ja/can never silently drift behind.No tags to write, no translation awareness needed. If a new product term gets machine-translated when it shouldn't be, the fix is a one-line addition to
localization/ja-do-not-translate-glossary.yaml— effective everywhere on the next translation.The workflows
translate-ja-docs.ymldocs/**ANTHROPIC_API_KEYsecret is the ON switch.refresh-ui-library.ymllocalization/. No Claude API — pure scanning; needs theCI_REPO_TOKENsecret.ja-style-check.ymli18n/ja/**.mdx.Why it works this way
Verified
npm run buildfor both locales; pilot page and locale switching checked in a served production build.Known gaps (documented in
JA-TRANSLATION-GUIDE.md)No translation memory yet; partials, front matter, and the sidebar tree are untranslated; the textlint check is a no-op on
.mdxfor now;/ja/*stays noindexed until coverage justifies flipping it;ja-preferred-translations.yamlseeds need a fluent Japanese reviewer.🤖 Generated with Claude Code