diff --git a/README.md b/README.md index dd1b2a5..a37c47f 100644 --- a/README.md +++ b/README.md @@ -1,14 +1,23 @@ # boost-docs-translation -Mirrors Boost library documentation repositories into the CppDigest org, maintains -submodule links on `master` and `local-{lang_code}` branches, and keeps local-branch -pointers up to date on a daily schedule. +Super-repository for Boost library documentation translations: it holds `libs/*` +submodules that point at per-library mirrors, keeps **`master`** and +**`local-{lang_code}`** branches aligned with upstream **`boostorg`** sources, and +notifies a Weblate instance when components change. A daily workflow advances +submodule pointers on every **`local-*`** branch to match each library repo’s +corresponding **`local-*`** tip. + +The GitHub org used for those library mirrors defaults to **this repository’s +org**; set repository variable **`SUBMODULES_ORG`** to use a +different org. + +For system context, branch model, and data flows, see **[ARCHITECTURE.md](docs/ARCHITECTURE.md)**. --- ## Workflows -### `add-submodules.yml` — Add new submodules +### `add-submodules.yml` — Create library mirrors and register submodules **Trigger:** `repository_dispatch` with `event_type: add-submodules` @@ -16,24 +25,27 @@ pointers up to date on a daily schedule. {"event_type": "add-submodules", "client_payload": {"version": "boost-1.90.0"}} ``` -For each Boost library submodule: -1. Skips if the repo already exists in the CppDigest org. -2. Fetches `meta/libraries.json` to determine doc paths, clones the `boostorg` repo at - the given ref, and prunes to doc folders only. -3. Creates `CppDigest/`, pushes doc content to `master`, creates - `local-{lang_code}` branches for each language, and installs `create-tag.yml` - (from `assets/`) into the new repo. -4. Updates submodule links in this repo (`libs/`) on `master` and each `local-{lang_code}` branch. +For each Boost library name in the resolved list: + +1. Skips if **`{MODULE_ORG}/{submodule}`** already exists (`MODULE_ORG` is + **`SUBMODULES_ORG`** if set, otherwise the translations repo’s org). +2. Fetches **`meta/libraries.json`** from **`boostorg/{submodule}`** to determine doc + paths, clones that repo at the given ref, and prunes to doc folders only. +3. Creates **`{MODULE_ORG}/{submodule}`**, pushes doc content to **`master`**, creates + **`local-{lang_code}`** branches for each configured language, and copies + **`create-tag.yml`** from **`.github/workflows/assets/`** into the new repo. +4. Updates submodule links in this repo under **`libs/`** on **`master`** and each + **`local-{lang_code}`** branch. | `client_payload` field | Required | Description | |---|---|---| | `version` | no | Boost ref (e.g. `boost-1.90.0`). Defaults to `develop`. | -| `submodules` | no | List-like string (e.g. `[algorithm, system]`). If omitted, reads from `.gitmodules`. | -| `lang_codes` | no | Comma-separated language codes (e.g. `zh_Hans,ja`). Defaults to `vars.LANG_CODES`. | +| `submodules` | no | List-like string (e.g. `[algorithm, system]`). If omitted, submodule names are taken from **`.gitmodules` on `boostorg/boost`** at **`version`** (only `libs/` paths). | +| `lang_codes` | no | Comma-separated language codes (e.g. `zh_Hans,ja`). Defaults to **`vars.LANG_CODES`**; the workflow fails if neither this field nor **`LANG_CODES`** is set. | --- -### `start-translation.yml` — Sync existing submodules +### `start-translation.yml` — Sync existing mirrors and notify Weblate **Trigger:** `repository_dispatch` with `event_type: start-translation` @@ -41,25 +53,33 @@ For each Boost library submodule: {"event_type": "start-translation", "client_payload": {"version": "boost-1.90.0"}} ``` -Reads the submodule list from `.gitmodules` of this repo (only `libs/` entries). -For each lang code and each submodule: -1. Ensures this repo has a `local-{lang_code}` branch. -2. Syncs `CppDigest/{lib}` master from the upstream `boostorg` repo. -3. Manages the `local-{lang_code}` branch in the lib repo: creates it if missing, - or merges master into it if no open translation PR exists; skips if a PR is open. -4. Updates submodule pointers on `master` and each `local-{lang_code}` branch. -5. POSTs to Weblate with an `add_or_update` map of `{lang_code: [submodules, ...]}`. - Skipped if all entries are empty (no submodules were updated). +Reads the submodule list from **this repo’s `.gitmodules`** on **`master`** (only +**`libs/`** entries). For each language and each submodule: + +1. Ensures this repo has a **`local-{lang_code}`** branch. +2. Syncs **`{MODULE_ORG}/{lib}` `master`** from the upstream **`boostorg`** repo + (same prune rules as **`add-submodules`**). +3. In the library repo: creates **`local-{lang_code}`** if missing, or merges + **`master`** into it when there is **no** open PR into **`local-{lang_code}`** + whose head branch starts with **`translation-{lang_code}-`**; otherwise skips + that lib for that language so in-flight Weblate work is not overwritten. +4. Updates submodule pointers here on **`master`** and each **`local-{lang_code}`** + branch ( **`local-*`** updates are force-pushed when finalizing). +5. **POST**s JSON to **`{WEBLATE_URL}/boost-endpoint/add-or-update/`** with + **`organization`**, **`version`**, optional **`extensions`**, and + **`add_or_update`**: `{lang_code: [submodule names, ...]}` for libs that were + actually updated for that language. Omits the call if the map would be empty. + A typical server response is **HTTP 202** (async); **200** is also accepted. | `client_payload` field | Required | Description | |---|---|---| | `version` | no | Boost ref (e.g. `boost-1.90.0`). Defaults to `develop`. | -| `lang_codes` | no | Comma-separated language codes (e.g. `zh_Hans,ja`). Defaults to `vars.LANG_CODES`. | -| `extensions` | no | File extensions for Weblate (e.g. `[.adoc, .md]`). Default: empty (all supported). | +| `lang_codes` | no | Comma-separated language codes (e.g. `zh_Hans,ja`). Defaults to **`vars.LANG_CODES`**; the workflow fails if neither this field nor **`LANG_CODES`** is set. | +| `extensions` | no | File extensions for Weblate (e.g. `[.adoc, .md]`). Default: empty (no filter in the payload). | --- -### `sync-translation.yml` — Sync local-branch pointers +### `sync-translation.yml` — Advance submodule pointers on `local-*` branches **Trigger:** `repository_dispatch` with `event_type: sync-translation`, or daily schedule (`0 0 * * *`) @@ -67,9 +87,10 @@ For each lang code and each submodule: {"event_type": "sync-translation"} ``` -Discovers all remote `local-*` branches in this repo, then for each one: -checks it out with submodules, advances every submodule pointer to the tip of that -submodule's own `local-*` branch, commits, and force-pushes. +Discovers all remote **`local-*`** branches in this repo, then for each one: checks +it out with submodules, sets each submodule’s tracking branch to that name, +runs **`git submodule update --remote`**, commits if pointers changed, and +**force-pushes** the branch. No `client_payload` fields. @@ -77,15 +98,38 @@ No `client_payload` fields. ## Assets -### `.github/workflows/assets/create-tag.yml` +Shared workflow snippets live under **`.github/workflows/assets/`**. + +### `create-tag.yml` + +Copied into each library mirror repo when **`local-{lang_code}`** is created. +When a Weblate PR (**`translation-{lang_code}-{version}`** → **`local-{lang_code}`**) +is merged, it creates tag **`{version}-{repo}-{lang_code}`** if it does not already +exist. + +See [`.github/workflows/assets/README.md`](.github/workflows/assets/README.md) for +branch and tag naming details. + +### `env.sh` and `lib.sh` + +Sourced by **`add-submodules`** and **`start-translation`**: org/repo names, clone +and prune helpers, translations-repo branch setup, submodule pointer updates, and +list parsing. + +--- + +## Scripts (local `repository_dispatch`) + +From a clone of this repo: -A workflow template copied into each CppDigest lib repo by `add-submodules.yml` and -`start-translation.yml`. Triggers when a Weblate translation PR -(`translation-{lang_code}-{version}` → `local-{lang_code}`) is merged, and creates a -versioned tag of the form `{version}-{repo}-{lang_code}` -(e.g. `boost-1.90.0-algorithm-zh_Hans`). Skipped if the tag already exists. +- **`scripts/trigger-add-submodules.sh`** — fires **`add-submodules`**. +- **`scripts/trigger-start-translation.sh`** — fires **`start-translation`** (optional + **`--version`**, **`--lang-codes`**, **`--extensions`**). -See [`assets/README.md`](.github/workflows/assets/README.md) for details. +Copy **`.env.example`** to **`.env`** and set **`GH_TOKEN`** (or **`GITHUB_TOKEN`**) +with permission to call **`POST /repos/{owner}/{repo}/dispatches`** on the target +repo. The workflows still use GitHub **secrets** and **variables** on the server as +documented below. --- @@ -93,14 +137,15 @@ See [`assets/README.md`](.github/workflows/assets/README.md) for details. | Secret | Used by | Description | |---|---|---| -| `SYNC_TOKEN` | all workflows | PAT with `repo` scope (and org repo-create permission for `add-submodules`). | -| `WEBLATE_URL` | `start-translation` | Weblate instance URL. | -| `WEBLATE_TOKEN` | `start-translation` | Weblate API token. | +| `SYNC_TOKEN` | all workflows | PAT with **`repo`** scope; **`add-submodules`** also needs permission to create org repositories when creating new mirrors. | +| `WEBLATE_URL` | `start-translation` | Base URL of the Weblate instance (the workflow appends **`boost-endpoint/add-or-update/`**). | +| `WEBLATE_TOKEN` | `start-translation` | API token for that endpoint. | ## Repository variables | Variable | Used by | Description | |---|---|---| -| `LANG_CODES` | `add-submodules`, `start-translation` | Optional. Default comma-separated language codes (e.g. `zh_Hans,ja`). | +| `LANG_CODES` | `add-submodules`, `start-translation` | Default language codes when **`client_payload.lang_codes`** is omitted (comma- or bracket-list, e.g. `zh_Hans,ja`). Must be set here or passed in the dispatch payload. | +| `SUBMODULES_ORG` | `add-submodules`, `start-translation` | Optional. GitHub org for **`boostorg`** mirror repos (e.g. `CppDigest`). If unset, the org is the same as this repository’s owner. **`sync-translation`** relies on **`.gitmodules`** URLs already pointing at the correct hosts. | --- diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md new file mode 100644 index 0000000..51c4836 --- /dev/null +++ b/docs/ARCHITECTURE.md @@ -0,0 +1,181 @@ +# Architecture — boost-docs-translation + +This document describes **why** the system is shaped the way it is, **how** the main +pieces relate, and **which** concerns cut across workflows. For triggers, secrets, +and copy-paste examples, see [README.md](../README.md). + +--- + +## 1. Problem and goals + +**Context.** Boost ships documentation inside many separate **`boostorg`** repositories. +Translation work needs stable, narrow remotes (doc-only trees), language-specific +branches, and automation that does not clobber in-progress translator submissions. + +**Goals.** + +- **Mirror** each relevant library’s documentation into a dedicated GitHub repo under + a configurable org (**`MODULE_ORG`**, from **`SUBMODULES_ORG`**). +- **Super-repo** (**this repository**): one checkout that pins exact commits of every + mirrored lib via **`libs/`** submodules on **`master`** and on + **`local-{lang_code}`** branches. +- **Upstream sync**: refresh mirror **`master`** from **`boostorg`** at a chosen ref, + then fold changes into **`local-*`** when safe. +- **Weblate handoff**: after successful git updates, tell the translation server which + org, version, languages, and components changed so it can add or refresh projects + asynchronously. +- **Consumer branches**: keep every **`local-*`** branch in the super-repo pointing at + the latest **`local-*`** tip of each submodule (scheduled **`sync-translation`**). + +Non-goals: building Boost, serving rendered HTML from this repo, or replacing +Weblate’s internal project configuration beyond the **`add-or-update`** contract. + +--- + +## 2. System context + +```mermaid +flowchart LR + subgraph upstream [Upstream] + BO[boostorg repos] + BB[boostorg/boost .gitmodules] + end + + subgraph ci [GitHub Actions] + AS[add-submodules] + ST[start-translation] + SY[sync-translation] + end + + subgraph mirrors [Library mirrors MODULE_ORG] + M1["repo: algorithm"] + M2["repo: json"] + end + + subgraph super [Translations super-repo] + TR[this repo master and local-*] + end + + subgraph tw [Weblate] + EP["POST .../boost-endpoint/add-or-update/"] + end + + BB --> AS + BO --> AS + BO --> ST + AS --> M1 + AS --> M2 + ST --> M1 + ST --> M2 + AS --> TR + ST --> TR + ST --> EP + M1 --> TR + M2 --> TR + SY --> TR +``` + +**Legend.** **`add-submodules`** may discover names from **`boostorg/boost`**; +**`start-translation`** discovers names from **this repo’s `.gitmodules`**. Both +mutate mirror repos and then update the super-repo. **`sync-translation`** only +moves submodule SHAs on existing **`local-*`** branches. + +--- + +## 3. Repository layout (codemap) + +| Path | Role | +|------|------| +| **`.gitmodules`** | Declares **`libs/`** → **`https://github.com/{MODULE_ORG}/{lib}.git`**, branch **`master`** for default checkout metadata. | +| **`.github/workflows/add-submodules.yml`** | On dispatch: create missing mirrors, push **`master`** / **`local-*`**, install **`create-tag.yml`**, record submodules in the super-repo. | +| **`.github/workflows/start-translation.yml`** | On dispatch: sync existing mirrors, merge policy on **`local-*`**, update super-repo pointers, call Weblate. | +| **`.github/workflows/sync-translation.yml`** | On dispatch or schedule: for each **`local-*`**, **`submodule update --remote`** and force-push. | +| **`.github/workflows/assets/env.sh`** | Derives **`ORG`**, **`TRANSLATIONS_REPO`**, **`MODULE_ORG`**, **`BOOST_ORG`**, **`MASTER_BRANCH`**, bot identity. | +| **`.github/workflows/assets/lib.sh`** | Shared implementation: GitHub **`gh`** helpers, clone/prune, **`meta/libraries.json`** parsing, translations-repo branch and submodule updates. | +| **`.github/workflows/assets/create-tag.yml`** | Template copied into each mirror; tags merged Weblate PRs (see **assets/README.md**). | +| **`scripts/trigger-*.sh`** | Optional local wrappers around **`repository_dispatch`**; no server-side logic. | + +**Dependency direction.** Inline bash in **`add-submodules.yml`** and +**`start-translation.yml`** sources **`env.sh`** then **`lib.sh`**. **`sync-translation`** +sources only **`env.sh`** (git-only loop, no **`lib.sh`**). + +--- + +## 4. Branch model + +| Branch | Where | Meaning | +|--------|--------|---------| +| **`master`** | Mirror and super-repo | Doc-only content aligned with upstream English sources for the synced ref. | +| **`local-{lang_code}`** | Mirror and super-repo | Translation line for **`lang_code`**; Weblate opens PRs from **`translation-{lang_code}-*`** into this branch on mirrors. | +| **`translation-{lang_code}-{version}`** | Mirror (head of PR) | Weblate working branch naming assumed by **`start-translation`** (open PR guard) and **`create-tag.yml`** (tag extraction). | + +**Super-repo `local-*` vs mirror `local-*`.** The super-repo’s **`local-*`** branch +commits **submodule pointers** that track each mirror’s **`local-*`** (after +**`finalize_translations_repo`** / **`sync-translation`**). The mirror’s **`local-*`** +holds **actual file** merges from **`master`** plus translator edits. + +--- + +## 5. Control and data flows + +### 5.1 Add mirrors (**`add-submodules`**) + +1. Resolve library names: **`client_payload.submodules`** or **`boostorg/boost`** + **`.gitmodules`** at **`LIBS_REF`**. +2. For each name: **`get_doc_paths`** ( **`meta/libraries.json`** ) → clone + **`boostorg/{name}`** → **`prune_to_doc_only`**. +3. If mirror missing: **`gh repo create`**, init from pruned tree, push **`master`**, + branch **`local-*`** with **`create-tag.yml`** committed on each. +4. Clone super-repo to a temp dir; **`ensure_local_branch_in_translations`** per + language; for each updated lib, **`update_translations_submodule`** on **`master`** + and each **`local-*`**; push ( **`local-*`** force-pushed in **`finalize_translations_repo`** ). + +### 5.2 Sync and notify (**`start-translation`**) + +1. Submodule names from **this** repo **`.gitmodules`** (**`libs/`** only). +2. Per lib: clone upstream and mirror, replace mirror **`master`** tree with pruned + upstream snapshot, push. +3. Per **`lang_code`**: if **`local-*`** exists and an open PR matches + **`translation-{lang_code}-*`** → skip merge for that pair; else create branch or + **`merge origin/master`** into **`local-*`** and push. +4. **`finalize_translations_repo`**: refresh **`UPDATES`** submodules on **`master`** + (normal push) and on each **`local-*`** (**`--force`**). +5. Build **`add_or_update`** map only for languages that had at least one successful + update; **`jq`**-construct JSON; **POST** to Weblate **`boost-endpoint`**; tolerate + **202** (async) or **200**. + +### 5.3 Pointer roll-up (**`sync-translation`**) + +1. List **`refs/heads/local-*`** on the super-repo. +2. For each branch: checkout, **`git submodule update --init`**, set + **`submodule..branch`**, **`submodule update --remote`**, commit, **`push --force`**. + +--- + +## 6. Cross-cutting concerns + +**Authentication.** **`SYNC_TOKEN`** is passed to **`actions/checkout`** and +**`GITHUB_TOKEN`** for **`gh`** and **`gh auth setup-git`**, so pushes and API calls +share one credential. **`WEBLATE_TOKEN`** is only used for the outbound HTTP call. + +**Idempotency and safety.** + +- **`add-submodules`** skips if the mirror repo already exists. +- **`start-translation`** skips mirror **`master`** merge into **`local-*`** when a + Weblate PR is open, reducing race risk with translators. +- **`create-tag`** skips if the derived tag already exists. + +**Observability.** Shell steps echo progress to Actions logs; Weblate response bodies +are printed for **`start-translation`** on success or failure paths where implemented. + +**Configuration surface.** **`env.sh`** centralizes org and branch constants; workflow +YAML supplies **`LIBS_REF`**, **`LANG_CODES`**, **`EXTENSIONS`**, and secrets so +behavior is traceable without reading the entire inline script. + +--- + +## 7. Stability of this document + +Update this file when **branch naming**, **Weblate contract**, **ownership model** +(**`MODULE_ORG`**), or **repository layout** changes. Routine secret rotation or +version bumps in **`actions/checkout`** alone do not require edits here.