Skip to content

Add architecture documentation. #8

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
wpak-ai merged 3 commits into from
May 6, 2026
+267 −41
Merged

Add architecture documentation. #8

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
dd35cd7
Add architecture documentation.
whisper67265 May 6, 2026
64e2fcf
Remove double one.
whisper67265 May 6, 2026
25867fd
Update the documentations due to the first review.
whisper67265 May 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
127 changes: 86 additions & 41 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,106 +1,151 @@
# boost-docs-translation

Mirrors Boost library documentation repositories into the CppDigest org, maintains
submodule links on `master` and `local-{lang_code}` branches, and keeps local-branch
pointers up to date on a daily schedule.
Super-repository for Boost library documentation translations: it holds `libs/*`
submodules that point at per-library mirrors, keeps **`master`** and
**`local-{lang_code}`** branches aligned with upstream **`boostorg`** sources, and
notifies a Weblate instance when components change. A daily workflow advances
submodule pointers on every **`local-*`** branch to match each library repo’s
corresponding **`local-*`** tip.

The GitHub org used for those library mirrors defaults to **this repository’s
org**; set repository variable **`SUBMODULES_ORG`** to use a
different org.

For system context, branch model, and data flows, see **[ARCHITECTURE.md](docs/ARCHITECTURE.md)**.

---

## Workflows

### `add-submodules.yml` — Add new submodules
### `add-submodules.yml` — Create library mirrors and register submodules

**Trigger:** `repository_dispatch` with `event_type: add-submodules`

```json
{"event_type": "add-submodules", "client_payload": {"version": "boost-1.90.0"}}
```

For each Boost library submodule:
1. Skips if the repo already exists in the CppDigest org.
2. Fetches `meta/libraries.json` to determine doc paths, clones the `boostorg` repo at
the given ref, and prunes to doc folders only.
3. Creates `CppDigest/<submodule>`, pushes doc content to `master`, creates
`local-{lang_code}` branches for each language, and installs `create-tag.yml`
(from `assets/`) into the new repo.
4. Updates submodule links in this repo (`libs/`) on `master` and each `local-{lang_code}` branch.
For each Boost library name in the resolved list:

1. Skips if **`{MODULE_ORG}/{submodule}`** already exists (`MODULE_ORG` is
**`SUBMODULES_ORG`** if set, otherwise the translations repo’s org).
2. Fetches **`meta/libraries.json`** from **`boostorg/{submodule}`** to determine doc
paths, clones that repo at the given ref, and prunes to doc folders only.
3. Creates **`{MODULE_ORG}/{submodule}`**, pushes doc content to **`master`**, creates
**`local-{lang_code}`** branches for each configured language, and copies
**`create-tag.yml`** from **`.github/workflows/assets/`** into the new repo.
4. Updates submodule links in this repo under **`libs/`** on **`master`** and each
**`local-{lang_code}`** branch.

| `client_payload` field | Required | Description |
|---|---|---|
| `version` | no | Boost ref (e.g. `boost-1.90.0`). Defaults to `develop`. |
| `submodules` | no | List-like string (e.g. `[algorithm, system]`). If omitted, reads from `.gitmodules`. |
| `lang_codes` | no | Comma-separated language codes (e.g. `zh_Hans,ja`). Defaults to `vars.LANG_CODES`. |
| `submodules` | no | List-like string (e.g. `[algorithm, system]`). If omitted, submodule names are taken from **`.gitmodules` on `boostorg/boost`** at **`version`** (only `libs/` paths). |
| `lang_codes` | no | Comma-separated language codes (e.g. `zh_Hans,ja`). Defaults to **`vars.LANG_CODES`**; the workflow fails if neither this field nor **`LANG_CODES`** is set. |

---

### `start-translation.yml` — Sync existing submodules
### `start-translation.yml` — Sync existing mirrors and notify Weblate

**Trigger:** `repository_dispatch` with `event_type: start-translation`

```json
{"event_type": "start-translation", "client_payload": {"version": "boost-1.90.0"}}
```

Reads the submodule list from `.gitmodules` of this repo (only `libs/` entries).
For each lang code and each submodule:
1. Ensures this repo has a `local-{lang_code}` branch.
2. Syncs `CppDigest/{lib}` master from the upstream `boostorg` repo.
3. Manages the `local-{lang_code}` branch in the lib repo: creates it if missing,
or merges master into it if no open translation PR exists; skips if a PR is open.
4. Updates submodule pointers on `master` and each `local-{lang_code}` branch.
5. POSTs to Weblate with an `add_or_update` map of `{lang_code: [submodules, ...]}`.
Skipped if all entries are empty (no submodules were updated).
Reads the submodule list from **this repo’s `.gitmodules`** on **`master`** (only
**`libs/`** entries). For each language and each submodule:

1. Ensures this repo has a **`local-{lang_code}`** branch.
2. Syncs **`{MODULE_ORG}/{lib}` `master`** from the upstream **`boostorg`** repo
(same prune rules as **`add-submodules`**).
3. In the library repo: creates **`local-{lang_code}`** if missing, or merges
**`master`** into it when there is **no** open PR into **`local-{lang_code}`**
whose head branch starts with **`translation-{lang_code}-`**; otherwise skips
that lib for that language so in-flight Weblate work is not overwritten.
4. Updates submodule pointers here on **`master`** and each **`local-{lang_code}`**
branch ( **`local-*`** updates are force-pushed when finalizing).
5. **POST**s JSON to **`{WEBLATE_URL}/boost-endpoint/add-or-update/`** with
**`organization`**, **`version`**, optional **`extensions`**, and
**`add_or_update`**: `{lang_code: [submodule names, ...]}` for libs that were
actually updated for that language. Omits the call if the map would be empty.
A typical server response is **HTTP 202** (async); **200** is also accepted.

| `client_payload` field | Required | Description |
|---|---|---|
| `version` | no | Boost ref (e.g. `boost-1.90.0`). Defaults to `develop`. |
| `lang_codes` | no | Comma-separated language codes (e.g. `zh_Hans,ja`). Defaults to `vars.LANG_CODES`. |
| `extensions` | no | File extensions for Weblate (e.g. `[.adoc, .md]`). Default: empty (all supported). |
| `lang_codes` | no | Comma-separated language codes (e.g. `zh_Hans,ja`). Defaults to **`vars.LANG_CODES`**; the workflow fails if neither this field nor **`LANG_CODES`** is set. |
| `extensions` | no | File extensions for Weblate (e.g. `[.adoc, .md]`). Default: empty (no filter in the payload). |

---

### `sync-translation.yml` — Sync local-branch pointers
### `sync-translation.yml` — Advance submodule pointers on `local-*` branches

**Trigger:** `repository_dispatch` with `event_type: sync-translation`, or daily schedule (`0 0 * * *`)

```json
{"event_type": "sync-translation"}
```

Discovers all remote `local-*` branches in this repo, then for each one:
checks it out with submodules, advances every submodule pointer to the tip of that
submodule's own `local-*` branch, commits, and force-pushes.
Discovers all remote **`local-*`** branches in this repo, then for each one: checks
it out with submodules, sets each submodule’s tracking branch to that name,
runs **`git submodule update --remote`**, commits if pointers changed, and
**force-pushes** the branch.

No `client_payload` fields.

---

## Assets

### `.github/workflows/assets/create-tag.yml`
Shared workflow snippets live under **`.github/workflows/assets/`**.

### `create-tag.yml`

Copied into each library mirror repo when **`local-{lang_code}`** is created.
When a Weblate PR (**`translation-{lang_code}-{version}`** → **`local-{lang_code}`**)
is merged, it creates tag **`{version}-{repo}-{lang_code}`** if it does not already
exist.

See [`.github/workflows/assets/README.md`](.github/workflows/assets/README.md) for
branch and tag naming details.

### `env.sh` and `lib.sh`

Sourced by **`add-submodules`** and **`start-translation`**: org/repo names, clone
and prune helpers, translations-repo branch setup, submodule pointer updates, and
list parsing.

---

## Scripts (local `repository_dispatch`)

From a clone of this repo:

A workflow template copied into each CppDigest lib repo by `add-submodules.yml` and
`start-translation.yml`. Triggers when a Weblate translation PR
(`translation-{lang_code}-{version}` → `local-{lang_code}`) is merged, and creates a
versioned tag of the form `{version}-{repo}-{lang_code}`
(e.g. `boost-1.90.0-algorithm-zh_Hans`). Skipped if the tag already exists.
- **`scripts/trigger-add-submodules.sh`** — fires **`add-submodules`**.
- **`scripts/trigger-start-translation.sh`** — fires **`start-translation`** (optional
**`--version`**, **`--lang-codes`**, **`--extensions`**).

See [`assets/README.md`](.github/workflows/assets/README.md) for details.
Copy **`.env.example`** to **`.env`** and set **`GH_TOKEN`** (or **`GITHUB_TOKEN`**)
with permission to call **`POST /repos/{owner}/{repo}/dispatches`** on the target
repo. The workflows still use GitHub **secrets** and **variables** on the server as
documented below.

---

## Required secrets

| Secret | Used by | Description |
|---|---|---|
| `SYNC_TOKEN` | all workflows | PAT with `repo` scope (and org repo-create permission for `add-submodules`). |
| `WEBLATE_URL` | `start-translation` | Weblate instance URL. |
| `WEBLATE_TOKEN` | `start-translation` | Weblate API token. |
| `SYNC_TOKEN` | all workflows | PAT with **`repo`** scope; **`add-submodules`** also needs permission to create org repositories when creating new mirrors. |
| `WEBLATE_URL` | `start-translation` | Base URL of the Weblate instance (the workflow appends **`boost-endpoint/add-or-update/`**). |
| `WEBLATE_TOKEN` | `start-translation` | API token for that endpoint. |

## Repository variables

| Variable | Used by | Description |
|---|---|---|
| `LANG_CODES` | `add-submodules`, `start-translation` | Optional. Default comma-separated language codes (e.g. `zh_Hans,ja`). |
| `LANG_CODES` | `add-submodules`, `start-translation` | Default language codes when **`client_payload.lang_codes`** is omitted (comma- or bracket-list, e.g. `zh_Hans,ja`). Must be set here or passed in the dispatch payload. |
| `SUBMODULES_ORG` | `add-submodules`, `start-translation` | Optional. GitHub org for **`boostorg`** mirror repos (e.g. `CppDigest`). If unset, the org is the same as this repository’s owner. **`sync-translation`** relies on **`.gitmodules`** URLs already pointing at the correct hosts. |

---
181 changes: 181 additions & 0 deletions docs/ARCHITECTURE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,181 @@
# Architecture — boost-docs-translation

This document describes **why** the system is shaped the way it is, **how** the main
pieces relate, and **which** concerns cut across workflows. For triggers, secrets,
and copy-paste examples, see [README.md](../README.md).

---

## 1. Problem and goals

**Context.** Boost ships documentation inside many separate **`boostorg`** repositories.
Translation work needs stable, narrow remotes (doc-only trees), language-specific
branches, and automation that does not clobber in-progress translator submissions.

**Goals.**

- **Mirror** each relevant library’s documentation into a dedicated GitHub repo under
a configurable org (**`MODULE_ORG`**, from **`SUBMODULES_ORG`**).
- **Super-repo** (**this repository**): one checkout that pins exact commits of every
mirrored lib via **`libs/<name>`** submodules on **`master`** and on
**`local-{lang_code}`** branches.
- **Upstream sync**: refresh mirror **`master`** from **`boostorg`** at a chosen ref,
then fold changes into **`local-*`** when safe.
- **Weblate handoff**: after successful git updates, tell the translation server which
org, version, languages, and components changed so it can add or refresh projects
asynchronously.
- **Consumer branches**: keep every **`local-*`** branch in the super-repo pointing at
the latest **`local-*`** tip of each submodule (scheduled **`sync-translation`**).

Non-goals: building Boost, serving rendered HTML from this repo, or replacing
Weblate’s internal project configuration beyond the **`add-or-update`** contract.

---

## 2. System context

```mermaid
flowchart LR
subgraph upstream [Upstream]
BO[boostorg repos]
BB[boostorg/boost .gitmodules]
end

subgraph ci [GitHub Actions]
AS[add-submodules]
ST[start-translation]
SY[sync-translation]
end

subgraph mirrors [Library mirrors MODULE_ORG]
M1["repo: algorithm"]
M2["repo: json"]
end

subgraph super [Translations super-repo]
TR[this repo master and local-*]
end

subgraph tw [Weblate]
EP["POST .../boost-endpoint/add-or-update/"]
end

BB --> AS
BO --> AS
BO --> ST
AS --> M1
AS --> M2
ST --> M1
ST --> M2
AS --> TR
ST --> TR
ST --> EP
M1 --> TR
M2 --> TR
SY --> TR
```

**Legend.** **`add-submodules`** may discover names from **`boostorg/boost`**;
**`start-translation`** discovers names from **this repo’s `.gitmodules`**. Both
mutate mirror repos and then update the super-repo. **`sync-translation`** only
moves submodule SHAs on existing **`local-*`** branches.

---

## 3. Repository layout (codemap)

| Path | Role |
|------|------|
| **`.gitmodules`** | Declares **`libs/<lib>`** → **`https://github.com/{MODULE_ORG}/{lib}.git`**, branch **`master`** for default checkout metadata. |
| **`.github/workflows/add-submodules.yml`** | On dispatch: create missing mirrors, push **`master`** / **`local-*`**, install **`create-tag.yml`**, record submodules in the super-repo. |
| **`.github/workflows/start-translation.yml`** | On dispatch: sync existing mirrors, merge policy on **`local-*`**, update super-repo pointers, call Weblate. |
| **`.github/workflows/sync-translation.yml`** | On dispatch or schedule: for each **`local-*`**, **`submodule update --remote`** and force-push. |
| **`.github/workflows/assets/env.sh`** | Derives **`ORG`**, **`TRANSLATIONS_REPO`**, **`MODULE_ORG`**, **`BOOST_ORG`**, **`MASTER_BRANCH`**, bot identity. |
| **`.github/workflows/assets/lib.sh`** | Shared implementation: GitHub **`gh`** helpers, clone/prune, **`meta/libraries.json`** parsing, translations-repo branch and submodule updates. |
| **`.github/workflows/assets/create-tag.yml`** | Template copied into each mirror; tags merged Weblate PRs (see **assets/README.md**). |
| **`scripts/trigger-*.sh`** | Optional local wrappers around **`repository_dispatch`**; no server-side logic. |

**Dependency direction.** Inline bash in **`add-submodules.yml`** and
**`start-translation.yml`** sources **`env.sh`** then **`lib.sh`**. **`sync-translation`**
sources only **`env.sh`** (git-only loop, no **`lib.sh`**).

---

## 4. Branch model

| Branch | Where | Meaning |
|--------|--------|---------|
| **`master`** | Mirror and super-repo | Doc-only content aligned with upstream English sources for the synced ref. |
| **`local-{lang_code}`** | Mirror and super-repo | Translation line for **`lang_code`**; Weblate opens PRs from **`translation-{lang_code}-*`** into this branch on mirrors. |
| **`translation-{lang_code}-{version}`** | Mirror (head of PR) | Weblate working branch naming assumed by **`start-translation`** (open PR guard) and **`create-tag.yml`** (tag extraction). |

**Super-repo `local-*` vs mirror `local-*`.** The super-repo’s **`local-*`** branch
commits **submodule pointers** that track each mirror’s **`local-*`** (after
**`finalize_translations_repo`** / **`sync-translation`**). The mirror’s **`local-*`**
holds **actual file** merges from **`master`** plus translator edits.

---

## 5. Control and data flows

### 5.1 Add mirrors (**`add-submodules`**)

1. Resolve library names: **`client_payload.submodules`** or **`boostorg/boost`**
**`.gitmodules`** at **`LIBS_REF`**.
2. For each name: **`get_doc_paths`** ( **`meta/libraries.json`** ) → clone
**`boostorg/{name}`** → **`prune_to_doc_only`**.
3. If mirror missing: **`gh repo create`**, init from pruned tree, push **`master`**,
branch **`local-*`** with **`create-tag.yml`** committed on each.
4. Clone super-repo to a temp dir; **`ensure_local_branch_in_translations`** per
language; for each updated lib, **`update_translations_submodule`** on **`master`**
and each **`local-*`**; push ( **`local-*`** force-pushed in **`finalize_translations_repo`** ).

### 5.2 Sync and notify (**`start-translation`**)

1. Submodule names from **this** repo **`.gitmodules`** (**`libs/`** only).
2. Per lib: clone upstream and mirror, replace mirror **`master`** tree with pruned
upstream snapshot, push.
3. Per **`lang_code`**: if **`local-*`** exists and an open PR matches
**`translation-{lang_code}-*`** → skip merge for that pair; else create branch or
**`merge origin/master`** into **`local-*`** and push.
4. **`finalize_translations_repo`**: refresh **`UPDATES`** submodules on **`master`**
(normal push) and on each **`local-*`** (**`--force`**).
5. Build **`add_or_update`** map only for languages that had at least one successful
update; **`jq`**-construct JSON; **POST** to Weblate **`boost-endpoint`**; tolerate
**202** (async) or **200**.

### 5.3 Pointer roll-up (**`sync-translation`**)

1. List **`refs/heads/local-*`** on the super-repo.
2. For each branch: checkout, **`git submodule update --init`**, set
**`submodule.<path>.branch`**, **`submodule update --remote`**, commit, **`push --force`**.

---

## 6. Cross-cutting concerns

**Authentication.** **`SYNC_TOKEN`** is passed to **`actions/checkout`** and
**`GITHUB_TOKEN`** for **`gh`** and **`gh auth setup-git`**, so pushes and API calls
share one credential. **`WEBLATE_TOKEN`** is only used for the outbound HTTP call.

**Idempotency and safety.**

- **`add-submodules`** skips if the mirror repo already exists.
- **`start-translation`** skips mirror **`master`** merge into **`local-*`** when a
Weblate PR is open, reducing race risk with translators.
- **`create-tag`** skips if the derived tag already exists.

**Observability.** Shell steps echo progress to Actions logs; Weblate response bodies
are printed for **`start-translation`** on success or failure paths where implemented.

**Configuration surface.** **`env.sh`** centralizes org and branch constants; workflow
YAML supplies **`LIBS_REF`**, **`LANG_CODES`**, **`EXTENSIONS`**, and secrets so
behavior is traceable without reading the entire inline script.

---

## 7. Stability of this document

Update this file when **branch naming**, **Weblate contract**, **ownership model**
(**`MODULE_ORG`**), or **repository layout** changes. Routine secret rotation or
version bumps in **`actions/checkout`** alone do not require edits here.