From a14d24298779f60b85fd6b07bf7d7c46de686f9f Mon Sep 17 00:00:00 2001 From: AuraMindNest Date: Thu, 21 May 2026 20:13:12 -0600 Subject: [PATCH 1/3] Add Boost Endpoint API Documentation --- docs/boost-endpoint-api.md | 394 +++++++++++++++++++++++++++++++++++++ 1 file changed, 394 insertions(+) create mode 100644 docs/boost-endpoint-api.md diff --git a/docs/boost-endpoint-api.md b/docs/boost-endpoint-api.md new file mode 100644 index 0000000..b5e88e7 --- /dev/null +++ b/docs/boost-endpoint-api.md @@ -0,0 +1,394 @@ + + +# Boost Endpoint API + +The Boost Endpoint is the HTTP API surface of this plugin. It provides three routes mounted under `/boost-endpoint/` on the Weblate site and exposes one asynchronous operation — `add-or-update` — for bulk creation and maintenance of Weblate projects and components from Boost C++ library submodule repositories. + +## Contents + +- [Installation and registration](#installation-and-registration) +- [Authentication](#authentication) +- [Endpoints](#endpoints) + - [GET /boost-endpoint/plugin-ping/](#get-boost-endpointplugin-ping) + - [GET /boost-endpoint/info/](#get-boost-endpointinfo) + - [POST /boost-endpoint/add-or-update/](#post-boost-endpointadd-or-update) +- [Request reference](#request-reference) +- [Response reference](#response-reference) +- [Async execution model](#async-execution-model) +- [BoostComponentService internals](#boostcomponentservice-internals) +- [Error handling](#error-handling) +- [Component naming](#component-naming) + +--- + +## Installation and registration + +Adding the app to `INSTALLED_APPS` is required but not sufficient for routes to be active. Weblate's `urls.py` builds its route list by hand (`real_patterns`) and does not auto-discover URLconfs from arbitrary apps. + +`BoostEndpointConfig.ready()` (`src/boost_weblate/endpoint/apps.py`) appends to `weblate.urls.real_patterns` at Django startup: + +```python +wl_urls.real_patterns.append( + path( + "boost-endpoint/", + include(("boost_weblate.endpoint.urls", "boost_endpoint")), + ), +) +``` + +This is idempotent — a module-level flag (`_cppa_boost_weblate_urls_registered`) prevents double-registration. The routes inherit Weblate's `URL_PREFIX` handling because `real_patterns` is processed before the prefix wrapper is applied. + +For `INSTALLED_APPS` registration, use `settings_override.py` (recommended) or the `WEBLATE_ADD_APPS` Docker environment variable — **not both**. See the main [README](../README.md#weblate_add_apps) for the full comparison. + +--- + +## Authentication + +All endpoints except `plugin-ping` require an authenticated Weblate session or token. The API uses Django REST Framework's standard `IsAuthenticated` permission class. + +| Endpoint | Authentication | +|----------|---------------| +| `GET /boost-endpoint/plugin-ping/` | None | +| `GET /boost-endpoint/info/` | Required | +| `POST /boost-endpoint/add-or-update/` | Required | + +Unauthenticated requests to protected endpoints receive `HTTP 401 Unauthorized`. + +The `add-or-update` endpoint also checks object-level permissions inside the Celery worker: + +- `project.add` — required to create a new Weblate project. +- `project.edit` — required to modify components in an existing project. +- `translation.add` — required to add a language to a component. +- `translation.add_more` — if absent, the language must be in the project's allowed set rather than any globally available language. + +--- + +## Endpoints + +### GET /boost-endpoint/plugin-ping/ + +Minimal health check. Returns a plain-text `ok` string. No authentication required. Useful for smoke-testing that the URL registration succeeded. + +**Request** + +``` +GET /boost-endpoint/plugin-ping/ +``` + +**Response** + +``` +HTTP/1.1 200 OK +Content-Type: text/plain + +ok +``` + +--- + +### GET /boost-endpoint/info/ + +Returns metadata about the installed plugin: package name, version, and the list of supported capability strings. + +**Request** + +``` +GET /boost-endpoint/info/ +Authorization: Token +``` + +**Response** + +``` +HTTP/1.1 200 OK +Content-Type: application/json + +{ + "module": "cppa-weblate-plugin", + "version": "0.1.0", + "capabilities": ["info", "add-or-update"] +} +``` + +| Field | Type | Description | +|-------|------|-------------| +| `module` | string | PyPI package name (`cppa-weblate-plugin`) | +| `version` | string | Installed package version from `importlib.metadata`; falls back to `"0.0.0"` if metadata is not found | +| `capabilities` | array of strings | Fixed list of supported endpoint names | + +--- + +### POST /boost-endpoint/add-or-update/ + +Creates or updates Weblate projects and components for one or more Boost library submodules, for one or more target languages. Heavy work runs in a Celery worker; the view returns `HTTP 202 Accepted` immediately with a `task_id`. + +**Request** + +``` +POST /boost-endpoint/add-or-update/ +Authorization: Token +Content-Type: application/json +``` + +See [Request reference](#request-reference) for the full body schema. + +**Response (202 Accepted)** + +```json +{ + "status": "accepted", + "task_id": "d3b07384-d9a2-4f9b-a0cf-1234567890ab", + "detail": "Boost add-or-update is running in the background; check Celery logs or task result for completion." +} +``` + +**Response (400 Bad Request)** + +```json +{ + "errors": { + "organization": ["This field is required."], + "add_or_update": {"": ["This field is required."]} + } +} +``` + +--- + +## Request reference + +`POST /boost-endpoint/add-or-update/` accepts a JSON body validated by `AddOrUpdateRequestSerializer`. + +### Fields + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `organization` | string | Yes | GitHub organization that owns the Boost submodule repositories (e.g. `"boostorg"`) | +| `version` | string | Yes | Boost release tag used as the branch name for cloning and as the translation push branch suffix (e.g. `"boost-1.90.0"`) | +| `add_or_update` | object | Yes | Map of language code → list of submodule names. Each key must be a non-empty string BCP-47 language code. Each value must be a non-empty list of repository/submodule name strings. | +| `extensions` | array of strings | No | File extensions to restrict scanning to (e.g. `[".adoc", ".md"]`). Only extensions that are also supported by Weblate's `FILE_FORMATS` are effective. If omitted, `null`, or empty, all Weblate-supported extensions are used. | + +### Validation rules + +- `organization`: must be a non-empty string. +- `version`: must be a non-empty string. +- `add_or_update`: must be a non-empty object. Each key must be a non-empty language code string. Each value must be a non-empty list of submodule name strings. +- `extensions`: optional. Blank-stripped entries that reduce to empty strings are removed silently. An all-blank list is treated as no filter (same as omitting the field). + +### Example request + +```json +{ + "organization": "boostorg", + "version": "boost-1.90.0", + "add_or_update": { + "zh_Hans": ["json", "unordered"], + "ja": ["json"] + }, + "extensions": [".adoc", ".md"] +} +``` + +This processes the `json` and `unordered` submodules for Simplified Chinese, and only `json` for Japanese, restricting scanned files to AsciiDoc and Markdown. + +--- + +## Response reference + +### 202 Accepted + +| Field | Type | Description | +|-------|------|-------------| +| `status` | string | Always `"accepted"` | +| `task_id` | string | Celery task UUID; use to query task state via Weblate's Celery result backend or monitoring tools | +| `detail` | string | Human-readable message describing background execution | + +### 400 Bad Request + +| Field | Type | Description | +|-------|------|-------------| +| `errors` | object | DRF serializer error map: field name → list of error strings | + +### 401 Unauthorized + +Standard DRF 401 response when no valid authentication credentials are provided. + +### Celery task result shape + +The Celery task (`boost_add_or_update_task`) returns a dictionary keyed by language code. Each value is the output of `BoostComponentService.process_all()` for that language: + +```json +{ + "zh_Hans": { + "total_submodules": 2, + "successful": 2, + "failed": 0, + "submodule_results": [ + { + "submodule": "json", + "success": true, + "components_created": 4, + "components_updated": 0, + "components_failed": 0, + "components_deleted": 0, + "errors": [] + }, + { + "submodule": "unordered", + "success": true, + "components_created": 2, + "components_updated": 1, + "components_failed": 0, + "components_deleted": 0, + "errors": [] + } + ] + }, + "ja": { ... } +} +``` + +#### `process_all` result fields + +| Field | Type | Description | +|-------|------|-------------| +| `total_submodules` | integer | Number of submodules submitted for this language | +| `successful` | integer | Submodules where at least one component was created or updated | +| `failed` | integer | Submodules where every component failed or the clone failed | +| `submodule_results` | array | Per-submodule result objects (see below) | + +#### Per-submodule result fields + +| Field | Type | Description | +|-------|------|-------------| +| `submodule` | string | Submodule name | +| `success` | boolean | `true` if at least one component was created or updated | +| `components_created` | integer | New components added to Weblate | +| `components_updated` | integer | Existing components whose push branch was refreshed | +| `components_failed` | integer | Components where `create_or_update_component` returned `None` | +| `components_deleted` | integer | Components removed because they were no longer found in the repo scan | +| `errors` | array of strings | Non-fatal error messages (clone failure, permission denial, git errors) | + +--- + +## Async execution model + +``` +POST /boost-endpoint/add-or-update/ + │ + ▼ +AddOrUpdateView.post() + Deserialize + validate → AddOrUpdateRequestSerializer + │ valid + ▼ +boost_add_or_update_task.delay( + organization, add_or_update, version, extensions, user_id +) + │ │ + │ HTTP 202 + task_id │ (Celery worker picks up) + ◄───────────────────── ▼ + for each lang_code, submodule_list + in add_or_update.items(): + BoostComponentService(...).process_all( + submodule_list, user, request + ) + return dict[lang_code → process_all result] +``` + +The task uses Weblate's own Celery `app` instance (`weblate.utils.celery.app`) and runs inside the same worker pool as all other Weblate background tasks. No additional broker configuration is needed beyond a working Weblate Celery setup. + +`user_id` (an integer primary key) is passed rather than the user object itself because Celery serializes task arguments to JSON. The task re-fetches the user with `User.objects.get(pk=user_id)` inside the worker. + +Exceptions raised by `BoostComponentService` propagate out of the task function unhandled, causing Celery to mark the task `FAILURE`. Per-submodule errors that are recoverable (e.g. clone failure, permission denial for a single component) are collected into the `errors` list and do not raise exceptions. + +`trail=False` is set on the task to suppress Celery's default task-result trail and avoid unbounded result-backend growth in long-running deployments. + +--- + +## BoostComponentService internals + +`BoostComponentService` (`src/boost_weblate/endpoint/services.py`) performs all the heavy work for a single language. It is instantiated once per language code by the Celery task. + +### Constructor parameters + +| Parameter | Type | Description | +|-----------|------|-------------| +| `organization` | string | GitHub organization name | +| `lang_code` | string | BCP-47 language code for this run | +| `version` | string | Boost release tag (used in branch and push-branch names) | +| `extensions` | `list[str] \| None` | Extension filter; `None` means no filtering | + +### `process_all(submodules, user, request)` + +Top-level entry point called by the Celery task. Creates a temporary directory, processes each submodule in sequence, cleans up the temp directory in a `finally` block, and returns the aggregated result dictionary. + +### Per-submodule pipeline (`process_submodule`) + +For each submodule the following steps run in order: + +1. **Path validation** — the submodule name is checked against the temp directory root to prevent path traversal. + +2. **Clone** — `git clone -b local-{lang_code} --depth 1 https://github.com/{organization}/{submodule}.git` into a temporary subdirectory. A 300-second timeout applies. Clone failure is recorded in `errors` and processing stops for that submodule. + +3. **Scan** — `scan_documentation_files()` walks the cloned tree, skipping hidden directories, `__pycache__`, and `node_modules`. Files at the repository root are skipped (only files inside subdirectories are included). Files whose stem ends with `_{lang_code}` (existing translation files) are excluded. Each remaining file with a Weblate-supported extension (filtered by `self.extensions` when set) produces a component config dict. + +4. **Permission check** — before touching the database, the user's `project.add` or `project.edit` permission is verified against the existing or prospective project. Failure records an error and stops the submodule. + +5. **Project get-or-create** — `Project.objects.get_or_create(slug=...)`. The project slug is `boost-{submodule}-documentation-{lang_code}`. On creation, `project.post_create(user, billing=None)` is called to match the REST API path. + +6. **Component get-or-create** (per scanned file) — `Component.objects.get_or_create(project=project, slug=...)`. The first component in a project uses the real SSH remote (`git@github.com:{org}/{submodule}.git`) as its `repo`; subsequent components link to the first via `weblate://{project_slug}/{owner_slug}` to share a single clone. On creation, `component.post_create(user, origin="boost_endpoint")` runs, followed by `_sync_component_for_translation()`. + +7. **Sync** — for new repo-owner components, `component.sync_git_repo(skip_push=True)` performs the initial clone. For existing components, a targeted git update (fetch + merge/rebase) runs via `_do_update_git_only()` without a full `do_update`. Both paths finish with `component.create_translations_immediate(force=True)` to ensure translation files are on disk before the language is added. + +8. **Language add** — `component.add_new_language(language, request)` following the same permission and availability checks as the Weblate REST API (`translation.add`, `translation.add_more`, `can_add_new_language`). + +9. **Stale component deletion** — components in the project whose slugs are not in the set of scanned configs are deleted. Glossary components (`is_glossary=True`) are never deleted. For each deletion: translation files are removed from disk, staged with `git add`, committed, and pushed to `origin HEAD:{push_branch}`. + +### Component configuration + +| Setting | Value | +|---------|-------| +| `vcs` | `"github"` | +| `branch` | `"local-{lang_code}"` | +| `push_branch` | `"translation-{lang_code}-{version}"` | +| `source_language` | English (`"en"`) | +| `language_regex` | `"^{lang_code}$"` (one language per component) | +| `allow_translation_propagation` | `False` | +| `edit_template` | `False` | +| `manage_units` | `False` | +| `enable_suggestions` | `True` | + +--- + +## Error handling + +The service uses a non-fatal error collection strategy: individual failures are appended to the `errors` list in the submodule result and processing continues with the next item. The `success` flag is `false` only when every component in the submodule failed or the clone itself failed. + +Exceptions that escape `process_submodule` or `process_all` propagate to the Celery task, which lets them surface as a `FAILURE` state so monitoring systems can alert. Internal exceptions within `create_or_update_component`, `add_language_to_component`, and `_delete_component_and_commit_removal` are caught, logged via Weblate's `LOGGER`, reported to Weblate's error-tracking system via `report_error()`, and reflected as incremented failure counters or appended error strings. + +--- + +## Component naming + +Component names and slugs are derived from the relative file path within the cloned repository. + +**Name** — directory parts are joined with ` / `, each part title-cased with underscores and hyphens replaced by spaces, and the file extension (without the leading dot) is appended in parentheses: + +``` +doc/html/intro.adoc → "Doc / Html / Intro (adoc)" +``` + +**Slug** — directory and file parts are lowercased with underscores replaced by hyphens, joined with `-`, and the extension is appended: + +``` +doc/html/intro.adoc → "doc-html-intro-adoc" +``` + +If a name or slug exceeds Weblate's `COMPONENT_NAME_LENGTH` limit (100 characters), it is truncated with a hash suffix to guarantee uniqueness: + +- **Name**: keep first `(max_len - 10)` characters, append `[{8-hex}]` derived from SHA-256 of the full name. +- **Slug**: keep first `(max_len - 9)` characters, append `-{8-hex}` derived from SHA-256 of the full slug. From e1c3f6c2e88b88a6dd754da4a32b7492c77ac7a3 Mon Sep 17 00:00:00 2001 From: AuraMindNest Date: Thu, 21 May 2026 20:13:51 -0600 Subject: [PATCH 2/3] =?UTF-8?q?Update=20README=20=E2=80=94=20WEBLATE=5FADD?= =?UTF-8?q?=5FAPPS,=20Routes,=20and=20Celery?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- README.md | 165 +++++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 158 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 5f95bc7..b19f9e6 100644 --- a/README.md +++ b/README.md @@ -18,7 +18,7 @@ SPDX-License-Identifier: BSL-1.0 | ---------- | ------ | -------- | | QuickBook | `boost_weblate.formats.quickbook` | Implemented | -Additional formats should follow the same split: a thin class under `src/boost_weblate/formats/` that plugs into Weblate’s format APIs, with parsing and reconstruction under `src/boost_weblate/utils/`. +Additional formats should follow the same split: a thin class under `src/boost_weblate/formats/` that plugs into Weblate's format APIs, with parsing and reconstruction under `src/boost_weblate/utils/`. ## Quickstart @@ -69,37 +69,43 @@ prek install ## Architecture -Weblate discovers formats by **import path** (see [WEBLATE_FORMATS config](#weblate_formats-configuration)). This repository keeps a clear boundary between “what Weblate sees” and “how a file format works.” +Weblate discovers formats by **import path** (see [WEBLATE_FORMATS config](#weblate_formats-configuration)). This repository keeps a clear boundary between "what Weblate sees" and "how a file format works." ```mermaid flowchart TB subgraph weblate["Weblate"] WF["WEBLATE_FORMATS"] CF["ConvertFormat / store"] + RP["real_patterns (URL list)"] end subgraph plugin["boost_weblate"] FMT["formats/ — format adapters"] UTL["utils/ — parse & serialize"] + EP["endpoint/ — HTTP API + Celery"] TST["tests/ — mirrors src layout"] end WF --> FMT FMT --> CF FMT --> UTL + EP -->|AppConfig.ready()| RP TST -.-> FMT TST -.-> UTL + TST -.-> EP ``` -- **`src/boost_weblate/formats/`** — Weblate-facing **format classes** (subclasses of Weblate’s `BaseFormat` family, such as `weblate.formats.convert.ConvertFormat`). `QuickBookFormat` follows the same pattern as built-in convert formats (for example AsciiDoc): it turns a template file into a translation store and, on save, applies translations back using the template plus the store. +- **`src/boost_weblate/formats/`** — Weblate-facing **format classes** (subclasses of Weblate's `BaseFormat` family, such as `weblate.formats.convert.ConvertFormat`). `QuickBookFormat` follows the same pattern as built-in convert formats (for example AsciiDoc): it turns a template file into a translation store and, on save, applies translations back using the template plus the store. - **`src/boost_weblate/utils/`** — **Format-specific logic** with no Weblate import cycle: QuickBook parsing, segment extraction, translate-toolkit storage (`QuickBookFile` / `QuickBookUnit`), and reconstruction (`QuickBookTranslator`). New formats should add a sibling module (or package) here. -- **`tests/`** — **Pytest** layout mirrors `src/boost_weblate/` (`tests/formats/`, `tests/utils/`, `tests/endpoint/`). Shared fixtures live under `tests/fixtures/`. `tests/conftest.py` configures `sys.path`, sets `DJANGO_SETTINGS_MODULE` to `tests.django_qbk_format_settings`, and calls `django.setup()` so format tests can load Weblate’s Django stack without requiring PostgreSQL. +- **`src/boost_weblate/endpoint/`** — **HTTP API** for Boost documentation project/component management. Exposes three routes under `/boost-endpoint/` (see [Routes](#routes)), uses Django REST Framework for auth and serialization, and hands off heavy work to a Celery task (see [Celery task](#celery-task)). + +- **`tests/`** — **Pytest** layout mirrors `src/boost_weblate/` (`tests/formats/`, `tests/utils/`, `tests/endpoint/`). Shared fixtures live under `tests/fixtures/`. `tests/conftest.py` configures `sys.path`, sets `DJANGO_SETTINGS_MODULE` to `tests.django_qbk_format_settings`, and calls `django.setup()` so format tests can load Weblate's Django stack without requiring PostgreSQL. ## WEBLATE_FORMATS configuration Weblate discovers formats from the `WEBLATE_FORMATS` setting (see `FileFormatLoader` in upstream `weblate.formats.models`). The official Docker image evaluates a single optional file after base settings: if `/app/data/settings-override.py` exists, it is compiled and executed with `exec()` in the **same namespace** as the rest of `weblate.settings_docker`. -Stock `weblate.settings_docker` does **not** always bind `WEBLATE_FORMATS` in that namespace before the hook runs, so a bare `WEBLATE_FORMATS += (...)` in the override can raise `NameError`. This repository ships ``src/boost_weblate/settings_override.py`` as the Docker ``exec()`` fragment: it assigns ``WEBLATE_FORMATS`` by **reading** upstream ``weblate/formats/models.py`` and regex-slicing ``FormatsConf.FORMATS`` (aligned with the installed Weblate version without importing ``weblate.formats.models`` during settings load, which can raise ``AppRegistryNotReady``). It appends the endpoint Django app via ``INSTALLED_APPS += ("boost_weblate.endpoint.apps.BoostEndpointConfig",)``. If you also set ``WEBLATE_ADD_APPS`` to the same app, remove one source to avoid duplicate ``INSTALLED_APPS`` entries. +Stock `weblate.settings_docker` does **not** always bind `WEBLATE_FORMATS` in that namespace before the hook runs, so a bare `WEBLATE_FORMATS += (...)` in the override can raise `NameError`. This repository ships `src/boost_weblate/settings_override.py` as the Docker `exec()` fragment: it assigns `WEBLATE_FORMATS` by **reading** upstream `weblate/formats/models.py` and regex-slicing `FormatsConf.FORMATS` (aligned with the installed Weblate version, without importing `weblate.formats.models` during settings load, which can raise `AppRegistryNotReady`). It also appends the endpoint Django app to `INSTALLED_APPS` — see [`WEBLATE_ADD_APPS`](#weblate_add_apps) below. **Operators:** ensure the plugin package is installed in the Weblate environment (`pip` / image layer), then install the override file where Weblate expects it. For the stock Docker layout: @@ -109,7 +115,152 @@ COPY settings-override.py /app/data/settings-override.py That path is fixed; Weblate does not scan `DATA_DIR` for arbitrary override files. The override file is **not** the same as `WEBLATE_PY_PATH` / `python/customize` (importable customization on `sys.path`); for format registration, use this exec hook unless your image explicitly imports another settings module. See the comments in `settings_override.py` for the full distinction. -**Adding another format:** implement the class under `boost_weblate/formats/`, append its dotted class path in ``weblate_formats_with_quickbook()`` (or extend the tuple built there), redeploy, and restart Weblate. If upstream changes the layout of ``FormatsConf`` in ``models.py``, update the regex in ``settings_override.py`` accordingly. +**Adding another format:** implement the class under `boost_weblate/formats/`, append its dotted class path in `weblate_formats_with_quickbook()` (or extend the tuple built there), redeploy, and restart Weblate. If upstream changes the layout of `FormatsConf` in `models.py`, update the regex in `settings_override.py` accordingly. + +## WEBLATE_ADD_APPS + +`WEBLATE_ADD_APPS` is a Weblate Docker environment variable that appends entries to `INSTALLED_APPS` before the container starts (handled by Weblate's own Docker entrypoint, not by this plugin). + +This plugin registers the endpoint Django app in `settings_override.py` directly: + +```python +# excerpt from src/boost_weblate/settings_override.py +_INSTALLED_APPS = globals().get("INSTALLED_APPS") +if _INSTALLED_APPS is not None: + if isinstance(_INSTALLED_APPS, tuple): + globals()["INSTALLED_APPS"] = _INSTALLED_APPS + (_ENDPOINT_APP_CONFIG,) + else: + _INSTALLED_APPS += (_ENDPOINT_APP_CONFIG,) +``` + +where `_ENDPOINT_APP_CONFIG = "boost_weblate.endpoint.apps.BoostEndpointConfig"`. + +**Two approaches — pick one, not both:** + +| Approach | How it works | When to use | +|----------|-------------|-------------| +| `settings_override.py` (this repo) | `exec()`'d fragment appends to `INSTALLED_APPS` directly and also sets `WEBLATE_FORMATS` | Recommended — one file covers both format registration and app installation | +| `WEBLATE_ADD_APPS` env var | Weblate Docker entrypoint adds to `INSTALLED_APPS` before Django starts | Use only if you are not deploying `settings_override.py` at all | + +> **Important:** if you set `WEBLATE_ADD_APPS=boost_weblate.endpoint.apps.BoostEndpointConfig` **and** deploy `settings_override.py`, the app will be added to `INSTALLED_APPS` twice, which raises a `django.core.exceptions.ImproperlyConfigured` error at startup. Remove one source. + +Note that adding the app to `INSTALLED_APPS` (by either method) is **necessary but not sufficient** for HTTP routes to be active — see [Routes](#routes) below for why. + +## Routes + +The plugin exposes three HTTP endpoints, all under the `/boost-endpoint/` prefix on the Weblate site: + +| Method | Path | Handler | Auth | Response | +|--------|------|---------|------|----------| +| `GET` | `/boost-endpoint/plugin-ping/` | `plugin_ping` | None | `200 ok` (plain text) | +| `GET` | `/boost-endpoint/info/` | `BoostEndpointInfo` | Required | `200` JSON: `module`, `version`, `capabilities` | +| `POST` | `/boost-endpoint/add-or-update/` | `AddOrUpdateView` | Required | `202` JSON: `status`, `task_id`, `detail` | + +### Why routes need explicit registration + +Weblate's `urls.py` does **not** auto-discover URLconfs from arbitrary `INSTALLED_APPS` entries. It builds a single `real_patterns` list by hand and only extends it for known built-in apps (legal, SAML, git-export, etc.) via explicit `if "app" in settings.INSTALLED_APPS:` guards — there is no generic plugin scan. + +This plugin handles registration in `BoostEndpointConfig.ready()` (`src/boost_weblate/endpoint/apps.py`), which runs once at Django startup and appends to `weblate.urls.real_patterns`: + +```python +wl_urls.real_patterns.append( + path( + "boost-endpoint/", + include(("boost_weblate.endpoint.urls", "boost_endpoint")), + ), +) +``` + +The operation is idempotent (guarded by a `_cppa_boost_weblate_urls_registered` attribute on the module). Routes sit under Weblate's `URL_PREFIX` handling because `real_patterns` is used before the prefix wrapper is applied. + +### Request / response for `POST /boost-endpoint/add-or-update/` + +**Request body (JSON):** + +```json +{ + "organization": "boostorg", + "version": "boost-1.90.0", + "add_or_update": { + "zh_Hans": ["json", "unordered"], + "ja": ["json"] + }, + "extensions": [".adoc", ".md"] +} +``` + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `organization` | string | Yes | GitHub organization that owns the Boost submodule repos | +| `version` | string | Yes | Boost release tag, e.g. `"boost-1.90.0"` | +| `add_or_update` | object | Yes | Map of language code → list of submodule names (non-empty list per key) | +| `extensions` | array of strings | No | File extensions to scan (e.g. `[".adoc", ".md"]`); defaults to all Weblate-supported extensions | + +**Response (202 Accepted):** + +```json +{ + "status": "accepted", + "task_id": "d3b07384-d9a2-4f9b-a0cf-1234567890ab", + "detail": "Boost add-or-update is running in the background; check Celery logs or task result for completion." +} +``` + +The view validates the request with `AddOrUpdateRequestSerializer`, dispatches the Celery task, and returns immediately. A `400` response with an `errors` object is returned if validation fails. + +## Celery task + +Heavy work (git clone, file scanning, Weblate project/component create-or-update) runs asynchronously in a Celery worker via `boost_add_or_update_task` (`src/boost_weblate/endpoint/tasks.py`). The view enqueues the task with `.delay()` and returns HTTP 202 immediately. + +``` +POST /boost-endpoint/add-or-update/ + │ + ▼ +AddOrUpdateView.post() + Validate body → AddOrUpdateRequestSerializer + │ valid + ▼ +boost_add_or_update_task.delay( + organization, add_or_update, version, extensions, user_id +) + │ │ + │ HTTP 202 + task_id │ (worker picks up) + ◄─────────────────── ▼ + for each lang_code → submodule_list: + BoostComponentService(org, lang, version, extensions) + .process_all(submodules, user, request) + returns dict[lang_code → result] +``` + +**Task signature:** + +```python +@app.task(trail=False) +def boost_add_or_update_task( + *, + organization: str, + add_or_update: dict[str, list[str]], + version: str, + extensions: list[str] | None, + user_id: int, +) -> dict[str, Any]: +``` + +- Uses Weblate's own Celery `app` instance (`weblate.utils.celery.app`), so it runs in the same worker pool as all other Weblate tasks with no extra broker configuration. +- `user_id` is passed instead of the `User` object because Celery serializes task arguments to JSON; the task re-fetches the user from the database inside the worker. +- Exceptions propagate unhandled so Celery marks the task as `FAILURE` and monitoring/alerting can act on it. +- `trail=False` suppresses Celery's default task-result trail to avoid unbounded result-backend growth. + +**`BoostComponentService`** (`src/boost_weblate/endpoint/services.py`) performs the actual work for each language: + +1. Clone the GitHub submodule repository for the given organization, version, and language. +2. Scan the cloned tree for files matching the requested (or all supported) extensions. +3. Build Weblate `Project` and `Component` configurations from the scan results. +4. Call `get_or_create` on each `Project`/`Component` via the Weblate ORM; update existing ones. +5. Add the target language to each component via `add_new_language`. +6. Delete stale components no longer present in the scan, commit, and push. + +The service has no plugin-owned models; it operates entirely through Weblate's Django ORM. ## Contributing @@ -117,7 +268,7 @@ That path is fixed; Weblate does not scan `DATA_DIR` for arbitrary override file - **Tests:** add tests next to the code you touch (`tests/formats/`, `tests/utils/`, or `tests/endpoint/`). Keep `django.setup()`-friendly patterns; heavy DB or migration suites are intentionally avoided in the bundled Django test settings. -- **CI coverage:** the *Lint and format* workflow runs a **Tests and coverage** job that prints `term-missing` output, runs `coverage report`, writes `coverage.xml` and `htmlcov/`, and uploads those plus `.coverage` as a workflow artifact (download from the run’s *Artifacts* section on GitHub). Coverage is configured in `pyproject.toml` (`[tool.coverage.*]`); the job uses `uv sync --frozen --group dev --group pre-commit` so `pytest-cov` and `coverage[toml]` match the lockfile. +- **CI coverage:** the *Lint and format* workflow runs a **Tests and coverage** job that prints `term-missing` output, runs `coverage report`, writes `coverage.xml` and `htmlcov/`, and uploads those plus `.coverage` as a workflow artifact (download from the run's *Artifacts* section on GitHub). Coverage is configured in `pyproject.toml` (`[tool.coverage.*]`); the job uses `uv sync --frozen --group dev --group pre-commit` so `pytest-cov` and `coverage[toml]` match the lockfile. - **Pull requests:** open PRs against the default branch on GitHub. Keep changes focused; ensure CI is green (build/wheel checks, lint, tests). Respond to review feedback on the PR thread; for design questions or bug reports, use [Issues](https://github.com/cppalliance/cppa-weblate-plugin/issues). From fed34e96b3584a4de45944895b8db9af5fbf65a5 Mon Sep 17 00:00:00 2001 From: AuraMindNest Date: Thu, 21 May 2026 20:26:17 -0600 Subject: [PATCH 3/3] Update due to coderabbitai review --- README.md | 2 +- docs/boost-endpoint-api.md | 16 ++++++++-------- 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index b19f9e6..4d185f3 100644 --- a/README.md +++ b/README.md @@ -212,7 +212,7 @@ The view validates the request with `AddOrUpdateRequestSerializer`, dispatches t Heavy work (git clone, file scanning, Weblate project/component create-or-update) runs asynchronously in a Celery worker via `boost_add_or_update_task` (`src/boost_weblate/endpoint/tasks.py`). The view enqueues the task with `.delay()` and returns HTTP 202 immediately. -``` +```text POST /boost-endpoint/add-or-update/ │ ▼ diff --git a/docs/boost-endpoint-api.md b/docs/boost-endpoint-api.md index b5e88e7..6a60923 100644 --- a/docs/boost-endpoint-api.md +++ b/docs/boost-endpoint-api.md @@ -75,13 +75,13 @@ Minimal health check. Returns a plain-text `ok` string. No authentication requir **Request** -``` +```http GET /boost-endpoint/plugin-ping/ ``` **Response** -``` +```http HTTP/1.1 200 OK Content-Type: text/plain @@ -96,14 +96,14 @@ Returns metadata about the installed plugin: package name, version, and the list **Request** -``` +```http GET /boost-endpoint/info/ Authorization: Token ``` **Response** -``` +```http HTTP/1.1 200 OK Content-Type: application/json @@ -128,7 +128,7 @@ Creates or updates Weblate projects and components for one or more Boost library **Request** -``` +```http POST /boost-endpoint/add-or-update/ Authorization: Token Content-Type: application/json @@ -277,7 +277,7 @@ The Celery task (`boost_add_or_update_task`) returns a dictionary keyed by langu ## Async execution model -``` +```text POST /boost-endpoint/add-or-update/ │ ▼ @@ -378,13 +378,13 @@ Component names and slugs are derived from the relative file path within the clo **Name** — directory parts are joined with ` / `, each part title-cased with underscores and hyphens replaced by spaces, and the file extension (without the leading dot) is appended in parentheses: -``` +```text doc/html/intro.adoc → "Doc / Html / Intro (adoc)" ``` **Slug** — directory and file parts are lowercased with underscores replaced by hyphens, joined with `-`, and the extension is appended: -``` +```text doc/html/intro.adoc → "doc-html-intro-adoc" ```