docs: restructure and improve canonical documentation, with minimal Docusaurus setup and wiki cleanup

[nikazzio]

Context

The repository already has a strong documentation base, but the current documentation system is still difficult to navigate and inconsistent as a whole.

Main problems:

• canonical content exists, but the information architecture is weak;
• user guides, technical explanation, reference, and process documentation are mixed together;
• the GitHub Wiki currently creates a poor reading experience and often sends users to repository Markdown paths instead of a clean documentation flow;
• there is no proper documentation site yet;
• some terminology drift and content inconsistencies are already visible;
• README.md is good enough for now and should stay mostly untouched until the rest of the documentation system is stabilized.

This work should improve the documentation system with a content-first approach:

• better canonical content structure;
• minimal Docusaurus setup as the new documentation site;
• a much smaller and cleaner wiki;
• stronger validation in CI;
• explicit source-of-truth rules.

Scope

In scope:

• introduce a minimal Docusaurus setup as the canonical docs site;
• keep the docs work content-first, with Docusaurus acting as a publishing/navigation layer rather than the main focus of the effort;
• reorganize canonical docs into a simpler structure under intro, guides, reference, explanation, project, and ADRs;
• rewrite the wiki as a minimal entry layer;
• fix the wiki publishing/linking model so wiki pages point to the public docs site instead of awkward repo Markdown paths;
• add missing canonical documentation areas, especially CLI reference and core technical explanation pages;
• update CI to validate docs build and wiki link policy;
• leave README.md for a later final polish pass.

Out of scope:

• full multilingual rollout now;
• docs versioning rollout now;
• Backstage TechDocs;
• major README redesign in this phase.

Acceptance Criteria

1. A working minimal Docusaurus docs site exists.
2. Canonical docs are reorganized into a clearer structure.
3. The wiki is reduced to a minimal entry layer.
4. Wiki pages no longer depend on confusing links to repo Markdown files as their main navigation model.
5. Wiki sync rewrites canonical links to the public docs site and fails on non-publishable links.
6. CLI documentation exists and covers the current public CLI surface.
7. Core technical explanation pages exist for architecture, storage model, job lifecycle, discovery/provider behavior, and export/PDF behavior.
8. CI validates markdown, links, docs integrity, language policy, Docusaurus build, and wiki constraints.
9. README.md remains intentionally deferred to a later final pass.

Notes

Recommended branch: docs/content-first-docusaurus-minimal

This issue is intentionally content-first. The main success criterion is better documentation structure and clarity, not adopting tooling for its own sake.

[nikazzio]

Documentation refactor progress update:

Implemented so far:

• created the content-first docs branch: docs/content-first-docusaurus-minimal
• introduced a minimal Docusaurus documentation site as the canonical public docs surface
• reorganized canonical docs into intro, guides, reference, explanation, project, and adr
• reduced the GitHub Wiki to a small orientation layer
• updated wiki publishing so canonical links rewrite to the public docs site instead of repository Markdown blob URLs
• added GitHub Pages deployment workflow for the docs site
• updated docs CI to build the docs site and validate the new docs structure
• added ADRs for docs information architecture and docs site/wiki publishing model
• added a screenshot asset directory under docs/assets/screenshots/

Technical decision taken during implementation:

• rejected the experimental @docusaurus/faster / rspack path because it was not reliable in this environment
• removed the temporary webpack override workaround
• kept the stable solution based on pinned Docusaurus versions, committed package-lock.json, and npm ci in CI
• this keeps the docs toolchain deterministic without relying on experimental features or local patching

Current verification status:

• python3 scripts/check_docs_integrity.py passes
• python3 scripts/check_docs_language.py passes
• python3 scripts/sync_wiki.py --dry-run ... passes
• npm run docs:build passes

README remains intentionally deferred to a later pass, as planned.