β¨π§π‘ Proposal: evolving sphinx-needs into a monorepo #1803
Replies: 5 comments
|
My 5 cents:
|
Progress update, as of the 8.5.0 releaseThe groundwork phase of this proposal is done. Everything below is in Shipped
Decisions taken along the way (details in the linked PRs): a virtual workspace root with Next
Open threads that are not blocking: #1829 (nothing tests the built package), #1835 (the ty suppressions to burn down), #1725 (the |
Progress: the repository is now a uv workspaceMerged today: #1838 (a CI cell that builds the wheel, installs it into a throwaway Nothing about the published package changed (wheel verified file-for-file against 8.5.0's). A first Claude Code cloud session was pointed at the branch cold and ran every task from Next: the Python floor moves to 3.11 across the suite (sphinx-needs up from 3.10, the |
Progress: the release pipeline, and the first importMerged since the last update, all on
What the first import taught. The shared dependency groups, the one ruff/ty/pytest configuration and the one lock worked as designed, and the workspace fence now also refuses per-member Next: those two items, then sphinx-codelinks: its preparatory PRs first (Python floor to 3.11, |
Progress: sphinx-codelinks is in, and the workspace has one PlantUMLMerged since the last update, all on
Next: sphinx-test-reports (the last of the three), then the first releases from the workspace: sphinx-mounts 0.3.0 independently, sphinx-needs 9.0.0 (the 3.11 floor is the breaking change), and sphinx-codelinks after it with its floor moved to |
Uh oh!
There was an error while loading. Please reload this page.
Summary
We are considering restructuring this repository into a monorepo that can also host
sibling extensions from the useblocks org. The initial candidates are:
β though neither the list nor the timing is fixed: the restructuring is useful for
sphinx-needs on its own, extensions can move over one at a time as they are ready
(sphinx-test-reports, for example, needs the most modernisation and may follow later
without blocking anything), and other extensions could join the same way in future.
To be clear up front: this is about where the code lives and how it is developed and
tested β not about merging the packages themselves.
We've done a fairly deep investigation of the mechanics (uv workspaces, testing matrices,
Read the Docs, releases, history migration) including hands-on verification of the
sharp edges, and studied how comparable projects run the same shape. This post lays out
the motivation, the proposed design, and the open questions we'd like feedback on before
committing.
What changes for users β and what doesn't
Doesn't change: the PyPI packages, their names, their independent versions and
release cadences, how you install them, and the documentation content and URLs.
Changes: where issues and PRs live (here, with
package:labels), tag names, theextensions' sphinx-needs policy β a new sphinx-codelinks/mounts/test-reports release
will require the sphinx-needs version current at that release (question 2 below) β and
possibly the supported Python floor (question 1).
Why
These extensions are all tightly coupled to sphinx-needs, and the separate-repo setup
makes that coupling expensive:
sphinx-codelinks or sphinx-test-reports is only discovered when their CI next runs
against a released sphinx-needs. sphinx-mounts currently resorts to a documented
workaround β git-pinning
sphinx-needs@masterin its test dependencies β just to testagainst unreleased changes. In a monorepo, every sphinx-needs PR runs the extensions'
tests against that PR.
variant-data handling (documented in its source, kept behaviourally identical by hand)
rather than depending on it. In one repo, extracting genuinely shared code β its source
literally anticipates a shared
sphinx-variantspackage β becomes agit mvinsteadof a cross-repo coordination problem. (One caveat: the variant expression grammar is
also mirrored by ubCode's engine, so any extraction needs to treat that contract with
care.)
sphinx-codelinks store the origin of needs it creates require coordinated changes in
two repos today; in one repo they are a single PR whose tests cover both sides
immediately.
supported-version matrices. They have already diverged: Python floors are 3.10 in two
repos and 3.12 in the other two, ruff pins span three versions, sphinx-needs still
publishes with an API token while the other three use trusted publishing, and
sphinx-test-reports declares a
sphinx-needs>=1.0.1floor that is ~7 majors stale.One repo means one set of supported Sphinx/Python versions, tested together.
Proposed shape
Built on uv workspaces, which
we've verified against this exact topology (a core package + dependent extensions):
The mechanism that makes independent releases work: each extension declares a real
version range on sphinx-needs in
[project.dependencies]β that is exactly what shipsin its published wheel β plus a uv-only redirect that makes local development and CI use
the in-tree sphinx-needs:
uv sync --all-packagesgives every contributor (and every coding agent) all fourpackages editable in one command, and an edit in sphinx-needs is immediately visible to
the extensions' tests with no reinstall step.
Tooling consolidation
The four repos are really three generations of the same house style, and the merge is
the moment to converge on the newest one (which sphinx-codelinks and sphinx-mounts
already share):
much faster reimplementation, and its
workspace mode is what makes a multi-package repo
tractable: each package keeps its own
.pre-commit-config.yaml, unmodified, withhooks running relative to that package β no root config smeared with per-package
files:regexes. sphinx-mounts has already run prek in CI for months; every hookcurrently used across the four repos is supported.
byte-identical ruff configs; that becomes the shared root config. sphinx-needs and
sphinx-test-reports adopt it via ruff's
extendmechanism with temporary per-packageignores that get retired family-by-family (measured honestly: sphinx-needs is ~580
findings away from the shared ruleset, so this is a tracked backlog, not a big bang).
sphinx-mounts is already ty-only; for sphinx-needs, ty is ~80Γ faster than mypy but
still pre-1.0 with unstable diagnostics, and sphinx-codelinks depends on the
pydantic.mypyplugin which ty has no equivalent for yet. So: keep mypy blocking,add ty as a non-blocking job, revisit at ty 1.0.
--import-mode=importlibβ required, sincefour Sphinx-extension packages will inevitably have colliding
test_*.pybasenames).current tox / nox / Makefile mix. Tasks live in
[tool.poe.tasks]in the rootpyproject and run as
uv run poe β¦β nothing extra to install afteruv sync, andthe entry points (
poe test <package>,poe docs <package>,poe smoke <package>)are the same everywhere, for humans and coding agents alike.
Testing
A monorepo quietly inverts today's testing contract: sphinx-codelinks and
sphinx-test-reports currently prove compatibility against a range of released
sphinx-needs versions (36-combination matrices), while a monorepo tests everything
against the in-tree sphinx-needs. Rather than fight that, we propose to embrace it as
policy (see Releases: the extensions track the latest sphinx-needs). The wide
released-version matrices are then retired outright β sphinx-test-reports'
six-pinned-versions nox matrix gets deleted, not ported β and testing becomes:
in-tree sphinx-needs β so a sphinx-needs PR that breaks an extension fails that PR.
Matrix cells are expressed as
PEP 735 dependency groups plus uv's
conflictstable, which means every matrix cell is locked and reproducible β an improvement
over tox, where each env re-resolves against live PyPI.
sphinx-contrib/typer already ships exactly
this pattern in production. The local workflow tox provides today survives intact:
one command runs one cell (
poe cell sphinx-codelinks 3.12 sphinx8), each cell keepsits own persistent venv so re-runs are as fast as tox's cached envs, and a red CI
cell reproduces locally with the identical command. (Anyone attached to
tox -pparallel sweeps can keep a thin
tox.inias a purely local frontend via tox-uv'slock-runner β the one tox configuration that is safe inside a workspace.)
from its plain published metadata (
--no-sources) against PyPI, imported, andminimally exercised. Small but load-bearing: uv never checks that declared dependency
ranges are satisfiable by anything actually on PyPI, and inside the workspace the
local sphinx-needs silently satisfies everything. (We verified the sharp edges here
carefully β including that tox with tox-uv inside a workspace substitutes the local
package for the released one it claims to install, which is why these checks run
outside the workspace.) After a coordinated release this same check also proves the
declared floor, since the floor is the just-released version.
Documentation
Each package keeps its own docs build and its own versioned documentation site.
Read the Docs supports multiple projects building from one repository (per-project
config paths + automation rules that only trigger a build when that package's files
change). sphinx-codelinks and sphinx-mounts β currently unversioned GitHub Pages sites β
would gain versioned docs in the process. Existing URLs are preserved or redirected;
in particular
sphinx-needs.readthedocs.io(includingobjects.inv, which otherprojects' intersphinx builds depend on) stays reachable.
A possible later step β grouping all four under one docs domain with shared search
(RTD subprojects) β is explicitly not part of this proposal and would be discussed
separately.
Releases
sphinx-needs-v8.5.0,sphinx-codelinks-v1.5.0, β¦ (today all four repos use bareX.Y.Ztags β eight ofwhich literally collide between repos).
explicitly supported as the publisher for multiple PyPI projects). sphinx-needs moves
from its current API-token setup to trusted publishing as part of this.
requirement is set mechanically to
sphinx-needs>=<current>,<next major>β exactlythe version it was developed and tested against. The extensions never carry
backward-compatibility code; and because the cap sits at the major rather than an
exact pin, users can still upgrade sphinx-needs freely within a major without
waiting for extension re-releases. (This applies where a runtime dependency exists β
sphinx-codelinks and sphinx-test-reports today; sphinx-mounts deliberately has none.)
One consequence to own: when a sphinx-needs major ships, the
<next major>capshold users back until the extensions re-release β so a core major is followed by a
prompt re-release train of the extensions, which the release workflow can do in one
run.
extension first runs its test suite against the latest released sphinx-needs: green
means the extension ships with that as its floor and no core release is forced (so
unfinished work on sphinx-needs
masternever gets pushed out early by an unrelatedextension fix). Red means the extension depends on unreleased sphinx-needs changes β
the workflow then releases sphinx-needs first, waits until PyPI serves it, raises the
extension's floor, and releases the extension.
mcbeet/beet runs the core-first loop today
(core first, dependants after, floors bumped in between).
releases, with filtered release listings per package.
History, issues, and the old repos
git filter-repo, rewriting paths intopackages/<name>/β sogit logandgit blameon the moved files keep their fullhistory (most projects that did similar merges dropped history; we'd rather not).
Issue references in imported commit messages are rewritten to their fully-qualified
form (
useblocks/sphinx-codelinks#123) so they keep pointing at the right thing.package:labels; open PRs unfortunately cannot be transferred and would be closedwith a note inviting re-opening here (~18 across the three repos).
URL, and release keeps working.
Prior art
Projects we studied that run this shape today, and what we'd take from each:
docs/with ownconf.py; parameterised contributor/agent docsFor honesty's sake: the Sphinx ecosystem has historically gone the other way
(sphinx-contrib once had a monorepo and split it into per-extension repos). The
difference that makes us optimistic: the ecosystems where monorepos work well (airflow,
langchain, dbt) all share our situation β one team maintaining and releasing the core
and its extensions together β whereas sphinx-contrib was many unrelated maintainers
sharing a repo.
Open questions (feedback wanted)
requires-pythonacross members β theintersection of today's metadata is
>=3.12(sphinx-codelinks and sphinx-mounts arealready 3.12+). Aligning everything at >=3.12 alongside a sphinx-needs 9.0 would
be the clean answer, especially with Python 3.10 reaching end-of-life in October 2026
and Sphinx 9 itself requiring 3.12. Would dropping Python 3.10/3.11 support in the
next sphinx-needs major be a problem for your setups?
sets each extension release to require the sphinx-needs it was built against
(upgradeable within the major), in exchange for zero compatibility shims and instant
use of new sphinx-needs APIs. If you rely on running sphinx-codelinks or
sphinx-test-reports against an older, pinned sphinx-needs β including if you
re-package these for conda or a Linux distribution β tell us; that is the one
workflow this policy trades away. (For honesty: today's wide ranges are partly
nominal anyway β sphinx-test-reports declares
sphinx-needs>=1.0.1, which no CI hastested for years.)
useblocks/sphinx-needs(GitHub redirects mean nothingbreaks) or rename to reflect the wider scope?
first and transfer selectively?
or other extensions you think should join the monorepo (or candidates listed here
that you think shouldn't)?
Proposed first steps
Nothing structural lands until the questions above settle. The early steps are
deliberately non-disruptive β each is useful even if the monorepo never happens:
Step 1 β converge the tooling in place (independent PRs, per repo):
ruff version across config and hooks, move the package version from
__init__.pyto staticpyproject.tomlmetadata, and switch PyPI publishing from theAPI token to trusted publishing.
sphinx-needs>=1.0.1, unboundedsphinx>4.0), modernise the CI actions, and align its lint/type setup with the others.[project.urls]metadata.from any ruleset changes (two distinct churn sources; conflating them makes review
impossible).
Step 2 β add the install smoke checks:
import-tested, in each repo's CI. Cheap now, and it is the guard that stays honest
once the monorepo makes in-tree testing the default.
Step 3 β rehearse on this repo alone:
packages/sphinx-needs/+ aworkspace root, root pytest config, prek workspace mode) while it is still the only
package. Low risk, and it shakes out the tooling with no other package involved.
Step 4 β import incrementally, smallest first:
on arrival), then sphinx-codelinks; each via
git filter-repowith prefixed tagrenames, followed by its issue transfer and docs wiring β and each import stands on
its own, so a package that isn't ready simply stays where it is for now.
sphinx-test-reports carries the most modernisation work and can follow whenever it is
ready without blocking any of the above. Re-register each PyPI trusted publisher for
the new workflow before that package's first monorepo release.
Comments very welcome β especially from anyone maintaining downstream tooling or CI that
pins these packages, and from contributors on whether the monorepo makes your workflow
better or worse.
All reactions