Splitting COVET into its own distribution: what it would take

[Marius1311]

Splitting COVET into its own distribution

Following up on #91, where @Tobiaspk proposed keeping pip install scenvi as the
full ENVI install and publishing COVET separately:

repo/
├── COVET/
│   ├── pyproject.toml
│   └── src/covet.py
└── ENVI/
    ├── pyproject.toml
    └── src/envi.py

This is a writeup of what that would take, so the decisions can be made before
anyone writes code. I'm happy to implement it — the parts I can't decide are
marked below.

Why a second distribution, and not an extra

A single distribution has exactly one default install, and pip extras only add
to that default. So one sdist can offer light-by-default plus an [envi] extra
(what #91 originally did, now dropped), or full-by-default with no light path —
but not full-by-default and a light COVET install. If both are wanted, COVET
has to be its own distribution. So the proposal in #91 is the right mechanism,
not merely one option among several.

The code-level precondition is already in #91: COVET no longer imports the
deep-learning stack, and there's a test asserting it stays that way. covet.py
cannot move into its own distribution while utils.py defines flax modules.

The one genuine obstacle: the inter-package dependency

scenvi must depend on sccovet. That dependency has to resolve from the local
working tree during development and CI, but from PyPI in the published wheel.
Poetry cannot express both: a relative path dependency is not publishable (the
wheel standard forbids it), and develop = true is poetry-only metadata that
never reaches distribution metadata.

Three ways out, in my order of preference:

1. Two ordinary pyprojects, no path dependency. ENVI/pyproject.toml
   declares sccovet = ">=X.Y" as a normal PyPI constraint; development and CI
   just install COVET from the working tree first (pip install -e ./COVET
   before pip install -e ./ENVI). No plugins, no workspace support needed. The
   cost is that releases must be ordered — publish sccovet, then scenvi
   pinning the new version.
2. A uv workspace (tool.uv.sources + uv build --package), which is
   designed for exactly this and gives the nicest development experience. Implies
   moving off poetry.
3. Poetry plus poetry-monorepo-dependency-plugin,
   which rewrites path dependencies into pinned versions at build time. Keeps
   poetry, but adds a plugin to the release path.

What publish.yaml needs

The current workflow is single-package by construction — one poetry version -s,
one tag, one poetry build, one poetry publish, triggered on every merged PR.
For two packages it needs:

• two version streams (or an explicit decision to move them in lockstep);
• per-package change detection, otherwise every ENVI-only merge republishes
  sccovet unchanged;
• ordered publishing, with scenvi's constraint on the newly released sccovet.

Worth noting independently of the split: scverse's
cookiecutter template uses
hatchling, PEP 621 metadata, and releases on release: types: [published] via
PyPI trusted publishing rather than an API token. That is a better base than
auto-patch-bump-on-merge whether or not the split happens, and it would make the
two-package version story much easier. It does not itself support monorepos — it
is a single-package template — so it solves the release tooling, not the layout.

Also, unrelated but adjacent: publish.yaml triggers on pull_request from
forks, which receive no secrets and a read-only GITHUB_TOKEN. #91/#92 are the
first fork PRs on this repo, so on merge I'd expect the publish job to fail
rather than release.

Decisions I can't make

1. PyPI name and ownership. sccovet is currently unregistered (404), so it
   is available. Who registers and owns it?
2. Import name. import covet or import sccovet? The distribution name and
   the module name don't have to match, but picking deliberately is better than
   inheriting one.
3. Backwards compatibility. scenvi.compute_covet is what the README and
   MOp_MERFISH_tutorial.ipynb document, and what downstream code imports. I'd
   keep scenvi re-exporting it from sccovet indefinitely — but confirm.
4. Version coupling. Lockstep, or independent cadences? This drives most of
   the publish.yaml design.
5. Build backend. Stay on poetry (route 1 or 3), or move to hatchling/uv
   (route 2), possibly aligning with the scverse template at the same time.

Suggested sequencing

The split lands much more safely once CI is green, since a layout change is hard
to review against a red baseline. #93 fixes the import failure that is
currently failing every build job. After that, the restructure is mostly
mechanical.