RFC: package-cohesion analysis — graph-driven sub-package suggestions (cgis suggest-packages)

[zaebee]

Status: RFC / design. Origin: src/cgis/query/ grew to 19 flat files spanning ≥4 distinct subsystems — a cohesion smell. A spike (this issue) confirms it's measurable and that the dependency graph itself proposes the fix. This RFC covers (a) the cgis self-cleanup and (b) generalizing the analysis into a product capability.

1. The spike — graph confirms the smell

Built the intra-package import graph of query/*.py (19 files, 19 edges) and ran hand-rolled greedy modularity (Clauset-Newman-Moore; cgis avoids NetworkX by design):

Modularity Q = 0.434   (> 0.3 = strong community structure)

cluster 1: audit, context_service, engine, prompt, snippet   (agent-facing navigation)
cluster 2: _scc, fingerprint, health, triads                 (graph-metric primitives)
cluster 3: drift, drift_service, ontology_init, quotient      (drift subsystem — cleanest)
cluster 4: analyzer, anomaly                                  (god-object / anomalies)
isolated : fqn, graph_json, mermaid, metrics                 (pure leaves, no intra-query imports)

The directory structure (flat) and the dependency graph (Q=0.434) diverge — that divergence IS the smell, now quantified. The algorithm suggests boundaries; it doesn't dictate them (engine lands with its agent-facing consumers because it's shared core; the 4 leaves are sinks with no community).

2. The metric (best-practice grounding)

• Robert Martin's package cohesion principles — CCP (what changes together), CRP (what's reused together), REP. The organizing criterion; the graph is the evidence.
• Coupling metrics cgis already computes (anomaly.py): Ca/Ce, Instability I = Ce/(Ca+Ce), Abstractness, Distance-from-main-sequence. Apply at file scope within a package.
• Community detection (modularity Q + Louvain/CNM) on the intra-package import graph — high Q = natural split points; low Q = a genuinely cohesive unit where splitting would be arbitrary. This is the same machinery as the domain quotient (quotient.py), one scale down.

3. Product capability — cgis suggest-packages

A new analysis that, for any package: builds the intra-package import (and optionally CALLS) graph, runs community detection, reports modularity Q + the communities + the cross-community "bridge" edges (the cost of splitting), and flags packages whose directory layout diverges from their dependency communities. Reuses the quotient/census engine; surfaces via CLI + MCP like the other analyses. Turns "this feels messy" into a data-backed recommendation for any agent/architect.

4. cgis self-cleanup — proposed layout (data-informed)

sub-package	files
query/drift/	drift, drift_service, fingerprint, triads, quotient, ontology_init, _scc
query/analysis/	analyzer, anomaly, health
query/context/	context_service, prompt, snippet, audit
query/render/	graph_json, mermaid, metrics
query/ (core)	engine, fqn

5. Risks / constraints (these are load-bearing)

• Drift metric can be gamed by cosmetic re-exports (IMPORTS-layer laundering) #182 re-export laundering. Moving files breaks from cgis.query.X import … everywhere. Compatibility re-export shims would launder the import census (artificially smooth/inflate drift) — the exact anti-pattern Drift metric can be gamed by cosmetic re-exports (IMPORTS-layer laundering) #182 names. Either do an honest mechanical update of all import sites, or accept the census change consciously.
• Self-drift domain shift. The drift census treats cgis.query as ONE domain. Splitting into sub-packages reshapes the domains → patterns.yaml bindings move → the restructure moves the measurement itself. A re-baseline + new domain bindings are part of the cost. (Already at the edge: cgis.query drift sits at 0.18 today.)
• Lane coordination. The drift/* cluster is the most active zone (colleague's cgis_drift: report fit-quality (nearest-template residual) + domain coverage; surface 'no template fits' #177/Recursive-quotient: cross-scale motif analysis (fractality / mirror-symmetry) as an architecture signal #186/Recursive-quotient: cross-scale motif analysis (fractality / mirror-symmetry) as an architecture signal #186-tangle). A split touching those files = large merge conflict; sequence it for a quiet window or coordinate.
• Don't over-nest. Python favors flat until it hurts. Q=0.434 justifies a split, but ship it as one deliberate refactor with the import-site updates, not a drip.

6. Slices

1. Slice 0 (this RFC): decide go/no-go; pick whether cgis suggest-packages is built first (so the split is data-driven and reproducible) or the cleanup is done by hand from the spike.
2. Slice 1 — cgis suggest-packages (the analysis as a feature; low risk, new module).
3. Slice 2 — query/ restructure (honest import-site update + drift re-baseline + colleague coordination).

Refs: spike in-issue. Builds on quotient.py / _scc.py / anomaly.py. Related: #182 (re-export laundering), #117 (god-object decomposition — same cohesion family at class scope), #186 (motif/tangle work in the drift cluster).

[zaebee]

Cross-repo validation — the metric discriminates (and isn't size-driven)

Ran the same spike (AST intra-package imports → greedy CNM modularity) across 7 real packages — cgis + owner-api (aura is mostly non-Python). Result spans Q = 0.00 → 0.43 and tracks latent structure, not file count:

package	files	Q	read
cgis query/	19	0.434	strong communities → split justified
owner-api api/dependencies	13	0.319	moderate → borderline: a biz-deps cluster (currency/owner/reservation) vs an infra cluster (auth/redis/session)
owner-api domains/admin	14	0.163	weak → cohesive domain, leave
owner-api worker	16	0.134	weak → leave
owner-api services	11	0.122	weak → services already decoupled (5 isolated, single-responsibility)
owner-api app/utils	11	0.000	textbook flat: 11 independent helpers, zero intra-pkg edges → correctly "leave it"
owner-api app	5	0.000	entry points, no coupling

Key takeaways for the feature:

1. Q is not size-driven — worker (16 files) scores 0.134 while cgis query/ (19 files) scores 0.434. The signal is community structure, not bulk.
2. cgis query/ is a genuine outlier at 0.434 — the only package here with distinct subsystems crammed flat.
3. The Q=0 cases (utils, app) are the crucial counter-examples — a flat package of independent modules is correctly told "leave it", proving the tool doesn't just say "big = split".
4. api/dependencies (0.319) is an independent second hit — the method surfaces a real biz-vs-infra split outside cgis, so it generalizes.

This is the empirical backing for cgis suggest-packages: flag packages whose directory layout diverges from their dependency communities (high Q + no sub-dirs), and explicitly don't flag well-decoupled or genuinely-flat ones (low Q). A useful default threshold from this sample sits around Q ≳ 0.35 for "worth splitting", with 0.25–0.35 as "borderline, look at the bridges".

[zaebee]

External validation — the headline: cgis query/ (19 files) > rich (76 files)

Ran the spike on two well-known external libraries (shallow-cloned):

package	files	edges	Q
cgis query/	19	19 (sparse)	0.434
rich (Textualize)	76	348	0.232
requests (psf)	18	57 (dense)	0.108

rich is a famous, mature, 76-file flat package — and it scores lower modularity than cgis's 19-file query/. That's the strongest possible proof the metric is about structure, not size:

• rich is one cohesive rendering toolkit — everything routes through console/segment/style/text, so despite 76 files the communities blur (Q=0.232 → "large but coherent, leave it").
• requests is a tightly-coupled small lib (57 edges over 18 files, avg degree ~6) → very low Q → "leave it".
• cgis query/ is sparse (19 edges over 19 files, avg degree ~2) with the few edges clustered — the signature of "several subsystems sharing thin coupling", which is exactly what splits cleanly.

Refined signal: it's not density or file count — it's sparse + clustered topology. A dense package (rich/requests) is internally cohesive and stays whole; a sparse-but-clustered one (cgis query/) is distinct subsystems flattened. The cgis suggest-packages heuristic should weight modularity Q (not size), and the Q≳0.35 threshold holds across all 9 packages tested (cgis, owner-api ×6, rich, requests) — only cgis query/ crosses it.

[zaebee]

Framework contrast (django, fastapi) — found an external high-Q that validates the split signal

Final corpus: 15 packages across 5 codebases (cgis, owner-api, rich, requests, django, fastapi).

package	files	Q	verdict
django/utils	42	0.456	SPLIT — classic grab-bag
cgis query/	19	0.434	SPLIT
owner-api api/dependencies	13	0.319	borderline
django/db/models	16	0.289	borderline
rich	76	0.232	leave (large but coherent)
fastapi/fastapi	22	0.213	leave
django/template	14	0.179	leave
fastapi/security	6	0.167	leave
owner-api domains/admin	14	0.163	leave
owner-api worker	16	0.134	leave
owner-api services	11	0.122	leave
requests	18	0.108	leave
django/forms	8	0.008	leave (dense, cohesive)
owner-api utils	11	0.000	leave (independent)
owner-api app	5	0.000	leave

Two decisive findings:

1. Only django/utils (0.456) and cgis query/ (0.434) cross Q≳0.35 — two independent, mature codebases. The split signal is real and generalizes; django/utils is the textbook 42-file utility grab-bag (cache/crypto/encoding/html/text/…) that genuinely wants sub-packages, and the spike's clusters even map to system-utils / formatting / web-data.

2. owner-api/utils (Q=0.000) vs django/utils (Q=0.456) — same name, opposite verdict. owner-api's utils are truly independent (zero cross-references → no structure to extract); django's utils reference each other in clusters (encoding↔functional, html↔text). The metric reads the graph, not the name — which is exactly why cgis suggest-packages would be more than a "big package" linter.

The Q≳0.35 threshold cleanly separates the 2 split-candidates from the 13 leave-its across all 5 codebases. Validation complete — the heuristic is empirically grounded.

[zaebee]

Honest dogfood — reproduced on the real cgis graph (not the AST shortcut)

The earlier numbers came from a standalone AST-import prototype. Re-ran the community detection on cgis's actual ingested graph (cgis ingest src → query the IMPORTS/IMPORTS_SYMBOL/CALLS edges, map each endpoint to its cgis.query.* file). The community-detection step is still the hand-rolled CNM (that's the proposed feature); only the graph now comes from cgis.

graph source	Q	clusters
AST import shortcut	0.434	4
cgis IMPORTS + IMPORTS_SYMBOL	0.453	same 4
cgis CALLS	0.441	same 4
cgis combined	0.447	same 4

Findings:

1. The cgis graph reproduces the spike — Q ≈ 0.45 and the identical 4 communities (agent-facing / metric-primitives / drift / anomalies). The cheap AST proxy was faithful, and cgis's resolved IMPORTS_SYMBOL/CALLS make it slightly richer (0.453 vs 0.434).
2. imports ≈ calls ≈ combined (0.453 / 0.441 / 0.447 → same clustering). For query/, the import-coupling and the call-coupling yield the same community structure (files import what they call). So cgis suggest-packages can run on either layer or the union — the result is robust to the choice.

This confirms feasibility: cgis's own graph is the right substrate (more accurate than raw imports), the quotient/census engine already builds it, and only the modularity pass (CNM/Louvain) is net-new. Slice 1 is well-scoped.