Skip to content

fix(attractor): correct inverted community hierarchy (#33)#37

Merged
raphasouthall merged 1 commit into
mainfrom
fix/33-community-hierarchy
Apr 16, 2026
Merged

fix(attractor): correct inverted community hierarchy (#33)#37
raphasouthall merged 1 commit into
mainfrom
fix/33-community-hierarchy

Conversation

@raphasouthall
Copy link
Copy Markdown
Owner

Summary

Fixes the inverted community hierarchy reported in #33, where vault_stats returned communities_coarse=12, communities_fine=8 (fine had fewer, larger basins than coarse).

Three compounding issues drove the inversion: (1) a shared TOP_K_NEIGHBORS=50 at both β levels funnelled notes through the same hub attractors at high β, (2) singleton merging absorbed the narrow 1-note basins that fine is supposed to expose, (3) only β differed between levels.

Changes

  • Per-level top-K (TOP_K_COARSE=80, TOP_K_FINE=20) plumbed through _attractor_convergence so each pass uses an appropriate neighbourhood.
  • Singleton-merge gate_assign_communities(merge_singletons=bool); the fine pass calls it with False so narrow basins survive.
  • Modularity + size stats_modularity() computes weighted Newman Q on pre-sparsification S; detect_communities logs a warning if Q(fine) <= Q(coarse) so a future regression is visible at index time.
  • Persistence — new community_level_stats table (schema v13 → v14) stores n, min/max/mean size, modularity per level.
  • vault_stats output — adds community_levels: [...] array (legacy communities_coarse/fine/summarized preserved).

Results on production vault (539 notes)

Metric Before (issue #33) After
coarse n 12 10
fine n 8 (inverted) 13
fine min size 11 3
fine giants ≥100 2 1
n_fine > n_coarse

The modularity sanity warning fires honestly — Q(fine)=0.0527 < Q(coarse)=0.0628. The count hierarchy is correct but weighted modularity still favours coarse, a known property with heavy-tailed edge weights. Surfaced for future investigation rather than silently hidden.

Test plan

  • 544 tests pass locally (uv run pytest tests/)
  • Ruff clean on changed files
  • 13 new test cases cover top_k override, merge gate, modularity (2-block / single-community / degenerate), size stats
  • Schema migration v13→v14 idempotent (existing tests updated)
  • Deployed to production LXC — service restart clean, vault_stats over MCP returns new block, vault_communities at both levels returns coherent results with narrow basins (3-note, 7-note) surviving at fine level

Notes

  • Issue's proposed item 5 (rename coarse/finebroad/narrow) deferred. The modularity warning will signal at index time if the hierarchy tuning fails to deliver on any given vault; revisit the rename only if the warning persists.

Closes #33

The fine partition was returning fewer (and larger) clusters than coarse
because three settings were shared across β levels: top-K=50 neighbours,
always-on singleton merging, and identical softmax initialisation. At
high β the shared top-K funnelled notes through the same hub attractors
and merge_singletons swallowed the narrow basins that fine is supposed
to expose.

Changes:
- Per-level top-K: TOP_K_COARSE=80, TOP_K_FINE=20; plumbed through
  _attractor_convergence so each pass has an appropriate neighbourhood.
- Singleton-merge gate: _assign_communities(merge_singletons=bool); the
  fine pass calls it with False so 1-note basins survive.
- Modularity + size stats: _modularity() computes weighted Newman Q on
  the pre-sparsification S; _size_stats() reports size distribution.
  detect_communities logs a warning if Q(fine) <= Q(coarse) so a future
  regression is visible at index time.
- Per-level stats persisted in new community_level_stats table
  (migration v13 -> v14). vault_stats now emits a community_levels
  array with level, label, n_communities, min/max/mean_size, modularity.
- Tests cover top_k override, merge gate, modularity (2-block positive,
  single-community zero, degenerate zero), size stats.
@raphasouthall raphasouthall merged commit ed91e5b into main Apr 16, 2026
5 checks passed
@raphasouthall raphasouthall deleted the fix/33-community-hierarchy branch April 16, 2026 18:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: community hierarchy is inverted (fine partition has fewer clusters than coarse)

1 participant