Skip to content

Bug: community hierarchy is inverted (fine partition has fewer clusters than coarse) #33

@raphasouthall

Description

@raphasouthall

Summary

`vault_stats` reports `communities_coarse=12` and `communities_fine=8`. Textbook Hopfield dynamics (low β = broad basins, few clusters; high β = narrow basins, many clusters) says fine should be greater than coarse. On a 458-note vault, the fine partition has 2 giant (≥100-note) clusters and zero small ones, while the coarse partition has 1 giant and 2 tiny ones.

level count min max mean giants ≥100
0 β=0.5 ('coarse') 12 3 121 38 1
1 β=2.0 ('fine') 8 11 123 57 2

Root cause

Three compounding issues in `src/neurostack/attractor.py`:

  1. Shared top-K sparsification (`TOP_K_NEIGHBORS=50`, line 61; applied at L237-238). At high β, softmax concentrates on the largest neighbour, and identical top-K neighbour sets funnel notes into a handful of hub-attractors. Raising β counter-intuitively helps hubs monopolise basins.
  2. Singleton merging (`_assign_communities`, L296-315) merges every 1-note basin into its nearest non-singleton. At β=2.0 many notes converge to their own attractor and then get absorbed — fine level has 0 singletons pre-merge.
  3. Only β differs between levels. Same similarity matrix, same initial state (`np.eye`, L244), same sparsification.

`communities_summarized: 20` is not a third count — it's `12+8` concatenated (rows from both levels live in the same table). See `tools/search_tools.py:400-402` and `community.py:183-190`.

Proposed fix

  1. Per-level top-K. Add `TOP_K_COARSE=80` and `TOP_K_FINE=20` in `attractor.py:61`; thread through `_attractor_convergence`.
  2. Gate singleton-merge at fine level. Add `merge_singletons: bool = True` arg to `_assign_communities`; call with `False` at β=2.0 or only merge if similarity to a non-singleton exceeds a threshold.
  3. Report sizes + modularity in stats. `search_tools.py:394-402` — add `min/max/mean_size` and modularity per level so regressions are visible.
  4. Modularity sanity check after assignment in `attractor.py` — warn if `Q(fine) <= Q(coarse)`.
  5. Naming. If tuning can't reliably deliver `n_fine > n_coarse` on all vaults, rename `coarse`/`fine` → `broad`/`narrow` (describes attractor width, not partition-size). Update `BETA_COARSE`/`BETA_FINE` constants, stats keys, and `vault_communities` docstring.

Expected effect

Fine partition retains narrow basins. Hierarchy is either truly hierarchical or honestly named. Stats surface the size distribution so anomalies are obvious.

Key files

  • `src/neurostack/attractor.py:42-50, 61, 237-238, 273-325, 417-441`
  • `src/neurostack/tools/search_tools.py:394-402`
  • `src/neurostack/community.py:183-190`

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions