The problem
graphify.cluster.cluster() runs community detection from scratch on every call. On a large graph that is refreshed incrementally, this makes re-clustering far more destructive than the underlying change.
Measured on a 793k-node, 1.77M-edge multi-repo graph where a source refresh changed roughly 1.5% of nodes:
- 88 previously-stable communities shattered into 2–8 fragments each;
- 349 of 418 successor communities were compositionally pure (median 95% of members from one predecessor) yet held less than half of that predecessor's members — splits, not renames;
- downstream, every per-community authored artefact keyed to those communities had to be regenerated, despite
remap_communities_to_previous correctly preserving ids for the communities that did survive intact.
remap_communities_to_previous fixes labelling after the fact, but nothing steers the partition itself toward the previous one, so unchanged regions of the graph can land on a different local optimum every run.
The ask
An optional seed for cluster():
cluster(G, resolution=1.0, exclude_hubs_percentile=None,
seed_partition: dict[str, int] | None = None)
initialising Louvain/Leiden from the previous membership rather than from singletons. Both algorithms support warm-start partitions; python-louvain exposes it directly (community_louvain.best_partition(G, partition=seed)), and leidenalg via initial_membership. Nodes absent from the seed start as singletons; nodes in the seed start in their previous community and move only if modularity actually improves.
This composes naturally with remap_communities_to_previous — seeding reduces how much there is to remap, and the existing function still handles genuine drift.
Why it matters beyond us
Any consumer attaching content to communities (summaries, labels, docs) pays the shatter cost on every refresh. The LLM community labels graphify itself generates are an in-tree example: our re-cluster invalidated 22,507 of 28,004 saved labels by membership signature, almost all of which would have survived a seeded run.
Related context: #2436 (the cluster-only persistence report) came out of the same refresh.
Versions: graphify 0.9.30, networkx 3.6.1, Python 3.13.
The problem
graphify.cluster.cluster()runs community detection from scratch on every call. On a large graph that is refreshed incrementally, this makes re-clustering far more destructive than the underlying change.Measured on a 793k-node, 1.77M-edge multi-repo graph where a source refresh changed roughly 1.5% of nodes:
remap_communities_to_previouscorrectly preserving ids for the communities that did survive intact.remap_communities_to_previousfixes labelling after the fact, but nothing steers the partition itself toward the previous one, so unchanged regions of the graph can land on a different local optimum every run.The ask
An optional seed for
cluster():initialising Louvain/Leiden from the previous membership rather than from singletons. Both algorithms support warm-start partitions; python-louvain exposes it directly (
community_louvain.best_partition(G, partition=seed)), and leidenalg viainitial_membership. Nodes absent from the seed start as singletons; nodes in the seed start in their previous community and move only if modularity actually improves.This composes naturally with
remap_communities_to_previous— seeding reduces how much there is to remap, and the existing function still handles genuine drift.Why it matters beyond us
Any consumer attaching content to communities (summaries, labels, docs) pays the shatter cost on every refresh. The LLM community labels graphify itself generates are an in-tree example: our re-cluster invalidated 22,507 of 28,004 saved labels by membership signature, almost all of which would have survived a seeded run.
Related context: #2436 (the cluster-only persistence report) came out of the same refresh.
Versions: graphify 0.9.30, networkx 3.6.1, Python 3.13.