Cross-file nodes whose labels normalize identically are never merged (making them less similar does merge them)

[rays23]

Version: graphifyy 0.9.26 (fresh uv tool install graphifyy==0.9.26), macOS, Python 3.12.

Summary

Two non-code nodes in different source files whose labels normalize to the same string are never merged. Pass 1 partitions by source_file and explicitly defers cross-file matches to Pass 2, but Pass 2's candidate filter drops the second node because its normalized label was already seen, so the pair Pass 1 handed over can never be formed.

The behavior is inverted in a way that makes it easy to spot: changing one character so the labels no longer normalize equal makes them merge.

Repro

from graphify.dedup import deduplicate_entities

def survivors(a, b):
    nodes = [
        {"id": "a", "label": a, "file_type": "concept", "source_file": "doc1.md"},
        {"id": "b", "label": b, "file_type": "concept", "source_file": "doc2.md"},
    ]
    kept, _ = deduplicate_entities(nodes, [], communities={"a": 0, "b": 0})
    return len(kept)

print(survivors("SHENZHEN INTERNATIONAL", "Shenzhen international"))  # 2  <- expected 1
print(survivors("SHENZHEN INTERNATIONAL", "Shenzhen internationaI"))  # 1  <- merges

Output on 0.9.26:

'SHENZHEN INTERNATIONAL' / 'Shenzhen international'   _norm equal: True    -> 2 survivors   <- BUG
'SHENZHEN INTERNATIONAL' / 'Shenzhen internationaI'   _norm equal: False   -> 1 survivor    (1 fuzzy)

Same outcome for any concept-typed pair in different files: WeLab / Welab, Greater Bay Biotechnology / GREATER BAY BIOTECHNOLOGY, and so on.

Root cause

dedup.py:414 (Pass 1) deliberately hands cross-file matches to Pass 2:

# Partition by source_file — only merge within the same file in Pass 1.
# Cross-file matches fall through to Pass 2 fuzzy matching.

But Pass 2's candidate filter at dedup.py:441-445 deduplicates the candidate list by normalized label:

key = _norm(node.get("label", node.get("id", "")))
if key and key not in seen_norms:
    seen_norms.add(key)
    if _entropy(node.get("label", "")) >= _ENTROPY_THRESHOLD:
        candidates.append(node)

Only the first node carrying a given normalized label ever enters candidates. The hand-off therefore fails for exactly the pairs Pass 1 sends over, namely identical normalized labels.

_is_code nodes are correctly excluded from this path; this report is only about semantic (concept) nodes. Note that _FILE_ANCHORED_NONCODE intentionally excludes concept with the comment "it is the type meant to unify across files", so the intent to unify concept across files already exists in the code.

Impact

Measured on a real corpus of 19,389 distinct company-name strings spread across many documents:

configuration	merges
all nodes share one source_file	539
nodes in different files (the realistic multi-document case)	160

Ground truth for that corpus is roughly 435 to 632 merges. The easiest possible entity-resolution case, the same name with different capitalization in two documents, fails silently, and that is also the most common case in any prose corpus.

Suggested direction

Either key the Pass 2 LSH index and seen_norms by node id instead of collapsing the candidate list by normalized label, or let Pass 1 union same-normalized-label nodes across files for non-_is_code types, where the existing guards (_numeric_tokens_differ, _short_label_blocked, _crossfile_fileanchored_blocked) already apply.

Happy to send a PR if you have a preferred direction.

Related

Distinct from #296 and #1651, which both ask for semantic or registry-based canonicalization. This one is narrower: labels that are already byte-identical after _norm never reach the merge at all. Same failure shape as the fixed #1453, where a seen_names dedup silently dropped entries.

[safishamsi]

Fixed in v0.9.27. Cross-file concept nodes with identical normalized labels now merge (gated to concept nodes with provenance, so code/rationale/document/image and cross-repo guards are preserved). https://github.com/Graphify-Labs/graphify/releases/tag/v0.9.27