Wave 11: dynamic graph entity types

[earayu]

Summary

• Hard-cut graph entity types to collection-local KnowledgeGraphConfig.entity_types: list[str] with default [].
• Remove the _DEFAULT_ENTITY_TYPES fallback and stop treating entity types as a closed enum in the extraction prompt.
• Add dynamic type helpers plus async merge_entity_types(session, collection_id, new_types) with SELECT ... FOR UPDATE and first-write-wins dedupe.
• Wire a per-document entity-type merge after kg.jsonl is written; merge failures log a warning and do not poison graph output.
• Keep graph storage and curation on entity_type: str; no type_id / label split and no registry table.
• Add the locked Chinese Wave 11 design doc under docs/modularization/.

CR Fixes

• Removed the prompt-side 50-type cap; the prompt now renders every cleaned existing type.
• Added a concurrent merge test proving row-lock serialization keeps both writers' entity types.
• Expanded this PR body with the §16 hard-gate sections below.

Spec Gates

• Hard cut only: no old-data migration, no lazy seed, no backfill.
• No DB schema change / alembic migration.
• New collections start with entity_types=[]; LLM can propose the first types from real documents.
• Invalid or overlong type clears only the type field (entity_type=""); the entity is preserved.
• ASCII case dedupe is first-write-wins.
• Prompt language comes from collection.config.language; entity names preserve source text.
• Prompt receives the full cleaned entity_types list; no prompt list cap, size cap, LRU, or fallback default.
• Per-document merge happens after graph derive writes kg.jsonl; no per-chunk or per-run merge.

12-Invariant Gate

1. EntityRecord.entity_type remains a string; no type_id field is introduced.
2. kg.jsonl entity/relation record shape remains compatible.
3. Graph storage backends continue storing/filtering string entity labels.
4. Graph curation continues using CurationEntity.entity_type string paths.
5. Wave 7 compacted entity/relation descriptions are untouched.
6. Wave 7 merge-candidate/vector sync hooks are untouched.
7. Wave 8 bulk-upsert entity type behavior remains covered by existing tests.
8. Tenant scope and parse-version artifact paths are unchanged.
9. Graph derive still writes kg.jsonl even if config merge fails.
10. Config merge never deletes or rewrites existing graph entities.
11. Empty or bad entity type values do not pollute collection config.
12. No production-data migration/backfill path is required for this hard cut.

Pattern Gate

• Pattern 1: reuse the existing graph entity_type: str storage and API surface.
• Pattern 2: bind graph output language to collection.config.language; no auto language mode.
• Pattern 3: reuse existing KnowledgeGraphConfig.entity_types; no new registry table.
• Pattern 4: keep graph curation/search compatibility by preserving the string type surface.

11 Mini-Pattern Gate

1. merge_entity_types() is async and takes AsyncSession.
2. Atomic merge path uses SELECT ... FOR UPDATE.
3. No sync wrapper and no optimistic-retry fallback.
4. Merge call granularity is per-document after kg.jsonl flush.
5. _DEFAULT_ENTITY_TYPES is removed and no longer a fallback.
6. Unknown/new LLM type strings are accepted.
7. Overlong/bad type clears only the type field; entity row is preserved.
8. ASCII case dedupe is first-write-wins.
9. Merge failure logs warning and is non-fatal/self-healing.
10. Full existing type list is rendered into the prompt; no 50-item truncation.
11. Concurrent merge behavior is covered by a serialization test.

Simple-Stable Guardrails

• No scope expansion: no lifecycle/status/source/audit registry model.
• Fastest safe path: one existing JSON config field and one merge helper.
• Simple stable behavior: strings are the storage/display contract within a fixed-language collection.
• Low maintenance: no schema table, no default ontology upkeep, no size-cap/LRU policy to tune.

Validation

• uv run ruff check aperag/indexing/entity_types.py aperag/indexing/graph.py aperag/indexing/graph_extractor.py aperag/indexing/llm.py aperag/indexing/worker_factory.py aperag/schema/common.py tests/unit_test/indexing/test_entity_types.py tests/integration/test_graph_extractor.py tests/unit_test/indexing/test_t1_2_graph.py
• uv run --extra test pytest tests/unit_test/indexing/test_entity_types.py tests/integration/test_graph_extractor.py tests/unit_test/indexing/test_t1_2_graph.py -q → 84 passed, 1 warning
• uv run python -m compileall -q aperag/indexing/entity_types.py aperag/indexing/graph.py aperag/indexing/graph_extractor.py aperag/indexing/llm.py aperag/indexing/worker_factory.py aperag/schema/common.py tests/unit_test/indexing/test_entity_types.py tests/integration/test_graph_extractor.py tests/unit_test/indexing/test_t1_2_graph.py
• git diff --check
• rg -n "type_id|collection_entity_type|_DEFAULT_ENTITY_TYPES" aperag tests → no output

[earayu]

Findings:

1. BLOCKER — format_entity_types_for_prompt() silently caps the prompt list to 50 types, but the v3 spec and PR body explicitly say Wave 11 does not do a type-list size cap / LRU. More importantly, the runtime contract says the LLM should first reuse the current collection entity_types list; hiding types after the first 50 makes reuse impossible for those types and can reintroduce type drift. Please either remove MAX_PROMPT_ENTITY_TYPES truncation or get an explicit architect/user spec amendment that prompt-snapshot truncation is allowed.

2. BLOCKER — §15 requires atomic merge coverage for “concurrent merge does not lose type”. Current coverage only asserts the SQL statement has FOR UPDATE and preserves config in a fake single-session path. Please add a targeted race/serialization test that proves two merges for the same collection both survive, e.g. existing types [], concurrent additions ["疾病"] and ["药物"] result in both values, or an equivalent deterministic fake session/lock test.

3. BLOCKER — PR body does not include the required Wave hard-gate sections from the design doc §16: 12-invariant cross-check, 4-pattern + 11 mini-pattern pre-check, and simple-stable 4-guardrail. The existing Spec gates section covers part of the PR-specific gates, but not the required three hard-gate sections. Please update the PR body before merge so architect ratify has the agreed checklist.

Checked:

• gh pr checks 1789 is green: 10/10 passing.
• Local targeted gate passed with uv run --extra test pytest tests/unit_test/indexing/test_entity_types.py tests/integration/test_graph_extractor.py tests/unit_test/indexing/test_t1_2_graph.py -q: 82 passed, 1 warning.
• ruff check passed for the changed backend/test files.
• rg -n "type_id|collection_entity_type|_DEFAULT_ENTITY_TYPES" aperag tests returned no hits.

After these are fixed, the main dataflow shape looks aligned: collection language reaches prompt rendering, entity types stay as strings, _DEFAULT_ENTITY_TYPES is removed from runtime, and graph storage/curation remains on entity_type: str.

[earayu]

LGTM.

I rechecked the three previous blockers against head f8210c64:

• Prompt type-list cap removed: format_entity_types_for_prompt() now renders every cleaned type, and test_format_entity_types_for_prompt_does_not_cap_type_list covers 75 entries.
• Concurrent merge coverage added: test_merge_entity_types_serializes_concurrent_updates proves the row-lock path serializes two writers and preserves both new type sets.
• PR body now includes the required §16 gates: 12-invariant, 4-pattern + 11 mini-pattern, and simple-stable 4-guardrail.

Additional checks:

• gh pr checks 1789: 10/10 passing.
• Local targeted test: uv run --extra test pytest tests/unit_test/indexing/test_entity_types.py tests/integration/test_graph_extractor.py tests/unit_test/indexing/test_t1_2_graph.py -q → 82 passed before the amend; Weston reports 84 passed after the amend and the new tests cover the changed areas.
• Local ruff check and compileall passed for the changed files.
• Runtime grep gate clean: rg -n "type_id|collection_entity_type|_DEFAULT_ENTITY_TYPES" aperag tests returned no hits.

The implementation now matches the v3 design lock: hard-cut entity_types: list[str], no registry table, no type_id, no default fallback, collection language reaches prompt, per-document async merge uses SELECT ... FOR UPDATE, and graph storage/curation stays on entity_type: str.

[earayu]

Architect ratify ✅ — 三段 hard-gate 全过 (12-invariant + 4-pattern/11 mini-pattern + simple-stable 4-guardrail). Spot-check confirms async signature, SELECT FOR UPDATE single path, per-document granularity, prompt no cap. CI 10/10 + 84 tests pass + grep zero. Proceeding to merge.