Semantic extraction is executed in parallel chunks (typically 20–25 files per chunk, with each image isolated into its own chunk). Each sub-agent only receives FILE_LIST for its chunk, and the prompt explicitly scopes some relationships to “in this chunk” (notably semantically_similar_to). As a result, relationships that require jointly seeing artifacts that land in different chunks can be missed.
Current state (as implemented/documented):
Chunking strategy is defined in the skill orchestration docs, not in Python code.
Non-code files are split into chunks of ~20–25; images are one-per-chunk.
There is caching to skip unchanged files, and parallel dispatch for speed/cost, but no explicit “global linking” pass to recover cross-chunk semantic edges.
Many structural cross-file edges in code are covered by the local AST extractor, but doc/paper/image semantic edges can still be chunk-limited.
Impact:
Cross-chunk semantic relationships can be under-recalled, especially:
semantically_similar_to edges (prompt-scoped to the current chunk)
Concept links between docs/papers and images when they are chunk-separated
The issue grows with the number of semantic (non-code) files:
<= 25 semantic files: usually 1 chunk, minimal impact
26–50: 2 chunks, noticeable misses
100+: multiple chunks, increasing fragmentation and missed cross-chunk edges
Many images amplify fragmentation (each image becomes its own chunk)
Expected behavior:
Preserve parallelism/cost controls while improving recall of cross-chunk semantic relationships.
Proposed improvements (options):
Smarter chunking: group files by directory/module/topic to keep related artifacts together.
Add a lightweight “global linking” second pass:
After merging per-chunk outputs, generate node summaries and ask the model to propose cross-chunk edges (e.g., conceptually_related_to, semantically_similar_to) without re-reading all raw files.
Optional controlled overlap/bridging:
Include a small overlap window between adjacent chunks or repeat key “index/overview” files across chunks.
Acceptance criteria:
A documented and/or implemented mechanism that reduces missed cross-chunk semantic edges without requiring a full re-read of the entire corpus.
Tests or a reproducible example corpus demonstrating improved cross-chunk link recall.
Semantic extraction is executed in parallel chunks (typically 20–25 files per chunk, with each image isolated into its own chunk). Each sub-agent only receives FILE_LIST for its chunk, and the prompt explicitly scopes some relationships to “in this chunk” (notably semantically_similar_to). As a result, relationships that require jointly seeing artifacts that land in different chunks can be missed.
Current state (as implemented/documented):
Chunking strategy is defined in the skill orchestration docs, not in Python code.
Non-code files are split into chunks of ~20–25; images are one-per-chunk.
There is caching to skip unchanged files, and parallel dispatch for speed/cost, but no explicit “global linking” pass to recover cross-chunk semantic edges.
Many structural cross-file edges in code are covered by the local AST extractor, but doc/paper/image semantic edges can still be chunk-limited.
Impact:
Cross-chunk semantic relationships can be under-recalled, especially:
semantically_similar_to edges (prompt-scoped to the current chunk)
Concept links between docs/papers and images when they are chunk-separated
The issue grows with the number of semantic (non-code) files:
<= 25 semantic files: usually 1 chunk, minimal impact
26–50: 2 chunks, noticeable misses
100+: multiple chunks, increasing fragmentation and missed cross-chunk edges
Many images amplify fragmentation (each image becomes its own chunk)
Expected behavior:
Preserve parallelism/cost controls while improving recall of cross-chunk semantic relationships.
Proposed improvements (options):
Smarter chunking: group files by directory/module/topic to keep related artifacts together.
Add a lightweight “global linking” second pass:
After merging per-chunk outputs, generate node summaries and ask the model to propose cross-chunk edges (e.g., conceptually_related_to, semantically_similar_to) without re-reading all raw files.
Optional controlled overlap/bridging:
Include a small overlap window between adjacent chunks or repeat key “index/overview” files across chunks.
Acceptance criteria:
A documented and/or implemented mechanism that reduces missed cross-chunk semantic edges without requiring a full re-read of the entire corpus.
Tests or a reproducible example corpus demonstrating improved cross-chunk link recall.