feat(graph-rag): executable parser and chunker strategies - #28
Merged
Conversation
|
Warning Review limit reached
Next review available in: 33 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (24)
📝 WalkthroughWalkthroughIntroduces a registry-based document ingestion pipeline with canonical parsing, four chunking strategies, tokenizer and embedding adapters, chunk provenance, deterministic processing profiles, and persistence changes for processing metadata. ChangesIngestion pipeline
Estimated code review effort: 5 (Critical) | ~120 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 17
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
apps/worker/src/main/java/com/orgmemory/worker/ingestion/SourceDocumentReader.java (1)
65-97: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winText is normalized twice per document.
read()buildsnormalizedText(lines 89-95) by normalizing everyDocument's text, butparse()discardsparsed.normalizedText()and callscanonical(parsed.documents()), which normalizes the same texts again (line 103). This duplicate pass wastes CPU on every parse and risks the two normalization outputs silently drifting apart if one is edited without the other.Also applies to: 99-128
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@apps/worker/src/main/java/com/orgmemory/worker/ingestion/SourceDocumentReader.java` around lines 65 - 97, Update the read/parse flow around read(), ParsedSource, and canonical() so document text is normalized exactly once per document. Reuse the normalizedText produced by read() when constructing the final parsed result, and remove the later canonical normalization pass while preserving the existing filtering and no-extractable-text behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@apps/worker/src/main/java/com/orgmemory/worker/ingestion/DocumentProcessingEngine.java`:
- Around line 94-131: Update DocumentProcessingEngine.process and the chunking
flow to detect documents that will exceed properties.maximumChunks() before
semantic embedding is performed, using an existing token-count pre-check or a
maxChunks-aware ChunkerOptions path. Preserve the existing CHUNK_LIMIT_EXCEEDED
rejection and ensure oversized semantic-vector requests do not invoke the
external embedding provider.
- Around line 40-41: Update parser selection around LEGACY_SUFFIXES and
ParserRegistrySnapshot.route() so shared md/docx suffixes resolve to the
intended parser for unrequested files, rather than implicitly selecting the
later nativeParser. Ensure PassthroughParser or the appropriate legacy/native
parser is eligible according to the desired behavior, while preserving
deduplication by parser component ID and first-match routing for other suffixes.
In
`@apps/worker/src/main/java/com/orgmemory/worker/ingestion/ProcessedSourceDocument.java`:
- Around line 14-21: Update the empty-chunks validation in
ProcessedSourceDocument so it throws the established RejectedSourceException,
preserving the existing message or rejection context. Ensure
SourceIngestionProcessor recognizes this failure through its existing quarantine
path rather than treating it as retryable, while leaving validation for
parseResult, chunks, and profile unchanged.
In
`@apps/worker/src/main/java/com/orgmemory/worker/ingestion/SourceDocumentReader.java`:
- Around line 76-82: Update the document-loading branch in SourceDocumentReader
so isPlainText types use TikaDocumentReader(resource).get() instead of decoding
bytes directly as UTF-8; retain the PDF-specific PagePdfDocumentReader path and
the existing Tika fallback for other types.
In
`@apps/worker/src/main/java/com/orgmemory/worker/ingestion/SourceProcessingProperties.java`:
- Around line 52-59: In the validation logic of SourceProcessingProperties,
split the combined Assert.isTrue call into separate assertions for chunkSize,
chunkOverlap, semanticEmbeddingBatchSize, and maximumChunks constraints. Give
each assertion a specific message identifying the invalid field and preserve the
existing boundary requirements, including chunkOverlap < chunkSize and
maximumChunks < Integer.MAX_VALUE.
In
`@components/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/ChunkedText.java`:
- Around line 16-19: Update the content normalization in ChunkedText so boundary
whitespace is preserved rather than stripped, keeping content exactly aligned
with the character span recorded in provenance and canonical hashing. Retain the
null validation and blank-content rejection, but validate without mutating the
stored content.
In
`@components/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/ChunkerOptions.java`:
- Around line 4-8: Change ChunkerOptions from a sealed interface to an ordinary
public interface by removing the permits clause, so third-party ChunkerPlugin
implementations can define strategy-specific options classes while preserving
the existing built-in implementations.
In
`@components/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/ChunkerRegistrySnapshot.java`:
- Around line 17-21: Update ChunkerRegistrySnapshot.require() to normalize
chunkerId with Locale.ROOT when lowercasing, matching ProcessingComponentRef and
ensuring consistent lookups regardless of the platform default locale.
In
`@components/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/ParagraphSemanticChunker.java`:
- Around line 470-488: Update applyPartSuffixes to compare headings null-safely,
so consecutive drafts with null headings are treated as the same block and
increment part rather than resetting to 1. Preserve the existing behavior for
non-null headings and single-draft lists.
In
`@components/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/CanonicalDocument.java`:
- Around line 10-19: Update the CanonicalDocument constructor to enforce LF-only
content: reject any content containing carriage returns before computing the
hash and accepting blocks. Preserve the existing nonblank and contentSha256
validation, ensuring CanonicalDocument cannot be created with text that text()
would normalize.
In
`@components/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/ParserRegistrySnapshot.java`:
- Line 18: Update the parser-ID normalization in ParserRegistrySnapshot to use
Locale.ROOT with toLowerCase, ensuring IDs normalize identically regardless of
the JVM default locale. Add a regression test that sets a Turkish default locale
and verifies an uppercase-I parser ID resolves through the normalize-and-lookup
path.
In
`@components/graph-rag-core/src/main/java/com/orgmemory/graphrag/processing/ResolvedDocumentProcessingProfile.java`:
- Around line 29-38: Update the public constructor of
ResolvedDocumentProcessingProfile to derive the expected profile digest from the
fully normalized profile fields using the existing canonicalForm and SHA-256
helpers, then reject any supplied profileSha256 that differs while retaining
format validation. Ensure profileSha256 remains derived from the canonical
profile representation rather than accepting an unrelated valid digest.
In
`@components/graph-rag-testkit/src/test/java/com/orgmemory/graphrag/testkit/ProcessingRegistryConformanceTests.java`:
- Around line 58-63: Update
chunkerSnapshotDoesNotChangeWhenStartupRegistryChanges to register a distinct
chunker on registry after snapshot creation, then assert snapshot.chunkers()
still contains only the original entry and snapshot.require for the new chunker
ID throws IllegalArgumentException. Keep the existing initial snapshot
assertions intact.
In
`@core/src/main/java/com/orgmemory/core/knowledge/ConnectorSourceRevisionCoordinator.java`:
- Around line 125-126: Update the legacy profile construction in
ConnectorSourceRevisionCoordinator to avoid passing revision.getContentSha256()
as the canonical-text hash. Supply the actual parsed canonical-text hash if
available; otherwise use the legacy profile form that omits canonicalTextSha256,
while preserving the existing parser and chunker version values.
In
`@core/src/main/resources/db/migration/V30__document_processing_profiles_and_chunk_provenance.sql`:
- Around line 4-34: Add the new CHECK constraints in migration V30 as NOT VALID
so existing rows are not scanned while the stronger lock is held, including
chk_source_revision_processing_profile, chk_knowledge_chunk_source_span, and
chk_knowledge_chunk_block_indexes. Create a subsequent Flyway migration that
validates each constraint with VALIDATE CONSTRAINT, preserving enforcement for
new writes and PostgreSQL compatibility.
In
`@integrations/graph-rag-spring-ai/src/main/java/com/orgmemory/integrations/graphrag/springai/JtokkitTextTokenizer.java`:
- Around line 69-73: Update JtokkitTextTokenizer’s token-window handling to
avoid rejecting windows that decode with U+FFFD: derive character-safe chunk
boundaries before performing provenance lookup, rather than throwing on split
UTF-8 characters. Preserve valid source-span mapping for adjusted boundaries,
and add a regression test covering an emoji split across FixedTokenChunker token
windows.
In
`@integrations/graph-rag-spring-ai/src/main/java/com/orgmemory/integrations/graphrag/springai/SpringAiTextEmbeddingPort.java`:
- Around line 24-25: Validate providerId and modelId as nonblank constructor
inputs before creating ProcessingComponentRef in SpringAiTextEmbeddingPort,
rejecting both null and blank values while preserving the existing model
validation. Add constructor tests covering null and blank providerId/modelId
cases.
---
Outside diff comments:
In
`@apps/worker/src/main/java/com/orgmemory/worker/ingestion/SourceDocumentReader.java`:
- Around line 65-97: Update the read/parse flow around read(), ParsedSource, and
canonical() so document text is normalized exactly once per document. Reuse the
normalizedText produced by read() when constructing the final parsed result, and
remove the later canonical normalization pass while preserving the existing
filtering and no-extractable-text behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 4cfa8ff0-d182-4246-80d7-a78f9a529463
⛔ Files ignored due to path filters (2)
docs/increments/active/2026-07-23-full-lightrag-semantic-port/plan.mdis excluded by!docs/**docs/research/lightrag-v1.5.4-parity-manifest.mdis excluded by!docs/**
📒 Files selected for processing (72)
apps/worker/src/main/java/com/orgmemory/worker/ingestion/DocumentParsingException.javaapps/worker/src/main/java/com/orgmemory/worker/ingestion/DocumentProcessingEngine.javaapps/worker/src/main/java/com/orgmemory/worker/ingestion/NativeSourceDocumentParser.javaapps/worker/src/main/java/com/orgmemory/worker/ingestion/ProcessedSourceDocument.javaapps/worker/src/main/java/com/orgmemory/worker/ingestion/SourceDocumentReader.javaapps/worker/src/main/java/com/orgmemory/worker/ingestion/SourceIngestionProcessor.javaapps/worker/src/main/java/com/orgmemory/worker/ingestion/SourceProcessingProperties.javaapps/worker/src/main/java/com/orgmemory/worker/ingestion/SpringAiKnowledgeTextChunker.javaapps/worker/src/main/resources/application.ymlapps/worker/src/test/java/com/orgmemory/worker/ingestion/SpringAiKnowledgeTextChunkerTests.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/ChunkProvenance.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/ChunkProvenanceFactory.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/ChunkTokenLimitExceededException.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/ChunkedText.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/ChunkerOptions.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/ChunkerPlugin.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/ChunkerRegistry.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/ChunkerRegistrySnapshot.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/ChunkingRequest.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/EncodedText.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/FixedTokenChunker.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/FixedTokenOptions.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/ParagraphSemanticChunker.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/ParagraphSemanticOptions.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/RecursiveCharacterChunker.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/RecursiveCharacterOptions.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/SemanticEmbeddingInvocationException.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/SemanticEmbeddingUnavailableException.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/SemanticVectorChunker.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/SemanticVectorOptions.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/SourceSpan.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/TextChunker.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/TextEmbeddingPort.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/TextTokenizer.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/TokenRangeLocator.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/CanonicalDocument.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/DocumentBlock.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/DocumentBlockKind.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/DocumentParseRequest.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/DocumentParseResult.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/DocumentParser.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/ParserPlugin.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/ParserRegistry.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/ParserRegistrySnapshot.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/ParserSpec.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/ParserUnavailableException.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/PassthroughParser.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/ReuseParser.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/processing/ProcessingComponentRef.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/processing/ResolvedDocumentProcessingProfile.javacomponents/graph-rag-testkit/src/main/java/com/orgmemory/graphrag/testkit/CodePointTokenizer.javacomponents/graph-rag-testkit/src/main/java/com/orgmemory/graphrag/testkit/DeterministicEmbeddingPort.javacomponents/graph-rag-testkit/src/test/java/com/orgmemory/graphrag/testkit/LightRagChunkerGoldenTests.javacomponents/graph-rag-testkit/src/test/java/com/orgmemory/graphrag/testkit/ProcessingRegistryConformanceTests.javacomponents/graph-rag-testkit/src/test/resources/lightrag-v1.5.4/fixed-token-golden.propertiescore/src/main/java/com/orgmemory/core/knowledge/ConnectorSourceRevisionCoordinator.javacore/src/main/java/com/orgmemory/core/knowledge/DocumentProcessingProfileSnapshot.javacore/src/main/java/com/orgmemory/core/knowledge/KnowledgeChunkDraft.javacore/src/main/java/com/orgmemory/core/knowledge/KnowledgeChunkDraftAssembler.javacore/src/main/java/com/orgmemory/core/knowledge/KnowledgeChunkProjectionStore.javacore/src/main/java/com/orgmemory/core/knowledge/KnowledgeTextChunk.javacore/src/main/java/com/orgmemory/core/knowledge/KnowledgeTextChunker.javacore/src/main/java/com/orgmemory/core/knowledge/KnowledgeTextDocument.javacore/src/main/java/com/orgmemory/core/knowledge/SourceIngestionCoordinator.javacore/src/main/java/com/orgmemory/core/knowledge/SourceRevision.javacore/src/main/resources/db/migration/V30__document_processing_profiles_and_chunk_provenance.sqlintegrations/graph-rag-spring-ai/build.gradle.ktsintegrations/graph-rag-spring-ai/src/main/java/com/orgmemory/integrations/graphrag/springai/JtokkitTextTokenizer.javaintegrations/graph-rag-spring-ai/src/main/java/com/orgmemory/integrations/graphrag/springai/SpringAiTextEmbeddingPort.javaintegrations/graph-rag-spring-ai/src/main/java/com/orgmemory/integrations/graphrag/springai/TokenSourceMappingException.javaintegrations/graph-rag-spring-ai/src/test/java/com/orgmemory/integrations/graphrag/springai/JtokkitTextTokenizerTests.javaintegrations/graph-rag-spring-ai/src/test/java/com/orgmemory/integrations/graphrag/springai/SpringAiTextEmbeddingPortTests.java
💤 Files with no reviewable changes (4)
- core/src/main/java/com/orgmemory/core/knowledge/KnowledgeTextChunker.java
- apps/worker/src/test/java/com/orgmemory/worker/ingestion/SpringAiKnowledgeTextChunkerTests.java
- core/src/main/java/com/orgmemory/core/knowledge/KnowledgeTextDocument.java
- apps/worker/src/main/java/com/orgmemory/worker/ingestion/SpringAiKnowledgeTextChunker.java
📜 Review details
⏰ Context from checks skipped due to timeout. (1)
- GitHub Check: Backend · Java 25
🧰 Additional context used
📓 Path-based instructions (6)
**/*
📄 CodeRabbit inference engine (CLAUDE.md)
**/*: Before changing unfamiliar Spring Boot 4, Spring Modulith 2, Spring AI 2, Gradle, React, Vite, Tailwind, or TypeScript APIs, consult Context7/current official documentation and the projectorgmemory-*verification skills.
Readdocs/guidelines/agent-safety.mdbefore retrieval, AI, MCP, permission, upload, graph, or export work.
Never commit.envfiles, provider keys, tokens, or customer data.
Run the relevant gates fromdocs/guidelines/testing-harness.md; use a terminating clean test as the context gate, and do not treatbootRunas verification.
Current behavior belongs in architecture/specs only after it exists in code; intent belongs in vision, roadmap, or an active increment, and repository state must not be duplicated across documents.
Files:
apps/worker/src/main/java/com/orgmemory/worker/ingestion/DocumentParsingException.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/ChunkerPlugin.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/ParserPlugin.javacomponents/graph-rag-testkit/src/test/resources/lightrag-v1.5.4/fixed-token-golden.propertiescomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/SemanticEmbeddingUnavailableException.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/TokenRangeLocator.javaintegrations/graph-rag-spring-ai/src/main/java/com/orgmemory/integrations/graphrag/springai/TokenSourceMappingException.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/ChunkTokenLimitExceededException.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/TextTokenizer.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/ChunkerOptions.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/ParserUnavailableException.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/processing/ProcessingComponentRef.javacore/src/main/resources/db/migration/V30__document_processing_profiles_and_chunk_provenance.sqlcomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/DocumentBlockKind.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/DocumentParser.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/ChunkingRequest.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/TextChunker.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/RecursiveCharacterOptions.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/ChunkerRegistry.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/TextEmbeddingPort.javaapps/worker/src/main/java/com/orgmemory/worker/ingestion/ProcessedSourceDocument.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/ChunkProvenance.javaapps/worker/src/main/resources/application.ymlcomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/ParagraphSemanticOptions.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/DocumentParseResult.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/ChunkedText.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/ParserRegistry.javacomponents/graph-rag-testkit/src/main/java/com/orgmemory/graphrag/testkit/CodePointTokenizer.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/PassthroughParser.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/SourceSpan.javacore/src/main/java/com/orgmemory/core/knowledge/ConnectorSourceRevisionCoordinator.javacomponents/graph-rag-testkit/src/main/java/com/orgmemory/graphrag/testkit/DeterministicEmbeddingPort.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/SemanticEmbeddingInvocationException.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/ChunkProvenanceFactory.javaintegrations/graph-rag-spring-ai/src/main/java/com/orgmemory/integrations/graphrag/springai/SpringAiTextEmbeddingPort.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/DocumentBlock.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/ParserSpec.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/ReuseParser.javaapps/worker/src/main/java/com/orgmemory/worker/ingestion/NativeSourceDocumentParser.javaintegrations/graph-rag-spring-ai/src/test/java/com/orgmemory/integrations/graphrag/springai/JtokkitTextTokenizerTests.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/ChunkerRegistrySnapshot.javacore/src/main/java/com/orgmemory/core/knowledge/SourceIngestionCoordinator.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/DocumentParseRequest.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/ParserRegistrySnapshot.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/SemanticVectorOptions.javaintegrations/graph-rag-spring-ai/src/test/java/com/orgmemory/integrations/graphrag/springai/SpringAiTextEmbeddingPortTests.javaapps/worker/src/main/java/com/orgmemory/worker/ingestion/SourceProcessingProperties.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/FixedTokenOptions.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/CanonicalDocument.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/EncodedText.javacore/src/main/java/com/orgmemory/core/knowledge/SourceRevision.javacomponents/graph-rag-testkit/src/test/java/com/orgmemory/graphrag/testkit/ProcessingRegistryConformanceTests.javaintegrations/graph-rag-spring-ai/build.gradle.ktscore/src/main/java/com/orgmemory/core/knowledge/KnowledgeTextChunk.javacore/src/main/java/com/orgmemory/core/knowledge/DocumentProcessingProfileSnapshot.javacore/src/main/java/com/orgmemory/core/knowledge/KnowledgeChunkDraftAssembler.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/processing/ResolvedDocumentProcessingProfile.javacomponents/graph-rag-testkit/src/test/java/com/orgmemory/graphrag/testkit/LightRagChunkerGoldenTests.javaapps/worker/src/main/java/com/orgmemory/worker/ingestion/DocumentProcessingEngine.javacore/src/main/java/com/orgmemory/core/knowledge/KnowledgeChunkProjectionStore.javaintegrations/graph-rag-spring-ai/src/main/java/com/orgmemory/integrations/graphrag/springai/JtokkitTextTokenizer.javacore/src/main/java/com/orgmemory/core/knowledge/KnowledgeChunkDraft.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/FixedTokenChunker.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/SemanticVectorChunker.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/ParagraphSemanticChunker.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/RecursiveCharacterChunker.javaapps/worker/src/main/java/com/orgmemory/worker/ingestion/SourceDocumentReader.javaapps/worker/src/main/java/com/orgmemory/worker/ingestion/SourceIngestionProcessor.java
**/*.{java,sql}
📄 CodeRabbit inference engine (CLAUDE.md)
Pair JPA schema changes with a Flyway migration.
Files:
apps/worker/src/main/java/com/orgmemory/worker/ingestion/DocumentParsingException.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/ChunkerPlugin.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/ParserPlugin.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/SemanticEmbeddingUnavailableException.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/TokenRangeLocator.javaintegrations/graph-rag-spring-ai/src/main/java/com/orgmemory/integrations/graphrag/springai/TokenSourceMappingException.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/ChunkTokenLimitExceededException.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/TextTokenizer.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/ChunkerOptions.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/ParserUnavailableException.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/processing/ProcessingComponentRef.javacore/src/main/resources/db/migration/V30__document_processing_profiles_and_chunk_provenance.sqlcomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/DocumentBlockKind.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/DocumentParser.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/ChunkingRequest.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/TextChunker.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/RecursiveCharacterOptions.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/ChunkerRegistry.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/TextEmbeddingPort.javaapps/worker/src/main/java/com/orgmemory/worker/ingestion/ProcessedSourceDocument.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/ChunkProvenance.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/ParagraphSemanticOptions.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/DocumentParseResult.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/ChunkedText.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/ParserRegistry.javacomponents/graph-rag-testkit/src/main/java/com/orgmemory/graphrag/testkit/CodePointTokenizer.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/PassthroughParser.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/SourceSpan.javacore/src/main/java/com/orgmemory/core/knowledge/ConnectorSourceRevisionCoordinator.javacomponents/graph-rag-testkit/src/main/java/com/orgmemory/graphrag/testkit/DeterministicEmbeddingPort.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/SemanticEmbeddingInvocationException.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/ChunkProvenanceFactory.javaintegrations/graph-rag-spring-ai/src/main/java/com/orgmemory/integrations/graphrag/springai/SpringAiTextEmbeddingPort.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/DocumentBlock.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/ParserSpec.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/ReuseParser.javaapps/worker/src/main/java/com/orgmemory/worker/ingestion/NativeSourceDocumentParser.javaintegrations/graph-rag-spring-ai/src/test/java/com/orgmemory/integrations/graphrag/springai/JtokkitTextTokenizerTests.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/ChunkerRegistrySnapshot.javacore/src/main/java/com/orgmemory/core/knowledge/SourceIngestionCoordinator.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/DocumentParseRequest.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/ParserRegistrySnapshot.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/SemanticVectorOptions.javaintegrations/graph-rag-spring-ai/src/test/java/com/orgmemory/integrations/graphrag/springai/SpringAiTextEmbeddingPortTests.javaapps/worker/src/main/java/com/orgmemory/worker/ingestion/SourceProcessingProperties.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/FixedTokenOptions.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/CanonicalDocument.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/EncodedText.javacore/src/main/java/com/orgmemory/core/knowledge/SourceRevision.javacomponents/graph-rag-testkit/src/test/java/com/orgmemory/graphrag/testkit/ProcessingRegistryConformanceTests.javacore/src/main/java/com/orgmemory/core/knowledge/KnowledgeTextChunk.javacore/src/main/java/com/orgmemory/core/knowledge/DocumentProcessingProfileSnapshot.javacore/src/main/java/com/orgmemory/core/knowledge/KnowledgeChunkDraftAssembler.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/processing/ResolvedDocumentProcessingProfile.javacomponents/graph-rag-testkit/src/test/java/com/orgmemory/graphrag/testkit/LightRagChunkerGoldenTests.javaapps/worker/src/main/java/com/orgmemory/worker/ingestion/DocumentProcessingEngine.javacore/src/main/java/com/orgmemory/core/knowledge/KnowledgeChunkProjectionStore.javaintegrations/graph-rag-spring-ai/src/main/java/com/orgmemory/integrations/graphrag/springai/JtokkitTextTokenizer.javacore/src/main/java/com/orgmemory/core/knowledge/KnowledgeChunkDraft.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/FixedTokenChunker.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/SemanticVectorChunker.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/ParagraphSemanticChunker.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/RecursiveCharacterChunker.javaapps/worker/src/main/java/com/orgmemory/worker/ingestion/SourceDocumentReader.javaapps/worker/src/main/java/com/orgmemory/worker/ingestion/SourceIngestionProcessor.java
**/*.java
📄 CodeRabbit inference engine (CLAUDE.md)
JetBrains IDE inspection is a Java-backend gate only.
Files:
apps/worker/src/main/java/com/orgmemory/worker/ingestion/DocumentParsingException.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/ChunkerPlugin.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/ParserPlugin.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/SemanticEmbeddingUnavailableException.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/TokenRangeLocator.javaintegrations/graph-rag-spring-ai/src/main/java/com/orgmemory/integrations/graphrag/springai/TokenSourceMappingException.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/ChunkTokenLimitExceededException.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/TextTokenizer.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/ChunkerOptions.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/ParserUnavailableException.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/processing/ProcessingComponentRef.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/DocumentBlockKind.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/DocumentParser.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/ChunkingRequest.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/TextChunker.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/RecursiveCharacterOptions.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/ChunkerRegistry.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/TextEmbeddingPort.javaapps/worker/src/main/java/com/orgmemory/worker/ingestion/ProcessedSourceDocument.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/ChunkProvenance.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/ParagraphSemanticOptions.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/DocumentParseResult.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/ChunkedText.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/ParserRegistry.javacomponents/graph-rag-testkit/src/main/java/com/orgmemory/graphrag/testkit/CodePointTokenizer.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/PassthroughParser.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/SourceSpan.javacore/src/main/java/com/orgmemory/core/knowledge/ConnectorSourceRevisionCoordinator.javacomponents/graph-rag-testkit/src/main/java/com/orgmemory/graphrag/testkit/DeterministicEmbeddingPort.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/SemanticEmbeddingInvocationException.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/ChunkProvenanceFactory.javaintegrations/graph-rag-spring-ai/src/main/java/com/orgmemory/integrations/graphrag/springai/SpringAiTextEmbeddingPort.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/DocumentBlock.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/ParserSpec.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/ReuseParser.javaapps/worker/src/main/java/com/orgmemory/worker/ingestion/NativeSourceDocumentParser.javaintegrations/graph-rag-spring-ai/src/test/java/com/orgmemory/integrations/graphrag/springai/JtokkitTextTokenizerTests.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/ChunkerRegistrySnapshot.javacore/src/main/java/com/orgmemory/core/knowledge/SourceIngestionCoordinator.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/DocumentParseRequest.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/ParserRegistrySnapshot.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/SemanticVectorOptions.javaintegrations/graph-rag-spring-ai/src/test/java/com/orgmemory/integrations/graphrag/springai/SpringAiTextEmbeddingPortTests.javaapps/worker/src/main/java/com/orgmemory/worker/ingestion/SourceProcessingProperties.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/FixedTokenOptions.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/CanonicalDocument.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/EncodedText.javacore/src/main/java/com/orgmemory/core/knowledge/SourceRevision.javacomponents/graph-rag-testkit/src/test/java/com/orgmemory/graphrag/testkit/ProcessingRegistryConformanceTests.javacore/src/main/java/com/orgmemory/core/knowledge/KnowledgeTextChunk.javacore/src/main/java/com/orgmemory/core/knowledge/DocumentProcessingProfileSnapshot.javacore/src/main/java/com/orgmemory/core/knowledge/KnowledgeChunkDraftAssembler.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/processing/ResolvedDocumentProcessingProfile.javacomponents/graph-rag-testkit/src/test/java/com/orgmemory/graphrag/testkit/LightRagChunkerGoldenTests.javaapps/worker/src/main/java/com/orgmemory/worker/ingestion/DocumentProcessingEngine.javacore/src/main/java/com/orgmemory/core/knowledge/KnowledgeChunkProjectionStore.javaintegrations/graph-rag-spring-ai/src/main/java/com/orgmemory/integrations/graphrag/springai/JtokkitTextTokenizer.javacore/src/main/java/com/orgmemory/core/knowledge/KnowledgeChunkDraft.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/FixedTokenChunker.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/SemanticVectorChunker.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/ParagraphSemanticChunker.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/RecursiveCharacterChunker.javaapps/worker/src/main/java/com/orgmemory/worker/ingestion/SourceDocumentReader.javaapps/worker/src/main/java/com/orgmemory/worker/ingestion/SourceIngestionProcessor.java
**/*.{yml,yaml,properties}
📄 CodeRabbit inference engine (CLAUDE.md)
Keep
ddl-auto=validatein application configuration.
Files:
components/graph-rag-testkit/src/test/resources/lightrag-v1.5.4/fixed-token-golden.propertiesapps/worker/src/main/resources/application.yml
core/src/main/resources/db/migration/*.sql
⚙️ CodeRabbit configuration file
core/src/main/resources/db/migration/*.sql: Flyway migrations are immutable after release. Check tenant isolation,
foreign keys, uniqueness, indexes, append-only evidence semantics, safe
defaults, and compatibility with PostgreSQL 18 plus pgvector.
Files:
core/src/main/resources/db/migration/V30__document_processing_profiles_and_chunk_provenance.sql
core/src/main/java/com/orgmemory/core/{authorization,knowledge,permission}/**/*.java
⚙️ CodeRabbit configuration file
core/src/main/java/com/orgmemory/core/{authorization,knowledge,permission}/**/*.java: Treat PostgreSQL ACL evidence as canonical and OpenFGA as the relationship
authorization decision point. Authorization must fail closed. Filtering
must happen before ranking, LIMIT, graph traversal, answer generation,
export, and citation rendering. Flag metadata or timing leak paths.
Files:
core/src/main/java/com/orgmemory/core/knowledge/ConnectorSourceRevisionCoordinator.javacore/src/main/java/com/orgmemory/core/knowledge/SourceIngestionCoordinator.javacore/src/main/java/com/orgmemory/core/knowledge/SourceRevision.javacore/src/main/java/com/orgmemory/core/knowledge/KnowledgeTextChunk.javacore/src/main/java/com/orgmemory/core/knowledge/DocumentProcessingProfileSnapshot.javacore/src/main/java/com/orgmemory/core/knowledge/KnowledgeChunkDraftAssembler.javacore/src/main/java/com/orgmemory/core/knowledge/KnowledgeChunkProjectionStore.javacore/src/main/java/com/orgmemory/core/knowledge/KnowledgeChunkDraft.java
🪛 ast-grep (0.44.1)
components/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/SemanticVectorOptions.java
[warning] 26-26: Regular expression is compiled from a non-literal, possibly user-controlled value. A crafted regex (or input matched against one) can trigger catastrophic backtracking and hang the thread (ReDoS). Use a hardcoded literal pattern, wrap untrusted text with Pattern.quote(...), or validate/length-limit the input and enforce a matching timeout before passing it to Pattern.compile / String.matches / String.replaceAll / String.replaceFirst.
Context: Pattern.compile(sentenceSplitRegex)
Note: [CWE-1333] Inefficient Regular Expression Complexity.
(redos-non-literal-regex-java)
components/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/SemanticVectorChunker.java
[warning] 40-40: Regular expression is compiled from a non-literal, possibly user-controlled value. A crafted regex (or input matched against one) can trigger catastrophic backtracking and hang the thread (ReDoS). Use a hardcoded literal pattern, wrap untrusted text with Pattern.quote(...), or validate/length-limit the input and enforce a matching timeout before passing it to Pattern.compile / String.matches / String.replaceAll / String.replaceFirst.
Context: Pattern.compile(options.sentenceSplitRegex())
Note: [CWE-1333] Inefficient Regular Expression Complexity.
(redos-non-literal-regex-java)
🪛 Squawk (2.59.0)
core/src/main/resources/db/migration/V30__document_processing_profiles_and_chunk_provenance.sql
[warning] 3-3: Changing the size of a varchar field requires an ACCESS EXCLUSIVE lock, that will prevent all reads and writes to the table. Use a TEXT field with a CHECK constraint.
(prefer-text-field)
[warning] 4-13: By default new constraints require a table scan and block writes to the table while that scan occurs. Use NOT VALID with a later VALIDATE CONSTRAINT call.
(constraint-missing-not-valid)
[warning] 16-16: Using 32-bit integer fields can result in hitting the max int limit. Use 64-bit integer values instead to prevent hitting this limit.
(prefer-bigint-over-int)
[warning] 17-17: Using 32-bit integer fields can result in hitting the max int limit. Use 64-bit integer values instead to prevent hitting this limit.
(prefer-bigint-over-int)
[warning] 18-18: Using 32-bit integer fields can result in hitting the max int limit. Use 64-bit integer values instead to prevent hitting this limit.
(prefer-bigint-over-int)
[warning] 19-19: Changing the size of a varchar field requires an ACCESS EXCLUSIVE lock, that will prevent all reads and writes to the table. Use a TEXT field with a CHECK constraint.
(prefer-text-field)
[warning] 20-31: By default new constraints require a table scan and block writes to the table while that scan occurs. Use NOT VALID with a later VALIDATE CONSTRAINT call.
(constraint-missing-not-valid)
[warning] 32-34: By default new constraints require a table scan and block writes to the table while that scan occurs. Use NOT VALID with a later VALIDATE CONSTRAINT call.
(constraint-missing-not-valid)
🔇 Additional comments (59)
components/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/SourceSpan.java (1)
1-10: LGTM!integrations/graph-rag-spring-ai/build.gradle.kts (1)
8-8: LGTM!integrations/graph-rag-spring-ai/src/main/java/com/orgmemory/integrations/graphrag/springai/JtokkitTextTokenizer.java (1)
23-49: LGTM!integrations/graph-rag-spring-ai/src/main/java/com/orgmemory/integrations/graphrag/springai/TokenSourceMappingException.java (1)
3-7: LGTM!integrations/graph-rag-spring-ai/src/main/java/com/orgmemory/integrations/graphrag/springai/SpringAiTextEmbeddingPort.java (1)
37-68: LGTM!integrations/graph-rag-spring-ai/src/test/java/com/orgmemory/integrations/graphrag/springai/JtokkitTextTokenizerTests.java (1)
14-31: LGTM!integrations/graph-rag-spring-ai/src/test/java/com/orgmemory/integrations/graphrag/springai/SpringAiTextEmbeddingPortTests.java (1)
18-42: LGTM!components/graph-rag-testkit/src/main/java/com/orgmemory/graphrag/testkit/CodePointTokenizer.java (1)
21-40: LGTM!components/graph-rag-testkit/src/main/java/com/orgmemory/graphrag/testkit/DeterministicEmbeddingPort.java (1)
17-30: LGTM!components/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/CanonicalDocument.java (1)
34-64: LGTM!components/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/DocumentBlock.java (1)
8-38: LGTM!components/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/DocumentBlockKind.java (1)
3-10: LGTM!components/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/DocumentParseRequest.java (1)
6-35: LGTM!components/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/DocumentParseResult.java (1)
7-19: LGTM!components/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/DocumentParser.java (1)
5-11: LGTM!components/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/ParserPlugin.java (1)
3-7: LGTM!components/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/TextTokenizer.java (1)
5-15: LGTM!components/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/TokenRangeLocator.java (1)
3-12: LGTM!components/graph-rag-testkit/src/test/java/com/orgmemory/graphrag/testkit/LightRagChunkerGoldenTests.java (1)
29-233: LGTM!components/graph-rag-testkit/src/test/resources/lightrag-v1.5.4/fixed-token-golden.properties (1)
1-7: LGTM!components/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/ParserRegistry.java (1)
11-33: LGTM!components/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/ParserRegistrySnapshot.java (1)
29-49: LGTM!components/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/ParserSpec.java (1)
17-43: LGTM!components/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/ParserUnavailableException.java (1)
3-8: LGTM!components/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/PassthroughParser.java (1)
7-24: LGTM!components/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/ChunkTokenLimitExceededException.java (1)
3-21: LGTM!components/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/EncodedText.java (1)
20-104: LGTM!components/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/TextChunker.java (1)
6-13: LGTM!components/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/TextEmbeddingPort.java (1)
8-13: LGTM!components/graph-rag-core/src/main/java/com/orgmemory/graphrag/parsing/ReuseParser.java (1)
1-25: LGTM!components/graph-rag-core/src/main/java/com/orgmemory/graphrag/processing/ProcessingComponentRef.java (1)
1-32: LGTM!components/graph-rag-core/src/main/java/com/orgmemory/graphrag/processing/ResolvedDocumentProcessingProfile.java (1)
41-135: LGTM!components/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/ChunkingRequest.java (1)
1-17: LGTM!components/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/ChunkerPlugin.java (1)
1-7: LGTM!components/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/ChunkerRegistry.java (1)
13-33: LGTM!components/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/ChunkerRegistrySnapshot.java (1)
26-51: LGTM!components/graph-rag-testkit/src/test/java/com/orgmemory/graphrag/testkit/ProcessingRegistryConformanceTests.java (1)
23-55: LGTM!Also applies to: 65-93
components/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/ChunkProvenanceFactory.java (1)
1-49: LGTM!components/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/ChunkProvenance.java (1)
14-36: LGTM!components/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/FixedTokenOptions.java (1)
9-25: LGTM!components/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/FixedTokenChunker.java (1)
26-92: LGTM!components/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/RecursiveCharacterOptions.java (1)
15-21: LGTM!components/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/ParagraphSemanticOptions.java (1)
13-20: LGTM!components/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/RecursiveCharacterChunker.java (1)
26-249: LGTM!components/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/SemanticVectorOptions.java (1)
26-27: 🚀 Performance & Scalability | ⚡ Quick winRetain and reuse the compiled sentence-split
Pattern.SemanticVectorOptionscompiles the regex only to validate it and discards the result, soSemanticVectorChunkerrecompiles the same pattern on everychunk()invocation. Compile once and reuse to avoid the redundant work and keep the ReDoS surface in a single validated place.
components/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/SemanticVectorOptions.java#L26-L27: store the compiledPattern(e.g., an additional field or a package-visible accessor) instead of discardingPattern.compile(sentenceSplitRegex).components/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/SemanticVectorChunker.java#L40-L41: use the pre-compiledPatternfrom options rather than callingPattern.compile(options.sentenceSplitRegex())again.components/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/SemanticEmbeddingInvocationException.java (1)
3-8: LGTM!components/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/SemanticEmbeddingUnavailableException.java (1)
3-8: LGTM!components/graph-rag-core/src/main/java/com/orgmemory/graphrag/chunking/ParagraphSemanticChunker.java (1)
100-104: 🩺 Stability & AvailabilityNo change needed.
DocumentBlock.headingLevelrejects values outside1..6, so the fixed-size heading array is protected at construction time.apps/worker/src/main/java/com/orgmemory/worker/ingestion/DocumentParsingException.java (1)
1-9: LGTM!apps/worker/src/main/java/com/orgmemory/worker/ingestion/NativeSourceDocumentParser.java (1)
15-40: LGTM!apps/worker/src/main/java/com/orgmemory/worker/ingestion/SourceIngestionProcessor.java (1)
247-259: 🗄️ Data Integrity & IntegrationConfirm canonical/normalized text is the intended "raw" payload for revision hashing.
registerRawSourcenow sources its content fromparsed.document().content()— the canonicalized text — rather than the original uploaded bytes. PerKnowledgeIngestionService, this content is hashed intopayloadShaand used to detect "same revision" uploads. If parsing/normalization is not perfectly deterministic across re-processing, or if two distinct source files normalize to the same canonical text, revision identity and evidence provenance could be affected. Please confirm this is intentional given the "raw source" naming implies original bytes.apps/worker/src/main/resources/application.yml (1)
68-78: LGTM!core/src/main/java/com/orgmemory/core/knowledge/DocumentProcessingProfileSnapshot.java (1)
10-45: LGTM!core/src/main/java/com/orgmemory/core/knowledge/KnowledgeChunkDraft.java (1)
13-47: LGTM!core/src/main/java/com/orgmemory/core/knowledge/KnowledgeChunkDraftAssembler.java (1)
40-52: LGTM!core/src/main/java/com/orgmemory/core/knowledge/KnowledgeChunkProjectionStore.java (1)
18-38: LGTM!Also applies to: 92-118
core/src/main/java/com/orgmemory/core/knowledge/KnowledgeTextChunk.java (1)
6-25: LGTM!core/src/main/java/com/orgmemory/core/knowledge/SourceIngestionCoordinator.java (1)
69-93: LGTM!core/src/main/java/com/orgmemory/core/knowledge/SourceRevision.java (1)
78-82: LGTM!Also applies to: 164-181, 271-277
Owner
Author
|
Addressed the completed CodeRabbit pass in commit
The parser-priority comment was verified as a false positive and answered inline: the registry is insertion ordered and Full local gate after fixes: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
LightRAG parity evidence
Verification
./gradlew.bat --no-daemon clean testgit diff --checkgatesMerge flow
This PR targets
light-rag. It must not merge intomain;mainwill only receive the completed program after PR12.Summary by CodeRabbit