Add secure source ingestion pipeline - #7
Conversation
|
Warning Review limit reached
Next review available in: 25 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (23)
📝 WalkthroughWalkthroughThis PR adds authenticated document upload and source listing, MinIO-backed immutable evidence storage, durable worker ingestion with parsing, chunking, embeddings and retries, PostgreSQL schemas and projections, and a Sources workspace with upload, filtering, status, and polling UI. ChangesSecure knowledge ingestion vertical slice
Estimated code review effort: 5 (Critical) | ~120 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
@coderabbitai review |
✅ Action performedReview finished.
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 16
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@apps/worker/src/main/java/com/orgmemory/worker/ingestion/SourceDocumentReader.java`:
- Around line 42-46: Update isPlainText in SourceDocumentReader so a filename
extension cannot override a detected non-text media type; require the detected
type to be plain text, using the filename only for the intended unknown-type
fallback consistent with isAllowed. Ensure detected OOXML docx/pptx continues
through TikaDocumentReader rather than Files.readString.
In
`@apps/worker/src/main/java/com/orgmemory/worker/ingestion/SourceIngestionProcessor.java`:
- Around line 90-107: Update process and copyAndVerify so every temporary file
created during copying is tracked and deleted when validation fails, including
RejectedSourceException and IOException paths. Ensure the cleanup does not
depend solely on copyAndVerify returning successfully, while preserving the
existing successful-ingestion flow and finally cleanup behavior.
In
`@apps/worker/src/main/java/com/orgmemory/worker/OrgMemoryWorkerApplication.java`:
- Around line 19-31: Update OrgMemoryWorkerApplication’s core bean registration
to use a narrow `@ComponentScan` covering the worker’s required core sub-packages,
such as knowledge, permission, and shared, instead of relying on the
individual-class `@Import` list. Keep entity and repository scanning unchanged,
and remove redundant imported component classes from the manual list.
In `@compose.yaml`:
- Around line 30-35: Update the MinIO service environment in the Compose
configuration to require injected values for MINIO_ROOT_USER and
MINIO_ROOT_PASSWORD by removing their predictable fallbacks. Since the service
is local-development only, bind both published ports to loopback while
preserving their container ports.
In `@core/src/main/java/com/orgmemory/core/knowledge/EmbeddingProfileSpec.java`:
- Around line 20-22: Update EmbeddingProfileSpec.profileKey() to serialize
provider and model unambiguously so slash-containing values cannot produce the
same key as different component combinations. Encode each delimiter-sensitive
component or validate and reject values containing the delimiter, while
preserving dimensions and normalized distanceMetric in the key.
In
`@core/src/main/java/com/orgmemory/core/knowledge/KnowledgeChunkProjectionStore.java`:
- Around line 39-82: Replace the per-chunk `.update()` loop in the chunk
projection method with a batched insert using JdbcTemplate.batchUpdate or the
repository’s supported JdbcClient batching API. Preserve the existing field
mappings, embedding validation, generated IDs, timestamps, and parameter types
while submitting all KnowledgeChunkDraft rows for the revision as one batch.
- Around line 35-37: Update the DELETE statement in
KnowledgeChunkProjectionStore to include an organization_id predicate bound to
the current organizationId alongside source_revision_id. Reuse the existing
organizationId value and parameter binding, preserving fail-closed tenant
scoping.
In `@core/src/main/java/com/orgmemory/core/knowledge/SourceIngestionJob.java`:
- Around line 114-120: Replace the duplicate private truncate(String value)
implementations with one shared package-private utility preserving null
handling, stripping, and the 512-character limit. Update SourceIngestionJob.java
lines 114-120 and SourceRevision.java lines 164-170 to delegate to that utility,
removing both local copies.
In `@core/src/main/java/com/orgmemory/core/knowledge/SourceQueryService.java`:
- Around line 24-36: Update SourceQueryService.listOwn to obtain authorized
source IDs through the canonical PostgreSQL ACL evidence and OpenFGA
relationship authorization path before loading revisions or rendering summaries.
Filter the repository results to those IDs, fail closed when authorization
cannot be established, and preserve the existing summary mapping only for
authorized sources.
In `@core/src/main/java/com/orgmemory/core/knowledge/storage/ObjectKey.java`:
- Around line 5-13: Update the compact constructor of ObjectKey to reject any
path containing a ".." segment, including the bare value "..", rather than
relying only on startsWith, contains("../"), and endsWith("/..") checks. Apply
segment-based validation after normalizing separators and trimming, while
preserving the existing relative canonical-path validation and exception
behavior.
In `@docs/decisions/0008-worker-owns-ingestion-and-derived-indexes.md`:
- Around line 20-24: Update EmbeddingProfileRegistry.resolve to detect when an
existing organization/profile key’s provider, model, dimensions, or distance
metric differs from the requested configuration, and fail deterministically
instead of silently returning the existing row; unchanged configurations should
continue resolving successfully. Document in this decision that profile keys are
immutable and changed configurations require a new key or produce an error.
In `@docs/increments/active/2026-07-20-secure-knowledge-vertical-slice/design.md`:
- Around line 47-68: Reduce this design section to the slice-specific choice:
use OpenAI text-embedding-3-large at 1536 dimensions for the first PostgreSQL
HNSW projection. Remove duplicated normative rules about EmbeddingProfile
fields, partial indexes, query pinning, and cross-profile ranking, and link to
the canonical contract in the referenced decision/specification documents.
Preserve the thin-map documentation harness and identify the canonical
architecture/specification records as the source of truth.
In `@web/src/features/session/csrf-fetch.ts`:
- Around line 6-22: Update getBrowserCsrfHeader to cache the header name
returned by getBrowserCsrfToken after the first successful request, reusing that
cached value on subsequent calls instead of making redundant API requests.
Preserve the existing cookie lookup, decoding, and error behavior.
In `@web/src/features/sources/api/upload-source.ts`:
- Around line 9-19: Update uploadSourceWithCsrf to remove the
getBrowserCsrfHeader call and the manually constructed headers option. Invoke
uploadSource with only the existing body, query, and throwOnError options,
relying on the global csrfFetch interceptor for CSRF handling.
In `@web/src/features/sources/components/source-upload-dialog.tsx`:
- Around line 70-98: Update the file Input and Classification Select in the
source upload dialog to be disabled whenever pending is true, matching the
existing footer button behavior. Apply the disabled state to the Select
component and the file input without changing their current value or change
handlers.
In `@web/src/features/sources/components/sources-page.tsx`:
- Around line 56-59: Remove the redundant dark-mode variants from the className
on the documents TabsTrigger, keeping the base
data-[state=active]:bg-emerald-500 and data-[state=active]:text-white utilities
unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 7bb5f197-3288-43d5-80fa-bcc815472df0
⛔ Files ignored due to path filters (2)
contracts/openapi.jsonis excluded by!contracts/openapi.jsonweb/pnpm-lock.yamlis excluded by!**/pnpm-lock.yaml,!**/pnpm-lock.yaml
📒 Files selected for processing (86)
apps/api/build.gradle.ktsapps/api/src/main/java/com/orgmemory/api/OrgMemoryApiApplication.javaapps/api/src/main/java/com/orgmemory/api/source/SourceController.javaapps/api/src/main/java/com/orgmemory/api/source/SourceResponse.javaapps/api/src/main/resources/application.ymlapps/api/src/test/java/com/orgmemory/api/OrgMemoryApiContextLoadTests.javaapps/api/src/test/java/com/orgmemory/api/capability/CapabilityAssetServiceIntegrationTests.javaapps/api/src/test/java/com/orgmemory/api/knowledge/KnowledgeIngestionIntegrationTests.javaapps/api/src/test/java/com/orgmemory/api/knowledge/KnowledgeRetrievalIntegrationTests.javaapps/api/src/test/java/com/orgmemory/api/permission/PermissionAuditIntegrationTests.javaapps/api/src/test/java/com/orgmemory/api/source/SourceUploadIntegrationTests.javaapps/worker/build.gradle.ktsapps/worker/src/main/java/com/orgmemory/worker/OrgMemoryWorkerApplication.javaapps/worker/src/main/java/com/orgmemory/worker/ingestion/ParsedSource.javaapps/worker/src/main/java/com/orgmemory/worker/ingestion/RejectedSourceException.javaapps/worker/src/main/java/com/orgmemory/worker/ingestion/SourceDocumentReader.javaapps/worker/src/main/java/com/orgmemory/worker/ingestion/SourceIngestionProcessor.javaapps/worker/src/main/java/com/orgmemory/worker/ingestion/SourceIngestionScheduler.javaapps/worker/src/main/java/com/orgmemory/worker/ingestion/SourceProcessingProperties.javaapps/worker/src/main/resources/application.ymlapps/worker/src/test/java/com/orgmemory/worker/ingestion/SourceDocumentReaderTests.javaapps/worker/src/test/java/com/orgmemory/worker/ingestion/SourceIngestionPipelineIntegrationTests.javacompose.yamlcore/src/main/java/com/orgmemory/core/knowledge/ClaimedSourceRevision.javacore/src/main/java/com/orgmemory/core/knowledge/CreateUploadSourceCommand.javacore/src/main/java/com/orgmemory/core/knowledge/EmbeddingDistanceMetric.javacore/src/main/java/com/orgmemory/core/knowledge/EmbeddingProfile.javacore/src/main/java/com/orgmemory/core/knowledge/EmbeddingProfileRef.javacore/src/main/java/com/orgmemory/core/knowledge/EmbeddingProfileRegistry.javacore/src/main/java/com/orgmemory/core/knowledge/EmbeddingProfileRepository.javacore/src/main/java/com/orgmemory/core/knowledge/EmbeddingProfileSpec.javacore/src/main/java/com/orgmemory/core/knowledge/EvidenceBlob.javacore/src/main/java/com/orgmemory/core/knowledge/EvidenceBlobRepository.javacore/src/main/java/com/orgmemory/core/knowledge/EvidenceScanStatus.javacore/src/main/java/com/orgmemory/core/knowledge/KnowledgeChunkDraft.javacore/src/main/java/com/orgmemory/core/knowledge/KnowledgeChunkProjectionStore.javacore/src/main/java/com/orgmemory/core/knowledge/SourceIngestionCoordinator.javacore/src/main/java/com/orgmemory/core/knowledge/SourceIngestionJob.javacore/src/main/java/com/orgmemory/core/knowledge/SourceIngestionJobRepository.javacore/src/main/java/com/orgmemory/core/knowledge/SourceIngestionJobStatus.javacore/src/main/java/com/orgmemory/core/knowledge/SourceIngestionProperties.javacore/src/main/java/com/orgmemory/core/knowledge/SourceObject.javacore/src/main/java/com/orgmemory/core/knowledge/SourceObjectRepository.javacore/src/main/java/com/orgmemory/core/knowledge/SourceObjectStatus.javacore/src/main/java/com/orgmemory/core/knowledge/SourceQueryService.javacore/src/main/java/com/orgmemory/core/knowledge/SourceRevision.javacore/src/main/java/com/orgmemory/core/knowledge/SourceRevisionRepository.javacore/src/main/java/com/orgmemory/core/knowledge/SourceRevisionStatus.javacore/src/main/java/com/orgmemory/core/knowledge/SourceSummary.javacore/src/main/java/com/orgmemory/core/knowledge/SourceType.javacore/src/main/java/com/orgmemory/core/knowledge/SourceUploadRegistrationService.javacore/src/main/java/com/orgmemory/core/knowledge/SourceUploadService.javacore/src/main/java/com/orgmemory/core/knowledge/storage/ObjectContent.javacore/src/main/java/com/orgmemory/core/knowledge/storage/ObjectKey.javacore/src/main/java/com/orgmemory/core/knowledge/storage/ObjectStorageException.javacore/src/main/java/com/orgmemory/core/knowledge/storage/ObjectStoragePort.javacore/src/main/java/com/orgmemory/core/knowledge/storage/ObjectWriteRequest.javacore/src/main/java/com/orgmemory/core/knowledge/storage/StoredObject.javacore/src/main/java/com/orgmemory/core/shared/JpaAuditingConfig.javacore/src/main/resources/db/migration/V15__source_upload_and_embedding_pipeline.sqldocs/conventions.mddocs/decisions/0005-secure-java-graph-kernel.mddocs/decisions/0008-worker-owns-ingestion-and-derived-indexes.mddocs/increments/active/2026-07-20-secure-knowledge-vertical-slice/design.mddocs/increments/active/2026-07-20-secure-knowledge-vertical-slice/plan.mddocs/specs/domains/knowledge-ingestion.mdgradle/libs.versions.tomlintegrations/object-storage-minio/build.gradle.ktsintegrations/object-storage-minio/src/main/java/com/orgmemory/integrations/storage/minio/MinioObjectStorageAdapter.javaintegrations/object-storage-minio/src/main/java/com/orgmemory/integrations/storage/minio/MinioObjectStorageAutoConfiguration.javaintegrations/object-storage-minio/src/main/java/com/orgmemory/integrations/storage/minio/MinioObjectStorageProperties.javaintegrations/object-storage-minio/src/main/resources/META-INF/spring/org.springframework.boot.autoconfigure.AutoConfiguration.importssettings.gradle.ktsweb/package.jsonweb/src/components/app-shell/app-shell.tsxweb/src/components/app-shell/app-sidebar.tsxweb/src/features/session/csrf-fetch.tsweb/src/features/sources/api/upload-source.tsweb/src/features/sources/components/source-status-badge.tsxweb/src/features/sources/components/source-upload-dialog.tsxweb/src/features/sources/components/sources-page.tsxweb/src/features/sources/components/sources-table.tsxweb/src/features/sources/source-status.tsweb/src/features/sources/store/document-manager-store.tsweb/src/routeTree.gen.tsweb/src/routes/_authenticated/sources.tsx
📜 Review details
⏰ Context from checks skipped due to timeout. (1)
- GitHub Check: Backend · Java 25
🧰 Additional context used
📓 Path-based instructions (5)
core/src/main/java/com/orgmemory/core/{authorization,knowledge,permission}/**/*.java
⚙️ CodeRabbit configuration file
core/src/main/java/com/orgmemory/core/{authorization,knowledge,permission}/**/*.java: Treat PostgreSQL ACL evidence as canonical and OpenFGA as the relationship
authorization decision point. Authorization must fail closed. Filtering
must happen before ranking, LIMIT, graph traversal, answer generation,
export, and citation rendering. Flag metadata or timing leak paths.
Files:
core/src/main/java/com/orgmemory/core/knowledge/SourceRevisionStatus.javacore/src/main/java/com/orgmemory/core/knowledge/EmbeddingProfileRef.javacore/src/main/java/com/orgmemory/core/knowledge/SourceObjectStatus.javacore/src/main/java/com/orgmemory/core/knowledge/storage/ObjectStorageException.javacore/src/main/java/com/orgmemory/core/knowledge/CreateUploadSourceCommand.javacore/src/main/java/com/orgmemory/core/knowledge/SourceObjectRepository.javacore/src/main/java/com/orgmemory/core/knowledge/storage/StoredObject.javacore/src/main/java/com/orgmemory/core/knowledge/SourceRevisionRepository.javacore/src/main/java/com/orgmemory/core/knowledge/EvidenceScanStatus.javacore/src/main/java/com/orgmemory/core/knowledge/SourceType.javacore/src/main/java/com/orgmemory/core/knowledge/SourceIngestionJobStatus.javacore/src/main/java/com/orgmemory/core/knowledge/ClaimedSourceRevision.javacore/src/main/java/com/orgmemory/core/knowledge/EvidenceBlobRepository.javacore/src/main/java/com/orgmemory/core/knowledge/storage/ObjectContent.javacore/src/main/java/com/orgmemory/core/knowledge/EmbeddingDistanceMetric.javacore/src/main/java/com/orgmemory/core/knowledge/SourceSummary.javacore/src/main/java/com/orgmemory/core/knowledge/EmbeddingProfileSpec.javacore/src/main/java/com/orgmemory/core/knowledge/KnowledgeChunkProjectionStore.javacore/src/main/java/com/orgmemory/core/knowledge/KnowledgeChunkDraft.javacore/src/main/java/com/orgmemory/core/knowledge/SourceIngestionProperties.javacore/src/main/java/com/orgmemory/core/knowledge/storage/ObjectStoragePort.javacore/src/main/java/com/orgmemory/core/knowledge/EmbeddingProfileRepository.javacore/src/main/java/com/orgmemory/core/knowledge/SourceObject.javacore/src/main/java/com/orgmemory/core/knowledge/storage/ObjectWriteRequest.javacore/src/main/java/com/orgmemory/core/knowledge/SourceQueryService.javacore/src/main/java/com/orgmemory/core/knowledge/EmbeddingProfileRegistry.javacore/src/main/java/com/orgmemory/core/knowledge/SourceUploadRegistrationService.javacore/src/main/java/com/orgmemory/core/knowledge/EvidenceBlob.javacore/src/main/java/com/orgmemory/core/knowledge/SourceIngestionJobRepository.javacore/src/main/java/com/orgmemory/core/knowledge/storage/ObjectKey.javacore/src/main/java/com/orgmemory/core/knowledge/SourceIngestionJob.javacore/src/main/java/com/orgmemory/core/knowledge/EmbeddingProfile.javacore/src/main/java/com/orgmemory/core/knowledge/SourceUploadService.javacore/src/main/java/com/orgmemory/core/knowledge/SourceIngestionCoordinator.javacore/src/main/java/com/orgmemory/core/knowledge/SourceRevision.java
web/src/**/*.{ts,tsx}
⚙️ CodeRabbit configuration file
web/src/**/*.{ts,tsx}: OAuth access and refresh tokens must never enter browser JavaScript or
browser storage. Use the HttpOnly BFF session, CSRF-protected mutations,
generated Hey API data clients, accessible states, and both light and
dark themes. Handwritten transport is reserved for documented protocol
flows such as navigation redirects and streaming.
Files:
web/src/features/sources/api/upload-source.tsweb/src/features/sources/components/source-status-badge.tsxweb/src/routes/_authenticated/sources.tsxweb/src/components/app-shell/app-shell.tsxweb/src/features/sources/store/document-manager-store.tsweb/src/components/app-shell/app-sidebar.tsxweb/src/features/sources/components/sources-page.tsxweb/src/features/sources/components/source-upload-dialog.tsxweb/src/features/session/csrf-fetch.tsweb/src/features/sources/components/sources-table.tsxweb/src/routeTree.gen.tsweb/src/features/sources/source-status.ts
apps/api/src/main/java/**/*.java
⚙️ CodeRabbit configuration file
apps/api/src/main/java/**/*.java: Enforce the browser-BFF and resource-server boundaries. Authentication
must resolve an active internal actor through the explicit issuer and
subject binding. Reject identity, tenant, roles, or permissions supplied
by request payloads, JWT email, or untrusted JWT role claims.
Files:
apps/api/src/main/java/com/orgmemory/api/source/SourceResponse.javaapps/api/src/main/java/com/orgmemory/api/source/SourceController.javaapps/api/src/main/java/com/orgmemory/api/OrgMemoryApiApplication.java
docs/**/*.md
⚙️ CodeRabbit configuration file
docs/**/*.md: Preserve the thin-map documentation harness. Current facts belong in
architecture, specs, and tests only after implementation; future intent
belongs in vision, roadmap, or one active increment. Flag duplicated or
contradictory sources of truth.
Files:
docs/decisions/0008-worker-owns-ingestion-and-derived-indexes.mddocs/decisions/0005-secure-java-graph-kernel.mddocs/increments/active/2026-07-20-secure-knowledge-vertical-slice/design.mddocs/conventions.mddocs/specs/domains/knowledge-ingestion.mddocs/increments/active/2026-07-20-secure-knowledge-vertical-slice/plan.md
core/src/main/resources/db/migration/*.sql
⚙️ CodeRabbit configuration file
core/src/main/resources/db/migration/*.sql: Flyway migrations are immutable after release. Check tenant isolation,
foreign keys, uniqueness, indexes, append-only evidence semantics, safe
defaults, and compatibility with PostgreSQL 18 plus pgvector.
Files:
core/src/main/resources/db/migration/V15__source_upload_and_embedding_pipeline.sql
🧠 Learnings (1)
📓 Common learnings
Learnt from: CR
Repo: kl3inIT/OrgMemory
Timestamp: 2026-07-21T17:26:28.730Z
Learning: Treat `ARCHITECTURE.md` and specs as records of implemented facts; keep vision and roadmap for intent, active increments for design and execution plans, and explicitly supersede append-only decisions rather than rewriting them.
Learnt from: CR
Repo: kl3inIT/OrgMemory
Timestamp: 2026-07-21T17:26:28.730Z
Learning: Do not create status-summary documents that duplicate existing sources of truth.
Learnt from: CR
Repo: kl3inIT/OrgMemory
Timestamp: 2026-07-21T17:26:28.730Z
Learning: Create a Gradle subproject only for deployability, reusable engine code, or a replaceable external integration.
Learnt from: CR
Repo: kl3inIT/OrgMemory
Timestamp: 2026-07-21T17:26:28.730Z
Learning: The API runs migrations; workers and MCP do not own schema evolution.
Learnt from: CR
Repo: kl3inIT/OrgMemory
Timestamp: 2026-07-21T17:26:28.730Z
Learning: Keep provider credentials outside clients, logs, documentation, and git.
Learnt from: CR
Repo: kl3inIT/OrgMemory
Timestamp: 2026-07-21T17:26:28.730Z
Learning: Treat the current prototype page structure as non-contractual.
Learnt from: CR
Repo: kl3inIT/OrgMemory
Timestamp: 2026-07-21T17:26:28.730Z
Learning: Run the narrowest tests while iterating and completion gates before handoff.
Learnt from: CR
Repo: kl3inIT/OrgMemory
Timestamp: 2026-07-21T17:26:28.730Z
Learning: Use a real browser flow when UI behavior changes; do not claim runtime behavior from compilation alone.
🪛 ast-grep (0.44.1)
apps/api/src/test/java/com/orgmemory/api/source/SourceUploadIntegrationTests.java
[warning] 66-66: Use a randomly-generated IV
Context: byte[] content = "Approved onboarding workflow".getBytes(StandardCharsets.UTF_8);
Note: [CWE-329] Generation of Predictable IV with CBC Mode.
(random-iv)
apps/worker/src/test/java/com/orgmemory/worker/ingestion/SourceIngestionPipelineIntegrationTests.java
[warning] 80-81: Use a randomly-generated IV
Context: byte[] content = "Open the customer request, verify the account, and record the resolution."
.getBytes(StandardCharsets.UTF_8);
Note: [CWE-329] Generation of Predictable IV with CBC Mode.
(random-iv)
apps/worker/src/main/java/com/orgmemory/worker/ingestion/SourceIngestionProcessor.java
[warning] 222-222: Temporary file not deleted
Context: Files.createTempFile("orgmemory-ingestion-", suffix)
Note: [CWE-377] Insecure Temporary File. Security best practice.
(tempfile-delete)
🪛 SQLFluff (4.2.2)
core/src/main/resources/db/migration/V15__source_upload_and_embedding_pipeline.sql
[error] 166-169: ADD CONSTRAINT ... FOREIGN KEY should use NOT VALID to avoid locking the table while validating existing rows.
(PG01)
[error] 246-247: CREATE INDEX should use CONCURRENTLY to avoid locking the table during the build.
(PG01)
[error] 248-249: CREATE INDEX should use CONCURRENTLY to avoid locking the table during the build.
(PG01)
[error] 250-252: CREATE INDEX should use CONCURRENTLY to avoid locking the table during the build.
(PG01)
[error] 253-254: CREATE INDEX should use CONCURRENTLY to avoid locking the table during the build.
(PG01)
[error] 255-257: CREATE INDEX should use CONCURRENTLY to avoid locking the table during the build.
(PG01)
🪛 Squawk (2.59.0)
core/src/main/resources/db/migration/V15__source_upload_and_embedding_pipeline.sql
[warning] 6-6: Changing the size of a varchar field requires an ACCESS EXCLUSIVE lock, that will prevent all reads and writes to the table. Use a TEXT field with a CHECK constraint.
(prefer-text-field)
[warning] 7-7: Changing the size of a varchar field requires an ACCESS EXCLUSIVE lock, that will prevent all reads and writes to the table. Use a TEXT field with a CHECK constraint.
(prefer-text-field)
[warning] 8-8: Changing the size of a varchar field requires an ACCESS EXCLUSIVE lock, that will prevent all reads and writes to the table. Use a TEXT field with a CHECK constraint.
(prefer-text-field)
[warning] 9-9: Changing the size of a varchar field requires an ACCESS EXCLUSIVE lock, that will prevent all reads and writes to the table. Use a TEXT field with a CHECK constraint.
(prefer-text-field)
[warning] 10-10: Changing the size of a varchar field requires an ACCESS EXCLUSIVE lock, that will prevent all reads and writes to the table. Use a TEXT field with a CHECK constraint.
(prefer-text-field)
[warning] 11-11: Changing the size of a varchar field requires an ACCESS EXCLUSIVE lock, that will prevent all reads and writes to the table. Use a TEXT field with a CHECK constraint.
(prefer-text-field)
[warning] 13-13: Changing the size of a varchar field requires an ACCESS EXCLUSIVE lock, that will prevent all reads and writes to the table. Use a TEXT field with a CHECK constraint.
(prefer-text-field)
[warning] 48-48: Changing the size of a varchar field requires an ACCESS EXCLUSIVE lock, that will prevent all reads and writes to the table. Use a TEXT field with a CHECK constraint.
(prefer-text-field)
[warning] 49-49: Changing the size of a varchar field requires an ACCESS EXCLUSIVE lock, that will prevent all reads and writes to the table. Use a TEXT field with a CHECK constraint.
(prefer-text-field)
[warning] 51-51: Changing the size of a varchar field requires an ACCESS EXCLUSIVE lock, that will prevent all reads and writes to the table. Use a TEXT field with a CHECK constraint.
(prefer-text-field)
[warning] 52-52: Changing the size of a varchar field requires an ACCESS EXCLUSIVE lock, that will prevent all reads and writes to the table. Use a TEXT field with a CHECK constraint.
(prefer-text-field)
[warning] 53-53: Changing the size of a varchar field requires an ACCESS EXCLUSIVE lock, that will prevent all reads and writes to the table. Use a TEXT field with a CHECK constraint.
(prefer-text-field)
[warning] 54-54: Changing the size of a varchar field requires an ACCESS EXCLUSIVE lock, that will prevent all reads and writes to the table. Use a TEXT field with a CHECK constraint.
(prefer-text-field)
[warning] 72-72: Changing the size of a varchar field requires an ACCESS EXCLUSIVE lock, that will prevent all reads and writes to the table. Use a TEXT field with a CHECK constraint.
(prefer-text-field)
[warning] 73-73: Changing the size of a varchar field requires an ACCESS EXCLUSIVE lock, that will prevent all reads and writes to the table. Use a TEXT field with a CHECK constraint.
(prefer-text-field)
[warning] 74-74: Changing the size of a varchar field requires an ACCESS EXCLUSIVE lock, that will prevent all reads and writes to the table. Use a TEXT field with a CHECK constraint.
(prefer-text-field)
[warning] 75-75: Using 32-bit integer fields can result in hitting the max int limit. Use 64-bit integer values instead to prevent hitting this limit.
(prefer-bigint-over-int)
[warning] 76-76: Changing the size of a varchar field requires an ACCESS EXCLUSIVE lock, that will prevent all reads and writes to the table. Use a TEXT field with a CHECK constraint.
(prefer-text-field)
[warning] 99-99: Changing the size of a varchar field requires an ACCESS EXCLUSIVE lock, that will prevent all reads and writes to the table. Use a TEXT field with a CHECK constraint.
(prefer-text-field)
[warning] 100-100: Changing the size of a varchar field requires an ACCESS EXCLUSIVE lock, that will prevent all reads and writes to the table. Use a TEXT field with a CHECK constraint.
(prefer-text-field)
[warning] 102-102: Changing the size of a varchar field requires an ACCESS EXCLUSIVE lock, that will prevent all reads and writes to the table. Use a TEXT field with a CHECK constraint.
(prefer-text-field)
[warning] 103-103: Changing the size of a varchar field requires an ACCESS EXCLUSIVE lock, that will prevent all reads and writes to the table. Use a TEXT field with a CHECK constraint.
(prefer-text-field)
[warning] 104-104: Changing the size of a varchar field requires an ACCESS EXCLUSIVE lock, that will prevent all reads and writes to the table. Use a TEXT field with a CHECK constraint.
(prefer-text-field)
[warning] 107-107: Changing the size of a varchar field requires an ACCESS EXCLUSIVE lock, that will prevent all reads and writes to the table. Use a TEXT field with a CHECK constraint.
(prefer-text-field)
[warning] 108-108: Changing the size of a varchar field requires an ACCESS EXCLUSIVE lock, that will prevent all reads and writes to the table. Use a TEXT field with a CHECK constraint.
(prefer-text-field)
[warning] 109-109: Changing the size of a varchar field requires an ACCESS EXCLUSIVE lock, that will prevent all reads and writes to the table. Use a TEXT field with a CHECK constraint.
(prefer-text-field)
[warning] 110-110: Changing the size of a varchar field requires an ACCESS EXCLUSIVE lock, that will prevent all reads and writes to the table. Use a TEXT field with a CHECK constraint.
(prefer-text-field)
[warning] 111-111: Changing the size of a varchar field requires an ACCESS EXCLUSIVE lock, that will prevent all reads and writes to the table. Use a TEXT field with a CHECK constraint.
(prefer-text-field)
[warning] 112-112: Changing the size of a varchar field requires an ACCESS EXCLUSIVE lock, that will prevent all reads and writes to the table. Use a TEXT field with a CHECK constraint.
(prefer-text-field)
[warning] 114-114: Using 32-bit integer fields can result in hitting the max int limit. Use 64-bit integer values instead to prevent hitting this limit.
(prefer-bigint-over-int)
[warning] 175-175: Changing the size of a varchar field requires an ACCESS EXCLUSIVE lock, that will prevent all reads and writes to the table. Use a TEXT field with a CHECK constraint.
(prefer-text-field)
[warning] 176-176: Changing the size of a varchar field requires an ACCESS EXCLUSIVE lock, that will prevent all reads and writes to the table. Use a TEXT field with a CHECK constraint.
(prefer-text-field)
[warning] 178-178: Changing the size of a varchar field requires an ACCESS EXCLUSIVE lock, that will prevent all reads and writes to the table. Use a TEXT field with a CHECK constraint.
(prefer-text-field)
[warning] 180-180: Using 32-bit integer fields can result in hitting the max int limit. Use 64-bit integer values instead to prevent hitting this limit.
(prefer-bigint-over-int)
[warning] 181-181: Using 32-bit integer fields can result in hitting the max int limit. Use 64-bit integer values instead to prevent hitting this limit.
(prefer-bigint-over-int)
[warning] 182-182: Changing the size of a varchar field requires an ACCESS EXCLUSIVE lock, that will prevent all reads and writes to the table. Use a TEXT field with a CHECK constraint.
(prefer-text-field)
[warning] 183-183: Changing the size of a varchar field requires an ACCESS EXCLUSIVE lock, that will prevent all reads and writes to the table. Use a TEXT field with a CHECK constraint.
(prefer-text-field)
[warning] 210-210: Using 32-bit integer fields can result in hitting the max int limit. Use 64-bit integer values instead to prevent hitting this limit.
(prefer-bigint-over-int)
[warning] 212-212: Changing the size of a varchar field requires an ACCESS EXCLUSIVE lock, that will prevent all reads and writes to the table. Use a TEXT field with a CHECK constraint.
(prefer-text-field)
[warning] 213-213: Using 32-bit integer fields can result in hitting the max int limit. Use 64-bit integer values instead to prevent hitting this limit.
(prefer-bigint-over-int)
[warning] 214-214: Using 32-bit integer fields can result in hitting the max int limit. Use 64-bit integer values instead to prevent hitting this limit.
(prefer-bigint-over-int)
[warning] 215-215: Using 32-bit integer fields can result in hitting the max int limit. Use 64-bit integer values instead to prevent hitting this limit.
(prefer-bigint-over-int)
[warning] 216-216: Changing the size of a varchar field requires an ACCESS EXCLUSIVE lock, that will prevent all reads and writes to the table. Use a TEXT field with a CHECK constraint.
(prefer-text-field)
[warning] 219-219: Using 32-bit integer fields can result in hitting the max int limit. Use 64-bit integer values instead to prevent hitting this limit.
(prefer-bigint-over-int)
[warning] 220-220: Changing the size of a varchar field requires an ACCESS EXCLUSIVE lock, that will prevent all reads and writes to the table. Use a TEXT field with a CHECK constraint.
(prefer-text-field)
🔇 Additional comments (77)
apps/worker/src/main/resources/application.yml (1)
19-52: LGTM!apps/worker/src/test/java/com/orgmemory/worker/ingestion/SourceDocumentReaderTests.java (1)
1-41: LGTM!integrations/object-storage-minio/src/main/java/com/orgmemory/integrations/storage/minio/MinioObjectStorageAutoConfiguration.java (1)
1-34: LGTM!docs/conventions.md (1)
43-50: LGTM!Also applies to: 60-62, 80-82
docs/decisions/0005-secure-java-graph-kernel.md (1)
30-39: LGTM!docs/decisions/0008-worker-owns-ingestion-and-derived-indexes.md (1)
36-38: LGTM!docs/increments/active/2026-07-20-secure-knowledge-vertical-slice/design.md (1)
13-15: LGTM!Also applies to: 33-33
docs/increments/active/2026-07-20-secure-knowledge-vertical-slice/plan.md (1)
16-30: LGTM!Also applies to: 49-50, 73-74
docs/specs/domains/knowledge-ingestion.md (1)
5-28: LGTM!Also applies to: 29-35, 36-39
apps/api/src/main/java/com/orgmemory/api/OrgMemoryApiApplication.java (1)
17-20: LGTM!apps/api/src/main/java/com/orgmemory/api/source/SourceController.java (1)
1-68: LGTM!apps/api/src/main/java/com/orgmemory/api/source/SourceResponse.java (1)
7-44: LGTM!apps/api/src/main/resources/application.yml (1)
41-56: LGTM!web/src/features/sources/components/source-status-badge.tsx (1)
7-34: LGTM!web/src/features/sources/source-status.ts (1)
3-68: LGTM!web/src/features/sources/store/document-manager-store.ts (1)
6-27: LGTM!web/src/features/sources/components/sources-table.tsx (1)
16-97: LGTM!web/src/routeTree.gen.ts (1)
31-111: LGTM!web/src/routes/_authenticated/sources.tsx (1)
1-7: LGTM!apps/api/src/test/java/com/orgmemory/api/OrgMemoryApiContextLoadTests.java (1)
6-13: LGTM!apps/api/src/test/java/com/orgmemory/api/source/SourceUploadIntegrationTests.java (1)
1-103: LGTM!apps/api/src/test/java/com/orgmemory/api/capability/CapabilityAssetServiceIntegrationTests.java (1)
44-56: LGTM!apps/api/src/test/java/com/orgmemory/api/knowledge/KnowledgeIngestionIntegrationTests.java (1)
41-48: LGTM!Also applies to: 375-379, 498-510, 521-521
apps/api/src/test/java/com/orgmemory/api/knowledge/KnowledgeRetrievalIntegrationTests.java (1)
43-55: LGTM!apps/api/src/test/java/com/orgmemory/api/permission/PermissionAuditIntegrationTests.java (1)
15-32: LGTM!Also applies to: 95-96
web/package.json (1)
40-41: LGTM!web/src/components/app-shell/app-shell.tsx (1)
2-2: LGTM!Also applies to: 11-13, 27-27
web/src/components/app-shell/app-sidebar.tsx (1)
1-1: LGTM!Also applies to: 17-20, 35-35
apps/worker/src/main/java/com/orgmemory/worker/ingestion/SourceProcessingProperties.java (1)
1-51: LGTM!apps/worker/src/main/java/com/orgmemory/worker/ingestion/SourceIngestionScheduler.java (1)
1-25: LGTM!apps/worker/src/main/java/com/orgmemory/worker/ingestion/ParsedSource.java (1)
1-12: LGTM!apps/worker/src/main/java/com/orgmemory/worker/ingestion/RejectedSourceException.java (1)
1-16: LGTM!apps/worker/src/main/java/com/orgmemory/worker/ingestion/SourceIngestionProcessor.java (1)
1-89: LGTM!Also applies to: 108-220, 240-312
apps/worker/src/test/java/com/orgmemory/worker/ingestion/SourceIngestionPipelineIntegrationTests.java (2)
42-47: 🗄️ Data Integrity & IntegrationConfirm the worker doesn't own schema migrations in production.
This test explicitly sets
spring.flyway.enabled=trueto provision schema for its own Testcontainers-backed context, which is a reasonable test-only pattern. Based on learnings, the API is intended to run migrations while workers should not own schema evolution — please confirm the worker's actualapplication.yml(not in this file set) keeps Flyway disabled by default outside of tests, so this override doesn't mask a production configuration drift.Based on learnings, "The API runs migrations; workers and MCP do not own schema evolution."
1-41: LGTM!Also applies to: 48-173
apps/worker/src/main/java/com/orgmemory/worker/OrgMemoryWorkerApplication.java (1)
34-36: 🩺 Stability & AvailabilityPackage-private
mainis supported here. Spring Boot 4.1.0's launcher tooling accepts package-privatemainmethods, so this visibility change does not breakbootRunor the packaged jar.> Likely an incorrect or invalid review comment.core/src/main/java/com/orgmemory/core/knowledge/ClaimedSourceRevision.java (1)
8-27: LGTM!core/src/main/java/com/orgmemory/core/knowledge/CreateUploadSourceCommand.java (1)
6-12: LGTM!core/src/main/java/com/orgmemory/core/knowledge/EmbeddingDistanceMetric.java (1)
3-5: LGTM!core/src/main/java/com/orgmemory/core/knowledge/EmbeddingProfile.java (1)
12-54: LGTM!core/src/main/java/com/orgmemory/core/knowledge/EmbeddingProfileRef.java (1)
5-13: LGTM!core/src/main/java/com/orgmemory/core/knowledge/EmbeddingProfileRegistry.java (1)
22-55: LGTM!core/src/main/java/com/orgmemory/core/knowledge/EmbeddingProfileRepository.java (1)
7-12: LGTM!integrations/object-storage-minio/build.gradle.kts (1)
1-13: LGTM!integrations/object-storage-minio/src/main/java/com/orgmemory/integrations/storage/minio/MinioObjectStorageProperties.java (1)
8-24: LGTM!gradle/libs.versions.toml (1)
10-10: 📐 Maintainability & Code QualityMinIO SDK pin is current.
9.0.3is the latest MinIO Java SDK release, so this dependency does not need an update.> Likely an incorrect or invalid review comment.core/src/main/java/com/orgmemory/core/knowledge/EmbeddingProfileSpec.java (1)
3-18: LGTM!Also applies to: 25-30
core/src/main/java/com/orgmemory/core/knowledge/SourceIngestionProperties.java (1)
7-15: LGTM!core/src/main/java/com/orgmemory/core/knowledge/SourceObject.java (1)
13-121: LGTM!core/src/main/java/com/orgmemory/core/knowledge/SourceObjectRepository.java (1)
7-10: LGTM!core/src/main/java/com/orgmemory/core/knowledge/SourceObjectStatus.java (1)
3-6: LGTM!core/src/main/java/com/orgmemory/core/knowledge/SourceQueryService.java (1)
8-22: LGTM!Also applies to: 39-60
core/src/main/java/com/orgmemory/core/knowledge/SourceRevisionRepository.java (1)
7-10: LGTM!integrations/object-storage-minio/src/main/resources/META-INF/spring/org.springframework.boot.autoconfigure.AutoConfiguration.imports (1)
1-1: LGTM!settings.gradle.kts (1)
23-23: LGTM!apps/api/build.gradle.kts (1)
7-10: LGTM!Also applies to: 34-36
apps/worker/build.gradle.kts (1)
5-12: LGTM!core/src/main/java/com/orgmemory/core/knowledge/EvidenceBlob.java (1)
1-83: LGTM!core/src/main/java/com/orgmemory/core/knowledge/EvidenceBlobRepository.java (1)
1-11: LGTM!core/src/main/java/com/orgmemory/core/knowledge/EvidenceScanStatus.java (1)
1-8: LGTM!core/src/main/java/com/orgmemory/core/knowledge/KnowledgeChunkDraft.java (1)
1-21: LGTM!core/src/main/java/com/orgmemory/core/knowledge/SourceIngestionCoordinator.java (1)
1-155: LGTM!core/src/main/java/com/orgmemory/core/knowledge/SourceIngestionJobRepository.java (1)
1-28: LGTM!core/src/main/java/com/orgmemory/core/knowledge/SourceIngestionJobStatus.java (1)
1-9: LGTM!core/src/main/resources/db/migration/V15__source_upload_and_embedding_pipeline.sql (1)
115-117: 🗄️ Data Integrity & IntegrationCheck the organization-scoped foreign keys
raw_source_object_id,normalized_record_id, andknowledge_asset_idshould use the same(id, organization_id)pattern as the other references in this migration, orsource_revisionscan point at rows from a different organization.core/src/main/java/com/orgmemory/core/knowledge/SourceRevision.java (1)
96-115: 🗄️ Data Integrity & Integration
revisionNumberneeds a source-specific next-value path. Any revision created here is assigned1, so re-uploads for the sameSourceObjectneed a way to supply the nextrevisionNumberor they will hituq_source_revision_number.core/src/main/java/com/orgmemory/core/knowledge/SourceRevisionStatus.java (1)
3-13: LGTM!core/src/main/java/com/orgmemory/core/knowledge/SourceSummary.java (1)
7-24: LGTM!core/src/main/java/com/orgmemory/core/knowledge/SourceType.java (1)
3-6: LGTM!core/src/main/java/com/orgmemory/core/knowledge/SourceUploadRegistrationService.java (1)
34-59: LGTM!core/src/main/java/com/orgmemory/core/knowledge/SourceUploadService.java (1)
20-143: LGTM!core/src/main/java/com/orgmemory/core/knowledge/storage/ObjectContent.java (1)
6-18: LGTM!core/src/main/java/com/orgmemory/core/knowledge/storage/ObjectStorageException.java (1)
3-8: LGTM!core/src/main/java/com/orgmemory/core/knowledge/storage/ObjectStoragePort.java (1)
5-14: LGTM!core/src/main/java/com/orgmemory/core/knowledge/storage/ObjectWriteRequest.java (1)
5-23: LGTM!core/src/main/java/com/orgmemory/core/knowledge/storage/StoredObject.java (1)
3-10: LGTM!core/src/main/java/com/orgmemory/core/shared/JpaAuditingConfig.java (1)
8-8: LGTM!
1 participant
What changed
Why
This establishes the first production-shaped ingestion path for OrgMemory without coupling canonical evidence to a storage vendor or prematurely implementing the graph UI/kernel.
Validation
gradlew --no-daemon clean testpnpm -C web lintpnpm -C web typecheckpnpm -C web buildtext-embedding-3-largevectorSummary by CodeRabbit