feat(knowledge-store): git-native knowledge base - #1651
Conversation
Restores the work from MODSetter#1649, which was reverted on main in MODSetter#1650 to keep it out of the last release. Content is identical to that merge. This is a revert of the revert (a8292f5) rather than a merge of kb_git_mvp, deliberately. Merging the branch would make its commits an ancestor of both dev and main; main's side deleted those files, so the next merge between the two branches would silently delete them again. A revert carries the content without the history, so dev and main share no ancestor that knows about these files, and the eventual dev -> main merge sees them as added on one side only and keeps them. Git becomes the source of truth for knowledge base content; Postgres and pgvector become a derived, rebuildable index. Both switches guarding the new path default to off: the KNOWLEDGE_STORE_ENABLED env var and the per-workspace knowledge_store_enabled column, so merging this changes no runtime behaviour. Migrations 175 and 176 only add columns and use ADD COLUMN IF NOT EXISTS. They are already applied on production, where alembic_version was moved back to 174 during the revert, so they will re-run harmlessly. Verified on this branch: 0 conflicts against dev, every app.* import in the restored files resolves, 2251 unit tests pass. The one collection error (platforms/google_maps) is missing a fixture that is untracked on both dev and main, and predates this change.
|
@CREDO23 is attempting to deploy a commit to the Rohan Verma's projects Team on Vercel. A member of the Team first needs to authorize it. |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
1 participant
Brings the git-native knowledge base to
dev. Same content as #1649, which was reverted frommainin #1650 so it wouldn't ship in the last release.Why this is a revert-of-the-revert and not a merge of
kb_git_mvpMerging the feature branch into
devwould make its commits an ancestor of bothdevandmain. Since main's history now contains a commit deleting all of these files, the next merge in either direction would pick the feature branch tip as the merge base, see main's deletion as the only change, and silently remove the feature again — no conflict, no warning.Reverting the revert carries the content without the history.
devandmainshare no ancestor that knows about these files, so the eventualdev->mainmerge sees them as added on one side only and keeps them.The cost is that the individual commits from #1649 aren't preserved here. That PR has the granular history if anyone needs it.
What it does
Git becomes the source of truth for knowledge base content. Postgres and pgvector become a derived index that can be rebuilt from git at any time. Agent turns commit through a
KnowledgeStore.transaction()unit of work, and a Celery consumer converges the index from the committed revision.Safety
Two independent switches guard every new path, both defaulting to off:
KNOWLEDGE_STORE_ENABLED, an env var, defaultFALSEworkspaces.knowledge_store_enabled, a per-workspace column, defaultfalseWith either off, the existing write path runs unchanged, so merging this changes no runtime behaviour.
Migrations 175 and 176 only add columns and are written with
ADD COLUMN IF NOT EXISTS. They are already applied on production, wherealembic_versionwas rolled back to 174 during the revert, so re-running them is a no-op.Verification on this branch
devapp.*import in the restored files resolves against dev's treeThe single collection error under
tests/unit/platforms/google_mapsis a fixture file that is untracked on bothdevandmain; it predates this change and is unrelated.High-level PR Summary
This PR introduces a git-native knowledge base to the
devbranch, making Git the single source of truth for workspace knowledge content while demoting Postgres to a derived, rebuildable index. The implementation pivots from a custom virtual filesystem over Postgres to using dulwich (pure-Python Git) for versioned storage, with each workspace getting its own Git repository. The change is controlled by two feature flags (KNOWLEDGE_STORE_ENABLEDenvironment variable and per-workspaceknowledge_store_enableddatabase column), both defaulting to off, so merging changes no runtime behavior. Key components include: a framework-agnosticKnowledgeStorefacade over Git storage, per-turn working copies for agent operations, end-of-turn commit middleware, an indexer that rebuilds Postgres chunks from Git, migration tooling to seed existing workspaces, and drift monitoring. The architecture follows ports-and-adapters pattern where deepagents is just one adapter. Database migrations 175 and 176 add nullable columns withIF NOT EXISTSclauses for safety. The implementation deletes three hand-rolled versioning systems (DocumentVersion,DocumentRevision/FolderRevision,AgentActionLog) in favor of native Git history, and includes extensive unit and integration tests covering the full stack from Git engine to agent middleware.⏱️ Estimated Review Time: 3+ hours
💡 Review Order Suggestion
docs/adr/0001-git-native-knowledge-base.mddocs/adr/0002-knowledge-core-ports-and-adapters.mdplans/git-native-kb/00-umbrella-plan.mdplans/git-native-kb/00c-shared-contract.mdsurfsense_backend/alembic/versions/175_add_workspace_knowledge_store_flag.pysurfsense_backend/alembic/versions/176_add_derived_index_columns.pysurfsense_backend/app/config/__init__.pysurfsense_backend/app/db.pysurfsense_backend/app/knowledge_store/settings.pysurfsense_backend/app/knowledge_store/engines/base.pysurfsense_backend/app/knowledge_store/engines/git.pysurfsense_backend/app/knowledge_store/store_path.pysurfsense_backend/app/knowledge_store/write_lock.pysurfsense_backend/app/knowledge_store/transaction.pysurfsense_backend/app/knowledge_store/store.pysurfsense_backend/app/knowledge_store/identities.pysurfsense_backend/app/agents/chat/runtime/path_resolver.pysurfsense_backend/app/agents/chat/multi_agent_chat/shared/middleware/filesystem/backends/git_tree.pysurfsense_backend/app/agents/chat/multi_agent_chat/shared/middleware/filesystem/backends/resolver.pysurfsense_backend/app/agents/chat/multi_agent_chat/shared/middleware/filesystem/backends/local_folder.pysurfsense_backend/app/agents/chat/multi_agent_chat/shared/middleware/filesystem/backends/multi_root_local_folder.pysurfsense_backend/app/agents/chat/multi_agent_chat/shared/middleware/filesystem/tools/read_file/description.pysurfsense_backend/app/agents/chat/multi_agent_chat/main_agent/middleware/knowledge_store_persistence/commit_message.pysurfsense_backend/app/agents/chat/multi_agent_chat/main_agent/middleware/knowledge_store_persistence/commit_turn.pysurfsense_backend/app/agents/chat/multi_agent_chat/main_agent/middleware/knowledge_store_persistence/middleware.pysurfsense_backend/app/agents/chat/multi_agent_chat/main_agent/middleware/knowledge_store_persistence/builder.pysurfsense_backend/app/agents/chat/multi_agent_chat/main_agent/middleware/stack.pysurfsense_backend/app/agents/chat/multi_agent_chat/main_agent/runtime/factory.pysurfsense_backend/app/tasks/chat/streaming/agent/event_loop.pysurfsense_backend/app/indexing_pipeline/document_chunker.pysurfsense_backend/app/indexing_pipeline/chunk_reconciler.pysurfsense_backend/app/indexing_pipeline/cache/cached_indexing.pysurfsense_backend/app/indexing_pipeline/indexing_pipeline_service.pysurfsense_backend/app/knowledge_store/index/converge.pysurfsense_backend/app/knowledge_store/index/queue.pysurfsense_backend/app/tasks/celery_tasks/knowledge_store/index_tasks.pysurfsense_backend/app/tasks/celery_tasks/knowledge_store/drift_monitor_task.pysurfsense_backend/app/knowledge_store/migrate.pysurfsense_backend/scripts/migrate_knowledge_store.pysurfsense_backend/app/services/document_revision_recorder.pysurfsense_backend/app/routes/documents_routes.pysurfsense_backend/app/routes/editor_routes.pysurfsense_backend/app/knowledge_store/janitor.pysurfsense_backend/app/tasks/celery_tasks/knowledge_store/janitor_task.pysurfsense_backend/app/observability/metrics.pysurfsense_backend/app/celery_app.pysurfsense_backend/pyproject.toml