[Feat] Bidirectional GitHub/GitLab folder sync (documents ↔ repo) - #1732
Merged
CREDO23 merged 43 commits intoSep 1, 2026
Conversation
Phase 13: one connected repo mounts under documents/GitHub|GitLab/{full_name}/{sourcepath}, md bijection, 3-way fail-closed. Phase 12 remains the shipped push-only export.
The work is done; the spec is no longer needed on this branch.
Spell out env, volume, and migrate-once steps from the live compose/entrypoint so the fleet flip is not a lean sketch.
One function decides take-remote / keep-local / conflict so later ticks share the same rules without git or Postgres.
The tree path is computed from provider + repo + sourcepath so it cannot clash with a user's notes folder.
Pathspec mistakes must not wipe pngs or src/ on their branch.
A commit is a full snapshot of their tree; our documents/ layout cannot be that snapshot without dumping the KB or deleting src/.
last_local_revision is the 3-way base after a successful agree.
SQLAlchemy needs the new columns so the worker can read the base SHA.
Connect and the UI need the folder plus conflict/need_direction without leaking the PAT.
Status and the next plan() tick both read these from Postgres, not git.
Pull is just another store writer so index/Zero stay one-way from our git.
Direction is only for first sync and conflict, when 3-way has no winner.
A save should mirror the mount, not push our store HEAD onto their branch.
Phase 12's engine.push of our repo is the product this phase supersedes.
Retry is POST /sync; resolve takes from_remote|from_local. Drop /push.
Real store plus a second git repo; no fakes of either side of the bijection.
…tings Stop empty-repo/push-only copy; Use GitHub vs Use SurfSense when 3-way cannot infer.
Retry in settings is the recovery until drops show up in logs.
It only re-drove lost enqueues; we have no evidence of drop rate yet.
Nothing schedules it after the beat entry is gone.
The leftover checkout is not source of truth and should not sit on disk.
We only need their current tree to make a pathspec commit, not history.
Conflicts and blocked ticks were collapsing into remote.push noop, so alerts could not tell a stuck bijection from a workspace with nothing to do.
HTTP and the worker share these methods, so one set of spans and counters covers both callers without double-counting a tick.
Broker failures were log-only, and the worker stamped every None return as idle, hiding conflict and worktree_busy behind a noop.
Replace the fragile App Setup-URL callback with a user-to-server OAuth flow: the callback exchanges the code for a user token, lists the user's installations, and routes to a repo picker, an installation chooser, or an install prompt. Add endpoints to list a repo's branches and its recursive folder tree so the settings UI can offer pickers instead of free-text, and carry each repo's default branch through the API.
Pulled and applied remote changes went into the store but never enqueued indexing, so they never reached the sidebar; enqueue an index after those transactions. Stamp last_pushed_* on an initial pull and a mirror so the card reads "Synced" instead of "Not synced yet". Commit the resolve path like add/remove so a cleared conflict is not rolled back by the request lifecycle while its git push already went out.
…kers Restructure the connected-repo card into a label/value grid with a provider badge and a status pill instead of run-on text. Start the connect flow through GitHub user OAuth, add an installation chooser and an install prompt, and replace the free-text branch and source-path inputs with pickers backed by the new branches and folders endpoints. Add the matching API service methods, zod types, cache keys, and locale strings.
The folder bijection hardcoded ".md", so a .txt/.rst/.mdx doc under a connected folder was silently dropped. Introduce a single allowlist (SYNCED_SUFFIXES) and is_syncable(), and widen the path guard to accept any tracked text document. Binary formats (PDF, images) stay excluded and are left untouched on the remote for the ingestion connectors to handle.
check_staged refused a staged set that was not markdown; broaden it to the text-document allowlist so mirroring a .txt/.rst change no longer trips the "would delete foreign" guard.
Rename md_under_mount -> text_under_mount, select by is_syncable, and update the facade call sites and local names (local_docs/remote_docs) plus the need-direction message to speak in terms of documents rather than markdown.
list_md/replace_md walked only *.md; rename to list_text/replace_text and select files by is_syncable so the shadow round-trips the full text-document set while leaving binaries (png, pdf) in place on the remote.
Rename the shadow helpers in the sync integration tests and add a case that pulls .txt/.rst into the mount, skips a .pdf, and pushes a locally authored .mdx while the pdf survives on the remote.
Connecting when both the folder and the remote already hold documents no longer errors and strands the user: the connect succeeds, mirrors nothing, and flags the row `need_direction` so the card asks which side to keep. Resolving stamps the synced marker so the card reads "Synced" afterward. Drop the legacy `reconnect_required` path — it only ever fired on pre-`sourcepath` rows, which do not exist in prod. `sourcepath` is now NOT NULL (normalized to "" = repo root at save), so the guard is dead.
Add a shared git-remote status hook and flag "Sync needs attention" as a small warning-triangle badge in the corner of the connect button, so a conflict or failed push is visible from the Sources panel without opening settings. Rename the button to the provider-agnostic "Connect a repository". On the settings page, offer searchable folder and branch pickers and show an info toast (not a success) when a connect lands in the need-direction state, pointing the user at the resolve buttons.
|
@CREDO23 is attempting to deploy a commit to the Rohan Verma's projects Team on Vercel. A member of the Team first needs to authorize it. |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Team Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Add resolve_folder_id_by_parts, a read-only twin of the hierarchy builder, so callers can look up an existing folder id from a sanitized name chain without creating rows.
Resolve the folder each connected repo mounts onto and return it on the status DTO, giving the frontend a reliable id instead of matching folder names. It stays null until the indexer creates the folder.
Add mount_folder_id to the git remote type and helpers that classify a folder against it: whether it is synced content, whether deleting it would touch the sync, and the repo label for display.
FolderTreeView and FolderNode take an optional set of git-synced folder ids and render a small green dot on those folders so a connected repo is visible at a glance.
Swap the "Connect a repository" label for "Repository settings" when a remote is already connected, with the manage_repo string translated across locales.
The indexer creates the mount folder a moment after connect, so poll the status every 8s while mount_folder_id is null and stop once it resolves. The synced badge then appears without a manual reload.
Wire the mount folder id into the panel: badge the synced folder and confirm before deletions that reach the sync. The heads-up wording distinguishes a folder that is synced from one that only contains synced files, and bulk delete flags when the selection touches the sync.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
documents/{Forge}/{owner}/{repo}/{sourcepath}tree; text docs (.md/.markdown/.mdx/.rst/.txt) round-trip both ways, binaries untouched.Test plan
High-level PR Summary
This PR introduces bidirectional GitHub/GitLab folder synchronization that mounts one repository folder (e.g.,
docs/) asdocuments/{GitHub|GitLab}/{owner}/{repo}/{sourcepath}/in the workspace. Text documents (.md,.markdown,.mdx,.rst,.txt) round-trip bidirectionally while binaries remain untouched. Sync runs via a worker using a 3-way merge planner on a shadow clone with fast-forward-only pushes. Conflicts and scenarios where both sides have content fail closed and require explicit user resolution (keep local or keep remote). The feature includes GitHub user OAuth for listing installations, repos, branches, and folders, plus UI for displaying sync state and handling conflicts. The implementation replaces the previous push-only export product (Phase 12) with a folder-based bidirectional sync model (Phase 13).⏱️ Estimated Review Time: 3+ hours
💡 Review Order Suggestion
plans/git-native-kb/13-bidirectional-folder-sync.mdplans/git-native-kb/00-umbrella-plan.mdplans/git-native-kb/12-connect-own-remote.mdplans/git-native-kb/11-one-write-path-design.mdsurfsense_backend/alembic/versions/191_git_remote_folder_sync.pysurfsense_backend/app/config/__init__.pysurfsense_backend/.env.examplesurfsense_backend/app/knowledge_store/remote/schemas/spec.pysurfsense_backend/app/knowledge_store/remote/persistence/models.pysurfsense_backend/app/knowledge_store/remote/persistence/repository.pysurfsense_backend/app/knowledge_store/remote/schemas/status.pysurfsense_backend/app/knowledge_store/remote/api/schemas.pysurfsense_backend/app/knowledge_store/remote/paths.pysurfsense_backend/app/knowledge_store/remote/planner.pysurfsense_backend/app/knowledge_store/remote/guards.pysurfsense_backend/app/knowledge_store/remote/shadow.pysurfsense_backend/app/knowledge_store/remote/sync.pysurfsense_backend/app/knowledge_store/remote/facade.pysurfsense_backend/app/knowledge_store/remote/forges/github.pysurfsense_backend/app/knowledge_store/remote/api/routes.pysurfsense_backend/app/knowledge_store/remote/exceptions.pysurfsense_backend/app/knowledge_store/remote/queue.pysurfsense_backend/app/knowledge_store/service.pysurfsense_backend/app/observability/domains/knowledge_store.pysurfsense_backend/app/tasks/celery_tasks/knowledge_store/push_task.pysurfsense_backend/app/celery_app.pysurfsense_backend/tests/unit/knowledge_store/remote/test_planner.pysurfsense_backend/tests/unit/knowledge_store/remote/test_paths.pysurfsense_backend/tests/unit/knowledge_store/remote/test_guards.pysurfsense_backend/tests/unit/knowledge_store/remote/test_shadow.pysurfsense_backend/tests/unit/knowledge_store/remote/test_forges.pysurfsense_backend/tests/unit/knowledge_store/remote/test_routes_github.pysurfsense_backend/tests/unit/knowledge_store/remote/test_after_record.pysurfsense_backend/tests/unit/knowledge_store/remote/test_queue.pysurfsense_backend/tests/unit/observability/test_domains.pysurfsense_backend/tests/integration/knowledge_store/remote/test_connect.pysurfsense_backend/tests/integration/knowledge_store/remote/test_sync.pysurfsense_web/contracts/types/git-remote.types.tssurfsense_web/lib/apis/git-remotes-api.service.tssurfsense_web/lib/query-client/cache-keys.tssurfsense_web/hooks/use-git-remote-status.tssurfsense_web/components/settings/git-remote-settings.tsxsurfsense_web/components/documents/DocumentsFilters.tsxsurfsense_web/components/layout/ui/right-panel/DocumentRightPanel.tsxsurfsense_web/app/dashboard/[workspace_id]/workspace-settings/git-remote/page.tsxsurfsense_web/messages/en.jsonsurfsense_web/messages/es.jsonsurfsense_web/messages/hi.jsonsurfsense_web/messages/ko.jsonsurfsense_web/messages/pt.jsonsurfsense_web/messages/zh.jsonplans/account-deletion/account-deletion-and-welcome-credit.mdplans/account-deletion/account-deletion-and-welcome-credit.md