feat(core): add the github source adapter - #65
Merged
Conversation
A repo becomes a context layer without a clone. One recursive git-tree call
builds the concept index; content and each file's last commit date are fetched
per concept. Default selectors are CLAUDE.md, AGENTS.md, README.md, docs/**,
and .context/** — overridable per layer.
Document parsing moves behind a shared parseDocument seam in files.mjs, so a
repo-hosted CLAUDE.md and a local one derive identical section keys and merge
section-for-section in the cascade rather than splitting into parallel sections.
Credentials are referenced, never carried: a layer's "auth" may only be
"keychain:<alias>" (resolved from the tokens map the caller injects, so the app
owns the keychain and the engine never opens it) or {"tokenEnv":"NAME"} for
headless runs. Every other shape throws, which is what structurally keeps a raw
token out of a manifest.
Failure degrades rather than breaking the resolve: an unreachable, rate-limited,
or unauthorized repo keeps serving its last good index with the cache age
surfaced, pauses retries for a cooldown, and warns once per window instead of
once per concept in a search sweep.
withCache.sync() now calls through to the wrapped adapter's own sync(), so a
user-triggered Sync clears both halves rather than only the outer one.
Tested against a local node:http fixture speaking the GitHub API shape — no
network. Eight mutations of the adapter were confirmed to fail the suite.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…e github adapter
Two defects found reviewing the adapter, both confirmed with probes before
being fixed and both now pinned by regression tests.
The commit-date memo outlived the index it belonged to. When the index expired
and refreshed, the memo survived, so a document edited upstream kept reporting
its old date for the life of the process — undermining the per-document
staleness dates that are the adapter's reason for reading commit history at all.
The memo now lives inside the index generation, so its lifetime is structurally
correct rather than maintained by hand.
Extension precedence was tree-order dependent. When a.md and a.txt collapse to
one concept id, files.mjs resolves by DOC_EXTENSIONS order; github kept whichever
GitHub happened to list last, so the same collision resolved differently on disk
and on GitHub. Ties now break by the shared extension order, and the test proves
it with the fixture tree served in both orders. A path that is nothing but an
extension (".md") is no longer indexed as an empty-named concept.
Also documents a residual risk in the manifest trust boundary: a github layer's
apiBase decides where an auth alias's credential is sent, so a hostile manifest
can aim a real token at a host of its choosing. That sits inside the existing
manifest trust boundary rather than beside it, but it was undocumented.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Jul 29, 2026
siracusa5
pushed a commit
that referenced
this pull request
Jul 29, 2026
Integrates the github source adapter (#65), git-history section dates (#67) and remote source health (#68) with background indexing and the utilityProcess engine. Four conflicts, all union rather than either-or: - okf-local: keeps main's commit-date history AND the async/bounded walk. documentDate's mtime fallback is now an async stat, since it runs in the read path the desktop engine must not block on. - files: keeps the shared parseDocument seam AND the async parseFile. Dates go through localDate, so a folder source and a repo source agree on the calendar day instead of one of them slicing UTC. - service: index state and adapter health are orthogonal, so both are reported. A source is 'indexing' until read, then 'degraded' if its adapter says its last request failed, then 'ok'. Resolution runs over anything with a snapshot, so a degraded source keeps contributing — main's warn-and-continue intent, expressed through the snapshot model. - types.ts: GraphSource carries indexing progress and the health timestamps. Two integration defects the merge exposed, both fixed: - httpError lost its `detail` argument when the HTTP internals moved to http-util.mjs, so a failed remote Sync answered with a bare message instead of ok:false and the health timestamps the API promises. - github-source-test asserted a settled source status straight after a Sync. Sync re-indexes, and reads answer from the background index, so the assertion raced the re-read. It now passes ?wait=, the same contract the service and playground suites use. Verified on the merged tree: engine 13/13 suites, console typecheck + 125 tests + build, site build, desktop unit (34) + navigation + cli-status + smoke + isolation (2ms main-loop lag while indexing 2,500 documents).
siracusa5
added a commit
that referenced
this pull request
Jul 29, 2026
Setup froze the Mac app on "Resolving…" until it stopped responding: an unbounded synchronous directory walk, every file read and tokenized twice per request, no timeout anywhere, and the engine sharing an event loop with the window that draws. - Sources are read by background jobs. /api/graph and /api/resolve-all answer from what is indexed right now and report `indexing`/`indexingSources`; clients render what they have and poll. Reads are partial by design, so completeness assertions pass ?wait=. First graph request against a 901-document folder: ~6ms. - The engine runs in its own Electron utilityProcess. Worst-case main-loop stall while indexing 2,500 documents: ~694ms sharing the loop -> ~30ms isolated, enforced by a new isolation test in CI. - Indexing limits are manifest-backed settings (Settings -> Indexing), not environment variables. Precedence is manifest > env > default. - New Files view: browse and edit the context files behind each source, as rendered Markdown or raw text. The layer file APIs moved into the engine so the desktop app can reach them, and now cover markdown-folder layers. - Markdown renders as typed data through React elements — no HTML string, no dangerouslySetInnerHTML, no hand-rolled sanitizer. - Adding a source only rejects what the user can fix on the form. A too-large folder is accepted, then surfaces as a visible source error pointing at Settings. - An unexpected engine exit is fatal, taking the same clean dialog-and-exit path as a failed boot. - The Mac app icon is regenerated from the canonical brand mark via `npm run icon`; the stale hand-maintained master is gone. Integrates the github source adapter (#65), git-history section dates (#67) and remote source health (#68): index state and adapter health are orthogonal, so a source reads `indexing` until read, `degraded` if its adapter reports its last request failed, then `ok`. Reviewed adversarially in a second session, which found real defects here — a rejected oversized body that kept buffering, a containment guard whose realpath result was discarded, and an inline SVG preview path. All fixed, along with revision-aware saves, cooperative index cancellation, and moving the API bearer out of renderer argv onto trusted IPC. Validation: engine 13/13 suites, console typecheck + 125 tests + build, site build, desktop unit/navigation/cli-status/smoke/boot-failure/isolation.
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the
githubsource kind fromspecs/contextcake-integrations/spec.md§4/§5. First remote adapter — until now a layer had to be an OKF bundle you authored, a local folder, or a foreign MCP server.What this does
A repo becomes a context layer with no clone and no OKF authoring. The markdown a team already keeps —
CLAUDE.md,AGENTS.md,README.md,docs/**,.context/**— resolves alongside local layers.{ "name": "payments-repo", "level": 3, "source": "github", "repo": "acme/payments", "paths": ["CLAUDE.md", "docs/**"], "auth": "keychain:github", "cache": { "ttlSeconds": 900 } }One recursive git-tree call builds the concept index, so
loadConceptnever guesses an extension. Concept ids are repo-qualified (acme/payments/docs/runbook), so several repos layer without colliding. Section dates come from each file's last commit rather than the repo's last push — staleness is per document, which is what the conflict surface needs.Design notes worth a reviewer's attention
Shared parsing seam. Document parsing moved into
parseDocumentinfiles.mjs, called by both adapters. If these ever diverge, a repo-hostedCLAUDE.mdand a local one stop merging and silently split into parallel sections instead. The suite pins this with a real cascade assertion (one mergedenginesection across two adapter kinds, github winning, okf-local dissent inconflicts[]).Credentials are referenced, never carried.
authaccepts only"keychain:<alias>"— resolved from atokensmap the caller injects, so the app owns the keychain and the engine never opens it — or{"tokenEnv":"NAME"}for headless runs. Every other shape throws. That's structural rather than advisory: a raw token cannot sit in a manifest by accident. An alias with nothing injected reads anonymously rather than failing.Failure degrades, it doesn't break. An unreachable, rate-limited, or unauthorized repo keeps serving its last good index with the cache age surfaced, pauses retries for a 60s cooldown, and warns once per window. Without the cooldown an mcp-server search sweep would mean one API call and one stderr line per concept against a rate-limited API — the worst possible response to a rate limit.
withCache.sync()now calls through to the wrapped adapter's ownsync(). Remote adapters hold their own index memo, so previously a user-triggered Sync cleared only the outer half.withGitSyncalready used this convention.Testing
packages/core/tests/github-source-test.shruns against a localnode:httpfixture speaking the GitHub API shape — no network. Covers glob selection, commit dates withpushed_atfallback, OKF-frontmatter delegation, cross-adapter section merging, credential indirection, foreign/traversal id rejection, truncated trees, stale-serve, cooldown, warn-once, and the cache wrapper.Eight mutations of the adapter were each confirmed to fail the suite: glob matcher disabled, repo-prefix guard removed, traversal guard removed, per-file commit date dropped, warn-and-continue removed, stale-serve removed, cooldown removed, warn-once removed.
Engine: exit 0, 12 suites. Site: exit 0.
Deliberately not here
apps/desktop/apps/console; the engine only ever sees tokens by value. Until that ships this is usable viatokenEnvand on public repos.🤖 Generated with Claude Code