[TEST] feat/per-call-llm-creds — per-call LLM provider + API key overrides#45
Merged
Conversation
There was a problem hiding this comment.
Pull request overview
Adds per-call LLM provider/API-key overrides so a single worker can serve multiple tenants. @bytebell/llm's askLLM now accepts apiKey and provider on AskLlmOptions, @bytebell/types adds a PayloadLlmOverrides mixin (llmApiKey / llmProvider / llmModel) extended by both GitHub payloads, and the entire @bytebell/ingest-github pipeline threads an AskLlmOptions "call context" from StrategyContext down to every askJsonLLM / askYesNoLLM call site (skip-decider, file analyzer, big-file chunk analyzer, folder/repo summaries, backfills). All context.md files in touched folders are updated to document the new flow. OSS standalone behaviour is preserved: with the overrides unset, the pipeline falls back to Config.OpenrouterApiKey + Config.LlmProvider.
Changes:
@bb/llm: addapiKey/providertoAskLlmOptions, exportLlmProviderName, route both OpenRouter calls throughopts.apiKey ?? Config.OpenrouterApiKey.@bb/types: addPayloadLlmOverridesmixin (llmApiKey,llmProvider,llmModel) on both GitHub payloads; re-export from root.@bb/ingest-github: build anAskLlmOptionsbag from the payload inrun.ts/pull.ts, stash it onStrategyContext.llmCallContext, and forward through every phase (classifyAndAnalyseSmall,processBigFilesQueue,analyseScannedFile,analyzeChunk,processBigFile,backfillMissingFields,backfillBigFiles,runFolderSummaryPhase,runSelectiveFolderSummary,summariseRepo,makeSkipDecider).
Reviewed changes
Copilot reviewed 35 out of 35 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/types/src/job.ts | Defines PayloadLlmOverrides and extends both GitHub payload interfaces with it. |
| packages/types/src/index.ts | Re-exports PayloadLlmOverrides from the package root. |
| packages/types/src/context.md, packages/types/context.md | Docs for the new mixin and its place in the queue vocabulary. |
| packages/llm/src/client.ts | Adds apiKey / provider to AskLlmOptions, exports LlmProviderName, lets per-call provider beat Config.LlmProvider. |
| packages/llm/src/openrouter.ts | Reads API key as opts.apiKey ?? Config.OpenrouterApiKey in both chain-resolver and call paths. |
| packages/llm/src/index.ts, packages/llm/src/context.md, packages/llm/context.md | Surface new types in barrel + docs; documents the per-call credential-override invariant. |
| packages/ingest-github/src/types/strategy.ts | Adds optional llmCallContext on StrategyContext. |
| packages/ingest-github/src/types/pipeline.ts | Adds llmCallContext to FileAnalyzer.analyze, ScanDeps, SkipDeciderInput; also introduces unused PullFactory* types. |
| packages/ingest-github/src/types/context.md | Docs for the new llmCallContext field across the port types. |
| packages/ingest-github/src/pipeline/run.ts | Builds llmCallContext from payload, stashes on StrategyContext. |
| packages/ingest-github/src/pipeline/pull.ts | Mirrors run.ts (duplicate helper) and threads context into each phase (but misses analyseChangedFiles). |
| packages/ingest-github/src/pipeline/scan.ts | Forwards llmCallContext from ScanDeps into each SkipDeciderInput. |
| packages/ingest-github/src/pipeline/skip-decisions/decider.ts | Threads llmCallContext into the LLM branch (askYesNoLLM). |
| packages/ingest-github/src/pipeline/context.md, .../skip-decisions/context.md | Documents the new flow. |
| packages/ingest-github/src/adapters/llm-file-analyzer.ts | analyze now accepts and forwards llmCallContext to askJsonLLM. |
| packages/ingest-github/src/adapters/context.md | Docs update. |
| packages/ingest-github/src/strategies/flat-folder/index.ts | Reads llmCallContext from StrategyContext and forwards into every phase. |
| .../strategies/flat-folder/analyse-file.ts | Accepts/forwards llmCallContext into analyzer.analyze. |
| .../strategies/flat-folder/folder-summary.ts, folder-summary-selective.ts, repo-summary.ts | Threads llmCallContext into askJsonLLM. |
| .../strategies/flat-folder/backfill/fields.ts, big-files.ts | Threads llmCallContext into backfill calls. |
| .../strategies/flat-folder/big-file/index.ts, chunk-analyzer.ts | Threads llmCallContext to the chunk analyzer's askJsonLLM. |
| .../strategies/flat-folder/phases/classify-and-analyse-small.ts, process-big-files.ts | Threads llmCallContext through phase inputs. |
| .../strategies/flat-folder/*context.md (× several), backfill/context.md, big-file/context.md, phases/context.md | Doc updates to mirror the new override flow. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+149
to
+159
| const analyseChangedInput: Parameters<typeof analyseChangedFiles>[0] = { | ||
| knowledgeId, | ||
| repoDir, | ||
| metaPaths, | ||
| analyzer: fileAnalyzer, | ||
| diff, | ||
| }); | ||
| }; | ||
| if (llmCallContext !== undefined) { | ||
| analyseChangedInput.llmCallContext = llmCallContext; | ||
| } | ||
| await analyseChangedFiles(analyseChangedInput); |
Comment on lines
+107
to
+134
| export interface PullFactoryInput { | ||
| knowledgeId: string; | ||
| payload: GithubPullPayload; | ||
| /** The commit currently anchored on the knowledge in Mongo. The factory diffs from here to `targetCommit`. */ | ||
| currentCommit: string; | ||
| /** Branch the knowledge tracks. The factory resolves the target commit relative to this branch. */ | ||
| branch: string; | ||
| } | ||
|
|
||
| export interface PullFactoryResult { | ||
| /** Reader pinned at the resolved target commit; used by every downstream phase for file I/O. */ | ||
| source: SourceReader; | ||
| /** Files changed between `currentCommit` and the resolved target. Same shape as `git diff --name-status`. */ | ||
| diff: DiffResult; | ||
| /** Resolved target commit hash. Either the payload's `targetCommitHash` or the branch HEAD chosen by the factory. */ | ||
| targetCommit: string; | ||
| /** Optional non-fatal sink. When set, the strategy archives analysed content via `push` after each file. */ | ||
| archiveSink?: ArchiveSink; | ||
| } | ||
|
|
||
| /** | ||
| * Optional injection hook used by `registerGithubWorkers` for pull jobs. | ||
| * When provided, `runPull` skips `syncRepository` + `computePullDiff` + | ||
| * `checkoutCommit` and uses the factory's reader + diff directly. The | ||
| * open-source binary leaves this undefined and pull runs against a local | ||
| * git clone via `node:child_process`. | ||
| */ | ||
| export type PullFactory = (input: PullFactoryInput) => Promise<PullFactoryResult>; |
Comment on lines
+37
to
+53
| function llmCallContextFromPayload(payload: { | ||
| llmApiKey?: string; | ||
| llmProvider?: "openrouter" | "ollama"; | ||
| llmModel?: string; | ||
| }): AskLlmOptions | undefined { | ||
| const ctx: AskLlmOptions = {}; | ||
| if (payload.llmApiKey !== undefined && payload.llmApiKey.length > 0) { | ||
| ctx.apiKey = payload.llmApiKey; | ||
| } | ||
| if (payload.llmProvider !== undefined) { | ||
| ctx.provider = payload.llmProvider; | ||
| } | ||
| if (payload.llmModel !== undefined && payload.llmModel.length > 0) { | ||
| ctx.model = payload.llmModel; | ||
| } | ||
| return Object.keys(ctx).length > 0 ? ctx : undefined; | ||
| } |
Dead-Bytes
added
enhancement
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
@bytebell/llmaskLLMnow accepts two optional per-call overrides onAskLlmOptions:apiKey— overridesConfig.OpenrouterApiKeyfor this call (ignored by the keyless Ollama provider)provider— overridesConfig.LlmProvider("openrouter" | "ollama") for this callLlmProviderNametype from@bytebell/llm.@bytebell/typesadds aPayloadLlmOverridesmixin (llmApiKey,llmProvider,llmModel) and extendsGithubIndexPayloadandGithubPullPayloadwith it. Exported from the package root.@bytebell/ingest-githubthreads the LLM context through every call site that talks to the LLM:pipeline/run.ts,pipeline/pull.ts,pipeline/scan.tspipeline/skip-decisions/decider.tsadapters/llm-file-analyzer.tsstrategies/flat-folder/index.ts,analyse-file.ts,folder-summary.ts,folder-summary-selective.ts,repo-summary.tsstrategies/flat-folder/backfill/{big-files,fields}.tsstrategies/flat-folder/big-file/{index,chunk-analyzer}.tsstrategies/flat-folder/phases/{classify-and-analyse-small,process-big-files}.tscontext.mdupdated in every touched folder to document the new override flow (llm/, types/, ingest-github/{adapters,pipeline,pipeline/skip-decisions,types,strategies/flat-folder,…}).8e48469).Why
Downstream consumers (the enterprise wrapper /
github-gatewayresolve-llm-for-jobpath) resolve per-org LLM credentials at the enqueue boundary and need to infuse them into the job payload, so a single worker process can serve multiple tenants without swapping global config.OSS standalone behaviour is unchanged: when
llmApiKey/llmProvider/llmModelare unset on the payload,askLLMfalls back toConfig.OpenrouterApiKeyandConfig.LlmProviderexactly as before.