[Refactor]: Round-robin provider registry without Runnable.withFallbacks chain#36
Conversation
…sue #14) Replaces random-pick + single-key model resolution with deterministic round-robin across every enabled (provider, model, key) tuple. The returned runnable is wrapped via LangChain's withFallbacks so a retryable error on the primary walks the rest in order. No priority field — tuples are sorted by (providerId, modelName, keyName) and the rotation is modulo N. Counter is process-local; per-process fairness is sufficient for a single-VPS self-host. Cache: tuple list (decrypted blobs + baseUrl + model names) is cached in a 60s LRU keyed on providerId:modelName. The wrapped RunnableWithFallbacks is rebuilt on every call so round-robin can advance — the previous 'cache the ChatOpenAI' model couldn't rotate. Docs: docs/PROVIDERS.md rewritten to describe round-robin + chain; docs/DB.md line 174 reflects new resolution path; CLAUDE.md docs index picks up PROVIDERS/CREDIT/ADMIN/DEPLOY. New CLAUDE.md rule #13 forbids ad-hoc test scripts under scripts/ (use /tmp). Refs #14.
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
All reported issues were addressed across 5 files
Tip: instead of fixing issues one by one fix them all with cubic
Tip: cubic used a learning from your PR history. Let your coding agent read cubic learnings directly with the cubic MCP.
Re-trigger cubic
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
Review summaryPR #36 cleanly implements the round-robin-with- A few concerns worth addressing before merge: 1.
|
FireTable
changed the title
…ructuredOutput RunnableWithFallbacks is Runnable, not BaseChatModel — drops .bindTools / .withStructuredOutput. The 6 LangGraph node consumers crashed at runtime whenever a >=2-tuple round-robin landed on the wrapped primary. Return the bare ChatOpenAI picked by round-robin; cross-tuple retry on error is intentionally gone (see ModelTuple comment). Adds a regression test that pins bindTools / withStructuredOutput as functions on every >=2-tuple round-robin pick so a future re-introduction of withFallbacks flips it red.
Drop 'withFallbacks' from the resolution-path step description and rename the section. Cross-tuple retry on retryable error is gone — add it back via a per-call-site try/catch loop when a per-key rate-limit becomes a real problem.
New 'destructive' Badge variant (red, light + dark, mirrors the success variant's emerald pattern). Admin Providers tab swaps the Disabled badge to it and gives the entire card bg-muted/40 so a glance separates live from off — no new shade, bg-muted/40 was already in use on UsersPanel info tiles.
…ry cast No behavior change. - registry: drop the through-unknown cast (ChatOpenAI IS a BaseChatModel) and rewrite the 'build all N first' rationale — keep the cost rationale, drop the speculative 'future per-tuple apply bindTools' YAGNI comment. - admin ProviderCard: 4-line ponytail comment on a 1-line className → 2 lines (rule #5: explain why, briefly). - badge.tsx: reformat destructive variant to multi-line, matching success's shape after a linter pass single-lined it.
…' to 'any thrown error' Greptile + Cubic review of PR #36: - sort collectTuples by (providerId, modelName, keyName) — the doc contract was partially implemented (only the provider dimension was sorted via DB orderBy); models and keys came out in JSONB insertion order, drifting rotation across cache misses. - replace the global nextTupleIndex counter with a per-cacheKey Map so different opt shapes don't bleed into each other; today all 7 callers use the default pool (so the bleed was inert), but the per-agent opt direction in PROVIDERS.md would have hit it. - reword 'On retryable error' → 'On any thrown error' in both code JSDoc and docs/PROVIDERS.md — withFallbacks walks on every thrown error (no exceptionsToHandle filter), so 'retryable' was inaccurate even when withFallbacks was still in play. - export resetRoundRobinCounters() so tests start from a known counter baseline; wire it into beforeEach. - dbSpy restore wrapped in try/finally so an assertion throw doesn't leak the spy between tests. - rename two existing tests whose assertion (reference inequality) is guaranteed by per-call rebuild, not by rotation order — the name was misleading.
FireTable
changed the title
Review summarySolid refactor — the round-robin + cache-on-tuple-list shape is sound and the explicit revert of Correctness
Test gaps
Minor
Non-issues (verified)
|
Review summaryThe fix is correct — dropping the One docs bug, one perf concern, one minor diagnostic regression — all flagged inline:
Nothing else. Type plumbing through |
|
Both reviews addressed. Summary of fixes shipped on this branch: Round-robin shape
Docs sync (CLAUDE.md rule #1)
Out of scope / declined
PR title + body rewritten; full stats end of branch. |
| * on the provider/models/apiKeys rows). With a specific key, clears | ||
| * just that `(providerId, modelName)` opt-shape. | ||
| */ | ||
| export function invalidateModelCache(key?: OptsKey): void { |
There was a problem hiding this comment.
🐛 Correctness/performance — invalidateModelCache(key) clears counters for ALL pools, not just the invalidated one.
nextTupleIndexByKey.clear() runs unconditionally, even when key is a specific (providerId:modelName) slot. The comment rationale ("a counter trained on the old layout would pick a stale index") is only true for the invalidated pool — every other pool's tuples are untouched, but their rotation counters are reset to 0 anyway, throwing away accumulated distribution progress on unrelated pools.
Cheap fix: scope the clear to the invalidated key only:
export function invalidateModelCache(key?: OptsKey): void {
if (key) {
tupleCache.delete(key);
nextTupleIndexByKey.delete(key);
} else {
tupleCache.clear();
nextTupleIndexByKey.clear();
}
}If the broader reset is intentional (defensive — any CUD is treated as "rotation state is suspect"), please update the comment so a future reader doesn't think it's a scope bug. Today the comment promises what the narrower fix delivers, but the code is broader.
| // pick. N decrypt + ctor is small and amortized over the 60s tuple | ||
| // TTL, so the "build only the picked one" optimization isn't worth | ||
| // the cache-tracker complexity. | ||
| const models = tuples.map( |
There was a problem hiding this comment.
🐛 Performance — N decrypts + N ChatOpenAI ctors per call, even though only the picked one is returned.
The tuple list stays cached for 60s, but models.map(...) does an aesGcmDecrypt + loadKek() (parses env hex + base64) + new ChatOpenAI(...) (instantiates an @langchain/openai SDK client per tuple) on every call. With K keys per provider, M models, P providers, an LLM call now does P·M·K decrypt/ctor pairs even though only 1 ChatOpenAI is returned. The PR description acknowledges this but undercounts the work — loadKek is called per tuple, and new ChatOpenAI builds an internal Client (it eagerly validates the apiKey on construction in some versions).
The prior code did 1 decrypt + 1 ctor per call (random pick from one entry). This is a regression: a 10-key production pool pays ~10× the per-call setup cost.
Two cheap mitigations:
- Memoize
decryptApiKeyper tuple — group identical(encryptedKey, iv)ciphertexts (different plaintexts naturally don't collide) so repeated LLM calls within the 60s window decrypt each tuple exactly once. Cache the decrypted plaintext on theModelTupleonce at fetch time. - Build only the picked one (
models[start]) — but then a follow-upinvalidateModelCachefrom a hot tuple doesn't see the prebuilt instances. Acceptable trade — the picked one is the only one that ever runs through the SDK.
(1) is the cleanest win and doesn't change the round-robin semantics. Worth considering before merge — the regression test only covers correctness, not call-site cost.
|
|
||
| // Round-robin pick, scoped to the cacheKey so interleaving different | ||
| // opt shapes doesn't drift a smaller pool onto a single index. | ||
| const counter = nextTupleIndexByKey.get(cacheKey) ?? 0; |
There was a problem hiding this comment.
🐛 Nit — counter Map values grow unbounded across the lifetime of the process.
nextTupleIndexByKey.set(cacheKey, counter + 1) stores the unmodulo'd counter. Each entry is just a number, so memory isn't the issue, but on a long-running self-host with rare admin CUDs the counter values drift toward Number.MAX_SAFE_INTEGER (still safe modulo-wise). Cleaner: store the index modulo tuples.length so the value stays bounded and the % op can go.
const counter = nextTupleIndexByKey.get(cacheKey) ?? 0;
const start = counter % tuples.length;
nextTupleIndexByKey.set(cacheKey, start + 1); // bounded by tuples.lengthWhen tuples.length is 1 (most common single-key setups) start is always 0 and the stored counter is always 1 — that's the cheapest representation. Tiny code, removes one JS-safe-int tick on the worry list.
|
|
||
| ## Round-robin (no fallback chain) | ||
|
|
||
| The registry distributes traffic **evenly** across every enabled `(provider, model, key)` tuple, with no priority field — each tuple is one slot in the rotation. On every call, `getChatModelFromDB`: |
There was a problem hiding this comment.
📝 Doc drift — round-robin section still describes a single global counter.
§ "Round-robin (no fallback chain)" says:
- Increments a process-local
nextTupleIndexcounter; the new primary istuples[counter % N].
But lib/provider/model-registry.ts now uses Map<OptsKey, count> (nextTupleIndexByKey), not a single global counter, so the bullet under-describes the cacheKey-scoping behaviour that the per-cacheKey Map is designed for. Either name the field (nextTupleIndexByKey) or add a parenthetical like (scoped to the (providerId:modelName) cache key so interleaved opt shapes don't share a rotation). The body of § Per-process elsewhere acknowledges the per-process part, but the "even distribution" contract is now cacheKey-scoped as well — worth surfacing.
| // ponytail: clear per-cacheKey rotation counters so this test sees a | ||
| // fresh "call 0 → tuple 0" baseline. Without it, calls from earlier | ||
| // tests in the same file would push the counter ahead. | ||
| resetRoundRobinCounters(); |
There was a problem hiding this comment.
📝 Test gap — resetRoundRobinCounters has no positive assertion.
resetRoundRobinCounters is wired into beforeEach but no test asserts its actual effect. A no-op export that just calls clear() is fine; the issue is that without a positive test, a future regression that makes resetRoundRobinCounters a no-op (e.g. accidental rename / typoed export) would silently leak counter state between tests and flip the named assertions in the "per-call rebuild" tests' results — the tests would still pass, just on a shifted baseline.
Cheapest fix: assert that the counter Map is empty after a call:
import { nextTupleIndexByKey } from "@/lib/provider/model-registry"; // or via spying
it("resetRoundRobinCounters empties the Map", () => {
for (let i = 0; i < 3; i++) void getChatModelFromDB();
resetRoundRobinCounters();
// hit one of the counter reads via another call
void getChatModelFromDB();
// Next call would map to tuples[0] only because counter was reset to 0
});Pragmatic: spy on Map.prototype.set and assert the post-reset write value is 0+1=1, not >1.
Or just check the export shape — expect(typeof resetRoundRobinCounters).toBe("function") — but that doesn't catch a typo'd body.
| // were ≥2 tuples. This test pins both methods as functions on every | ||
| // round-robin pick — a regression to `withFallbacks` flips these to | ||
| // undefined. | ||
| it("with ≥2 tuples, every round-robin pick still exposes bindTools + withStructuredOutput", async () => { |
There was a problem hiding this comment.
📝 Test gap — the regression test pins surface, not the underlying bug.
The regression for the prior withFallbacks crash pins typeof bindTools === "function" and typeof withStructuredOutput === "function" on the returned model. That's necessary but not sufficient:
- Surface-only — a future bug where
getChatModelFromDBreturns a stale cachedRunnableWithFallbacksthat happened to havebindToolspatched onto it manually (e.g. someone "fixes" the wrapper by monkey-patching) would pass this test but still drop the BaseChatModel contract forwithConfig,withRetry,bind,generate,stream, etc. - No consumer coverage — the PR description calls out "6 LangGraph node consumers" but none of them is exercised. A future regression where
bindToolslives on the returned model but returns the wrong shape for a LangGraph-compatible tool list would still flip green here.
Two cheap upgrades:
- Assert the returned object is a
ChatOpenAI(expect(model).toBeInstanceOf(ChatOpenAI)). That pins the BaseChatModel → ChatOpenAI contract directly. - Add a smoke test that does
await model.invoke("test")against a mockedChatOpenAI(or assertstypeof model.stream === "function") — covers the bare Runnable surface that RoundRobin broke before.
The PR description already acknowledges the rotation-order test is filed as a follow-up; this is the consumer-side counterpart.
| for (const m of p.models) { | ||
| if (!m.enabled) continue; | ||
| if (opts.modelName && m.name !== opts.modelName) continue; | ||
| for (const k of p.apiKeys) { |
There was a problem hiding this comment.
📝 apiKeys[] entry doesn't filter empty entries.
The collection loop adds a tuple for every p.apiKeys element, including:
{}/ null-tail entries (Drizzle would not strip, but a malformed row could land one)- Entries whose
encryptedKeyis empty (decrypt would throw at build time) - Entries without a
name(admin validators should prevent this, but the registry is the trust boundary)
The admin deriveKeyName produces first3…last4 from plaintext at creation, and the Zod schema in app/api/admin/providers/[id]/keys/route.ts rejects empty plaintext — so in practice this can't happen. But the registry has no defense if a bad row ever lands (e.g. a future bulk-import path that skips the API validators). decryptApiKey throws on a malformed ciphertext, but the throw surfaces from inside models.map(...) so the caller sees an opaque "decrypt failed on tuple M" instead of the friendly DB-level error.
Cheap belt-and-suspenders:
for (const k of p.apiKeys) {
if (!k.encryptedKey || !k.iv || !k.name) continue; // malformed row — skip, don't crash
tuples.push({ ... });
}Or even better — crypto.timingSafeEqual later, but that's beyond scope.
| if (keys.length === 0) return undefined; | ||
| return keys[Math.floor(Math.random() * keys.length)]; | ||
| async function collectTuples(opts: GetChatModelOpts): Promise<ModelTuple[]> { | ||
| const providerRows = opts.providerId |
There was a problem hiding this comment.
🐛 Correctness — getChatModelFromDB() without opts short-circuits the DB SELECT to ALL rows regardless of enabled.
Prior code had where(eq(providerTable.enabled, true)) in the no-opts branch. The new code dropped that filter and relies on the in-process if (!p.enabled) continue; check. Behaviorally equivalent, but the cache key is derived from opts only — and the cache is populated even when the SELECT returns all providers, including the disabled ones. The cache entry is the tuple list (ModelTuple[] only contains enabled tuples), so this is fine on the read path. Just flagging it as a deliberate widening of the SELECT shape: await db.select().from(providerTable).orderBy(asc(providerTable.id)) now reads every row.
For a self-host this is a single PK scan — not measurable. But the symmetry with the opts.providerId branch matters: there, the SELECT is also unfiltered (no eq(providerTable.enabled, true) was ever there for that branch). It's fine; just noting the intentional widening from the prior code so a future "re-add the filter" pass doesn't accidentally narrow it back without thinking through the cache behavior.
If you'd prefer the symmetry: await db.select().from(providerTable).where(eq(providerTable.enabled, opts.providerId ? undefined : true)) — Drizzle ignores undefined where clauses.
| // Return type stays BaseChatModel (not ChatOpenAI) so a non-OpenAI | ||
| // provider can land without touching the 6 call sites; today every | ||
| // registered model is ChatOpenAI. | ||
| return models[start] as BaseChatModel; |
There was a problem hiding this comment.
📝 as BaseChatModel cast hides a typed contract gap.
return models[start] as BaseChatModel; — ChatOpenAI IS a BaseChatModel, so the cast is a no-op at runtime. But TS doesn't know that here because models is inferred as ChatOpenAI[] and the return type annotation Promise<BaseChatModel> is the widening. The cast is what widens it.
A friendlier approach (rules #3 best practices) — change the return annotation to Promise<ChatOpenAI>. Either:
- The registry IS only ChatOpenAI today (the file comment says so) — say it in the signature.
- Future-proof for ChatAnthropic — keep
BaseChatModel, but the cast still hides nothing because every constructor path is aChatOpenAI. The cast signals "the type system doesn't yet know about ChatAnthropic."
If (1) wins, type as Promise<ChatOpenAI> and drop the cast — backend/model.ts already imports ChatOpenAI and the call site .bindTools(...) will narrow cleanly.
If (2) wins, the cast is honest and the file comment "every registered model is ChatOpenAI" pins the const at compile time. Acceptable as-is — just noting the cast exists for typing, not for runtime safety.
| @@ -186,13 +186,15 @@ function ProviderCard({ provider }: { provider: PublicProviderRow }) { | |||
| }; | |||
|
|
|||
| return ( | |||
There was a problem hiding this comment.
📝 Minor — section comment placement violates rule #5 (comments explain why, not what).
return (
// ponytail: muted background for disabled cards so the list separates
// "live" from "kept around but off" at a glance.
<Card className={provider.enabled ? undefined : "bg-muted/40"}>The comment paraphrases what the conditional className does ("muted background for disabled cards") rather than the non-obvious why ("kept around but off" is the user-facing intent, which IS a why). The commit message explicitly trimmed this comment to 2 lines — but the wording could collapse to the non-obvious bit and skip the restatement:
// ponytail: "kept around but off" — disabled cards stay listed so the operator can re-enable without re-adding keys.
<Card className={provider.enabled ? undefined : "bg-muted/40"}>Or drop entirely — the className reads as "muted on disabled" without a comment. The current 2-line ponytail is small and not wrong; just a CLAUDE.md #5 alignment note.
Review summaryRead through PR #36 — round-robin provider registry without a VerdictShip-able after the perf fix on What I checked
Issues filed (inline comments)
Not addressed here (parked in the PR description)
🤖 Generated with Claude Code |
1 participant
Closes #14 (round-robin load distribution across
(provider, model, key)tuples; health observability is parked in the future-direction section ofdocs/PROVIDERS.md).Problem
The original round-robin design (
a6b9b6b) wrapped the picked chain in LangChain'sRunnable.withFallbacks(...)for cross-tuple retry. The wrapper returnsRunnableWithFallbacks, which extendsRunnable— NOTBaseChatModel. The 6 LangGraph node call sites (router, thread-summarize, plus the four sub-agents) chain.bindTools(...)/.withStructuredOutput(...)on the result; both methods live onBaseChatModel, notRunnable. With ≥ 2 enabled tuples the wrap masked those methods withundefined, andwithStructuredOutput is not a functioncrashed every multi-tuple router pass.What changed
Three iterations on this branch (commits
3f885cf→699aee7→682698a, plus the cleanup313e407):lib/provider/model-registry.tswithFallbacks(...)wrap.getChatModelFromDB(opts)returns a bare round-robin-pickedChatOpenAI. Cross-tuple retry on error is intentionally NOT in the registry — add it via a per-call-sitetry/catchif/when a per-key rate-limit becomes a real problem.tuples.sort(...)by(providerId, modelName, keyName). DBorderByonly covered the provider dimension; models and keys came out of JSONB arrays in insertion order, drifting rotation across cache misses.nextTupleIndexfrom a global counter to aMap<cacheKey, count>. Today all 7 callers share the default pool so the bleed was inert, but per-agent opts (future-direction item indocs/PROVIDERS.md) would have hit it.On retryable error→On any thrown error(withFallbacks walks on every thrown error; the old "retryable" wording was inaccurate even when the chain existed).resetRoundRobinCounters()and clear the counter map frominvalidateModelCache()— admin CUD writes now bust both the tuple list and the rotation state.tests/lib/provider/model-registry.test.tsbeforeEachcallsresetRoundRobinCounters()so per-cacheKey counters start at 0.dbSpyrestore intry/finallyso an assertion throw doesn't leak the spy.with ≥2 tuples, every round-robin pick still exposes bindTools + withStructuredOutput. PinsbindToolsandwithStructuredOutputastypeof === "function"on every round-robin pick so a future re-introduction ofwithFallbacksflips it red.docs/PROVIDERS.md— drop "retryable" wording; rename section to "Round-robin (no fallback chain)".backend/node/router-agent-node.ts— dropped a debugconsole.warn(2222, ...)line that was tracing the original bindTools crash.components/ui/badge.tsx+app/admin/admin-tabs.tsx— disabled provider badge now uses a new reddestructivevariant and the disabled card getsbg-muted/40so a glance separates live from off in the admin Providers tab. (Tangential; bundled because both touch the disabled state.)Tests
Docs
docs/PROVIDERS.mdupdated in the fix commit (per CLAUDE.md rule feat(001): stage 1 — user auth (Better Auth + email verification + thread ownership) #1).docs/APIS.mdnot touched —/api/admin/providers/**endpoints are unchanged; only the rendering of the disabled state differs.Not in this PR (parked, same as before)
provider.last_used_at/last_success_at/last_error_*columns + admin "Test key" button). Issue [Feat]: admin-managed LLM API keys with sequential fallback #14 stays open for this.getChatModel({ agentName }). Still future direction; the registry signature already supports the extra opt shape, and the per-cacheKey counter added here means per-agent opt shapes won't bleed onto each other.call N picks tuple N mod len). Requires module-mockingChatOpenAIto peek at which key was used per call — out of scope here. Filed as a follow-up.🤖 Generated with Claude Code
Greptile Summary
This PR fixes a runtime crash where wrapping the round-robin pick in
Runnable.withFallbacks(...)produced aRunnableWithFallbacksthat lacked.bindToolsand.withStructuredOutput, breaking all 6 LangGraph node call sites whenever ≥ 2 provider/key tuples were enabled. The fix drops the fallback chain entirely, moves the LRU to cache the tuple list (not the model instance), and scopes the round-robin counter percacheKeyto prevent pool bleed.lib/provider/model-registry.ts: replaced single-model caching with tuple-list caching; addedMap<OptsKey, number>counters for per-pool round-robin; addedresetRoundRobinCounters()test hook;invalidateModelCache()now clears both the LRU and counters (but the partial-key path clears all counters, not just the evicted key's).tests/lib/provider/model-registry.test.ts: 6 new/updated tests, including abindTools/withStructuredOutputregression pin andtry/finallyspy cleanup.components/ui/badge.tsx+app/admin/admin-tabs.tsx: adds adestructive(red) badge variant; disabled provider cards get a muted background for visual separation.Confidence Score: 4/5
Safe to merge — the crash fix is correct and well-tested, but the partial-invalidation path resets all round-robin counters instead of only the evicted key's counter.
The core fix (dropping withFallbacks, returning a bare ChatOpenAI) is sound and the regression test covers the exact failure mode. The one defect is in invalidateModelCache: when called with a specific key — the path triggered by every admin CUD route — nextTupleIndexByKey.clear() wipes counters for every cache key in the process, not just the one being invalidated. This silently resets round-robin fairness for the default pool and any future per-agent pool on every admin write.
lib/provider/model-registry.ts — the invalidateModelCache function's counter-clearing is unconditional regardless of whether a specific key was supplied
Important Files Changed
Sequence Diagram
%%{init: {'theme': 'neutral'}}%% sequenceDiagram participant Node as LangGraph Node participant GM as getChatModelFromDB(opts) participant LRU as tupleCache (LRU 60s) participant DB as Postgres participant CTR as nextTupleIndexByKey Node->>GM: getChatModelFromDB() GM->>LRU: get(cacheKey) alt cache hit LRU-->>GM: ModelTuple[] else cache miss GM->>DB: SELECT providers ORDER BY id DB-->>GM: provider rows GM->>GM: collectTuples() → sort by (providerId, modelName, keyName) GM->>LRU: set(cacheKey, tuples) end GM->>CTR: get(cacheKey) → counter N CTR-->>GM: N GM->>CTR: set(cacheKey, N+1) GM->>GM: "pick = tuples[N % len]" GM->>GM: build all ChatOpenAI instances (decrypt each key) GM-->>Node: models[N % len] as BaseChatModel Note over Node: .bindTools() / .withStructuredOutput() work Node->>GM: admin CUD → invalidateModelCache(specificKey) GM->>LRU: delete(specificKey) GM->>CTR: clear() ⚠️ clears ALL keys, not just specificKey%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%% sequenceDiagram participant Node as LangGraph Node participant GM as getChatModelFromDB(opts) participant LRU as tupleCache (LRU 60s) participant DB as Postgres participant CTR as nextTupleIndexByKey Node->>GM: getChatModelFromDB() GM->>LRU: get(cacheKey) alt cache hit LRU-->>GM: ModelTuple[] else cache miss GM->>DB: SELECT providers ORDER BY id DB-->>GM: provider rows GM->>GM: collectTuples() → sort by (providerId, modelName, keyName) GM->>LRU: set(cacheKey, tuples) end GM->>CTR: get(cacheKey) → counter N CTR-->>GM: N GM->>CTR: set(cacheKey, N+1) GM->>GM: "pick = tuples[N % len]" GM->>GM: build all ChatOpenAI instances (decrypt each key) GM-->>Node: models[N % len] as BaseChatModel Note over Node: .bindTools() / .withStructuredOutput() work Node->>GM: admin CUD → invalidateModelCache(specificKey) GM->>LRU: delete(specificKey) GM->>CTR: clear() ⚠️ clears ALL keys, not just specificKeyReviews (3): Last reviewed commit: "docs(db): drop stale RunnableWithFallbac..." | Re-trigger Greptile
Context used: