fix(models): make structured-output calls resilient to weak OpenRouter models#919
Merged
harshithmullapudi merged 2 commits intoJun 26, 2026
Merged
Conversation
generateContactSummary builds a structured CRM profile (email, company, location, headline, narrative summary) from memory episodes, but its makeStructuredModelCall call omitted the useCase argument, which defaults to "chat". So instead of using the workspace's configured Memory Ingestion model, built for reliable structured extraction, it silently used the Chat model, which is often a cheaper/smaller model picked for conversational replies and not for strict structured-output tasks. Every other extraction job in the codebase, label-assignment, knowledgeGraph extraction, aspect-resolution, already passes useCase "memory" for this reason. With a weak chat model this produced blank or partially filled contact profiles, e.g. only a one-line headline with no email, company, location, or summary, since most ContactSummarySchema fields default to an empty string and silently pass validation even when the model fails to produce them. Also gives the call its own prompt-cache key, "contact-summary", instead of sharing the generic "structured-medium" bucket with unrelated calls.
…arsing makeStructuredModelCall has two paths for getting JSON out of a model: a strict path that relies on native structured-output/tool-calling enforcement, and a tolerant path (used for Ollama and self-hosted OpenAI-compatible proxies) that explicitly writes out the JSON schema in the prompt, parses leniently, and runs a repair pass if parsing or validation fails. needsTolerantParsing only checked for Ollama and the OpenAI-proxy case, so any other provider, including OpenRouter, took the strict path by default. That's an unsafe assumption for OpenRouter specifically: it routes to a huge range of third-party models with wildly inconsistent tool-calling/JSON-schema support. A smaller or reasoning-tuned OpenRouter model can fail structured output entirely, or return partial JSON that still passes validation because most schema fields default to "", masking the failure as a silent partial response instead of a visible error. Now resolves the actual provider of the model being called (not just the global default chat provider) and routes OpenRouter through the same tolerant-parsing + repair path Ollama already gets, so a less reliable OpenRouter model degrades gracefully instead of returning incomplete structured output. Builds on top of the contact-summary useCase fix (separate PR) so a contact-summary call that resolves to an OpenRouter model is now protected on both fronts: the right model slot is used, and if that slot is configured with an OpenRouter model that's unreliable at strict JSON, the call still produces a complete, schema-valid object instead of a partially blank one.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
Follow-up to #918. That PR fixed contact summaries silently using the wrong model slot (Chat instead of Memory Ingestion). But even with the right model slot selected, if that slot is configured with an OpenRouter model that isn't great at strict structured output (e.g. a small/reasoning-tuned model like
openrouter/xiaomi/mimo-v2.5-pro), the same class of bug can resurface: partial or blank structured responses that silently pass validation because most schema fields default to"".This branch includes #918's commit so it has no merge conflicts with it — it's meant to land after/alongside that PR.
Bug
makeStructuredModelCall(apps/webapp/app/lib/model.server.ts) has two ways of getting JSON out of a model:needsTolerantParsing()decided which path to use by checking only the global default chat provider (Ollama) and an OpenAI-proxy config flag — it never looked at which provider the actual model for this call resolves to, and it had no special handling for OpenRouter at all. So any OpenRouter model, regardless of how good or bad it is at structured output, went through the strict path, which assumes a level of native JSON-schema/tool-calling reliability that OpenRouter can't guarantee (it routes to hundreds of different third-party models).Result: a less capable OpenRouter model can fail structured output outright, or return a half-filled JSON object that still validates successfully (since most fields default to
""), producing a silently incomplete result instead of a visible failure.Fix
needsTolerantParsingnow takes the resolved model string for the current call and checks its actual provider (via the existinggetProvider()helper), not just the global default.isOpenRouterto the tolerant-parsing condition, so any call resolving to anopenrouter/...model now goes through the same schema-hinted prompt + lenient parse + repair-pass flow Ollama already gets, instead of assuming native structured-output support it may not have.This makes structured-output calls resilient to whichever model a workspace has configured via OpenRouter, rather than requiring the operator to hand-pick a model known to support strict structured output well.
Testing
pnpm exec vitest run app/services/contacts/__tests__/contact-summary.test.ts app/services/contacts/__tests__/reconcile.test.ts— 13/13 passing (these mockmakeStructuredModelCalldirectly, unaffected by the change, included to confirm no regression).pnpm exec tsc --noEmitinapps/webapp— no new type errors (7 pre-existing failures in unrelated files, none touchingmodel.server.ts).