Skip to content

feat: add fallback for models without tool calling support#3608

Merged
ComputelessComputer merged 1 commit intomainfrom
devin/1770134854-support-models-without-tool-calling
Feb 4, 2026
Merged

feat: add fallback for models without tool calling support#3608
ComputelessComputer merged 1 commit intomainfrom
devin/1770134854-support-models-without-tool-calling

Conversation

@devin-ai-integration
Copy link
Contributor

Summary

Fixes an issue where users get an error saying "tool calling not supported" when using models like Gemma via LM Studio for summarization.

The root cause: when no template is selected, generateTemplateIfNeeded uses the Vercel AI SDK's Output.object({ schema }) to generate template sections. This internally uses tool calling, which models like Gemma don't support.

Solution: Added a generateStructuredOutput helper that:

  1. First attempts structured output via tool calling (for models that support it)
  2. On failure, falls back to plain text generation with JSON extraction using regex
  3. Validates the parsed JSON against the Zod schema

Review & Testing Checklist for Human

  • Test with a model that doesn't support tool calling (e.g., Gemma via LM Studio) - verify summarization works without errors
  • Test with a model that does support tool calling (e.g., GPT-4, Claude) - verify existing behavior is unchanged
  • Review the regex pattern /\{[\s\S]*\}/ for JSON extraction - could it incorrectly match nested objects or fail on edge cases?
  • Verify the fallback gracefully returns null (same as original behavior) when both approaches fail

Recommended test plan:

  1. Set up LM Studio with Gemma model
  2. Create a session with transcript content but no template selected
  3. Trigger summarization and verify it completes without "tool calling not supported" error

Notes

  • The error variable on line 206 is intentionally unused - it's caught to trigger the fallback
  • Both primary and fallback paths return null on failure, maintaining the original graceful degradation behavior

Link to Devin run: https://app.devin.ai/sessions/9cc7c5ed1c564bb5b24bb602f3391179
Requested by: @ComputelessComputer

When generating template sections during summarization, the AI SDK's
Output.object uses tool calling under the hood. Models like Gemma via
LM Studio don't support tool calling, causing errors.

This change adds a fallback mechanism:
1. First attempts structured output via tool calling (for models that support it)
2. On failure, falls back to plain text generation with JSON extraction
3. Validates the parsed JSON against the schema using Zod

This allows models without tool calling support to still work with
the summarization feature.

Co-Authored-By: john@hyprnote.com <john@hyprnote.com>
@netlify
Copy link

netlify bot commented Feb 3, 2026

Deploy Preview for hyprnote canceled.

Name Link
🔨 Latest commit 1f2ec5f
🔍 Latest deploy log https://app.netlify.com/projects/hyprnote/deploys/69821ec14f65840008999c02

@netlify
Copy link

netlify bot commented Feb 3, 2026

Deploy Preview for hyprnote-storybook canceled.

Name Link
🔨 Latest commit 1f2ec5f
🔍 Latest deploy log https://app.netlify.com/projects/hyprnote-storybook/deploys/69821ec14f65840008999c06

@devin-ai-integration
Copy link
Contributor Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR that start with 'DevinAI' or '@devin'.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

@ComputelessComputer ComputelessComputer merged commit 9918d36 into main Feb 4, 2026
17 checks passed
@ComputelessComputer ComputelessComputer deleted the devin/1770134854-support-models-without-tool-calling branch February 4, 2026 10:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant