Skip to content

feat: improve model selection reliability, make gateway model source of truth#598

Merged
jonathanlab merged 1 commit intomainfrom
01-26-fix_improve_model_selection_reliability_make_gateway_model_source_of_truth
Jan 27, 2026
Merged

feat: improve model selection reliability, make gateway model source of truth#598
jonathanlab merged 1 commit intomainfrom
01-26-fix_improve_model_selection_reliability_make_gateway_model_source_of_truth

Conversation

@jonathanlab
Copy link
Contributor

@jonathanlab jonathanlab commented Jan 26, 2026 •
edited
Loading

TL;DR

Model selection was unreliable/broken.

Added dynamic model selection with PostHog Gateway integration, replacing hardcoded model options.

What changed?

  • Added API to fetch available models from PostHog's LLM Gateway
  • Created a new modelsStore to manage model state and selection
  • Updated the ModelSelector component to use dynamically fetched models
  • Added model information to session responses, including available models and current model
  • Implemented proper model persistence between sessions
  • Refactored the agent package to use more granular exports instead of a single entry point
  • Added model conversion between Gateway model IDs and Claude SDK model IDs

How to test?

  1. Log in to the application
  2. Start a new session and verify the model selector shows models from the Gateway
  3. Change models mid-session and verify the change is applied
  4. Start a new session and verify your model selection persists
  5. Reconnect to an existing session and verify the model selection is maintained

@jonathanlab jonathanlab mentioned this pull request Jan 26, 2026
Merged
@jonathanlab Graphite App
Copy link
Contributor Author

jonathanlab commented Jan 26, 2026 •
edited
Loading

This stack of pull requests is managed by Graphite. Learn more about stacking.

This was referenced Jan 26, 2026
Merged
Merged
Merged
@jonathanlab jonathanlab changed the title fix: improve model selection reliability, make gateway model source of truth feat: improve model selection reliability, make gateway model source of truth Jan 26, 2026
@jonathanlab jonathanlab marked this pull request as ready for review January 26, 2026 11:53
@jonathanlab jonathanlab requested a review from a team as a code owner January 26, 2026 11:53
@jonathanlab jonathanlab force-pushed the 01-26-fix_improve_model_selection_reliability_make_gateway_model_source_of_truth branch from 9f0cd38 to e3828d4 Compare January 26, 2026 11:58
@jonathanlab jonathanlab force-pushed the 01-23-chore_update_acp_sdk_and_mcp_sdk_to_latest branch from 8deb503 to ac1bbf2 Compare January 26, 2026 11:58
joshsny
joshsny reviewed Jan 26, 2026
View reviewed changes
packages/agent/src/gateway-models.ts Outdated
const modelsUrl = `${gatewayUrl}/v1/models`;

try {
const response = await fetch(modelsUrl, {
Copy link
Contributor

@joshsny joshsny Jan 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you shouldn't need an auth token to access this btw

packages/agent/src/gateway-models.ts
}

const PROVIDER_NAMES: Record<string, string> = {
anthropic: "Anthropic",
Copy link
Contributor

@joshsny joshsny Jan 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what about OpenAI / Gemini - Gemini models are currently disabled due to a limiting issue but OpenAI ones are available

Copy link
Contributor Author

@jonathanlab jonathanlab Jan 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found them very inconsistent in usage, I'll try adding OpenAI and see if it's reliable enough

packages/agent/src/gateway-models.ts Outdated
// The gateway exposes 200+ models.
// Update this list to control which we show to the client.
export const ALLOWED_GATEWAY_MODELS = [
"claude-opus-4-5",
Copy link
Contributor

@joshsny joshsny Jan 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this kinda defeats the point of this PR, no? We should expose the models from the gateway as the source of truth so we don't need to update this?

Copy link
Contributor Author

@jonathanlab jonathanlab Jan 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree, I'll get a PR in posthog out for restricting the models to the useful ones for this product so we do not need to do this

@jonathanlab jonathanlab force-pushed the 01-26-fix_improve_model_selection_reliability_make_gateway_model_source_of_truth branch from e3828d4 to 922af0c Compare January 27, 2026 11:39
@jonathanlab jonathanlab force-pushed the 01-23-chore_update_acp_sdk_and_mcp_sdk_to_latest branch from ac1bbf2 to 56fab62 Compare January 27, 2026 11:39
@jonathanlab jonathanlab force-pushed the 01-26-fix_improve_model_selection_reliability_make_gateway_model_source_of_truth branch from 922af0c to 81911ab Compare January 27, 2026 14:25
@jonathanlab jonathanlab force-pushed the 01-23-chore_update_acp_sdk_and_mcp_sdk_to_latest branch from 56fab62 to 929a2a4 Compare January 27, 2026 14:25
@jonathanlab jonathanlab force-pushed the 01-26-fix_improve_model_selection_reliability_make_gateway_model_source_of_truth branch from 81911ab to 3529539 Compare January 27, 2026 14:39
@jonathanlab Graphite App
Copy link
Contributor Author

jonathanlab commented Jan 27, 2026 •
edited
Loading

Merge activity

  • Jan 27, 2:52 PM UTC: A user started a stack merge that includes this pull request via Graphite.
  • Jan 27, 3:04 PM UTC: Graphite rebased this pull request as part of a merge.
  • Jan 27, 3:06 PM UTC: @jonathanlab merged this pull request with Graphite.

@jonathanlab jonathanlab changed the base branch from 01-23-chore_update_acp_sdk_and_mcp_sdk_to_latest to graphite-base/598 January 27, 2026 15:01
@jonathanlab jonathanlab changed the base branch from graphite-base/598 to main January 27, 2026 15:03
@jonathanlab jonathanlab force-pushed the 01-26-fix_improve_model_selection_reliability_make_gateway_model_source_of_truth branch from 3529539 to 8fe61a6 Compare January 27, 2026 15:03
@jonathanlab jonathanlab merged commit 8211a91 into main Jan 27, 2026
12 of 13 checks passed
@jonathanlab jonathanlab deleted the 01-26-fix_improve_model_selection_reliability_make_gateway_model_source_of_truth branch January 27, 2026 15:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@joshsny joshsny joshsny approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@jonathanlab @joshsny