Skip to content

feat: medium reasoning for GPT-5.6 Sol - #142

Merged
mattrossman merged 1 commit into
mainfrom
mattrossman/ai-977-switch-56-sol-evals-to-medium-reasoning
Jul 30, 2026
Merged

feat: medium reasoning for GPT-5.6 Sol#142
mattrossman merged 1 commit into
mainfrom
mattrossman/ai-977-switch-56-sol-evals-to-medium-reasoning

Conversation

@mattrossman

@mattrossman mattrossman commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator

Switches GPT-5.6 Sol experiment configs from low to medium reasoning to align with ChatGPT subscription default.

Sanity checked codex-gpt-5.6 against resolve-dataapi-001-empty-results ran and passed 7/7 checks, and the result recorded reasoningEffort: "medium".

Deferring the full matrix refresh since we'll be doing that before announcement anyways.

Closes AI-977

@linear-code

linear-code Bot commented Jul 29, 2026

Copy link
Copy Markdown
AI-977 Switch 5.6 Sol evals to medium reasoning

Current 5.6 Sol experiments are testing with low reasoning, but Codex appears to default to medium reasoning for typical users on a ChatGPT subscription.

The confusion comes from Codex shipping a local fallback catalog that defines low as the default, while ChatGPT-subscribed usage is overridden by a remote catalog that defaults to medium. ChatGPT docs also describe the default as medium.

Update the 5.6 experiments to use medium reasoning.

Relevant context: fallback catalog, remote override, ChatGPT docs.

CleanShot 2026-07-28 at 09.51.56@2x.png

CleanShot 2026-07-28 at 10.17.25@2x.png

Review in Linear

@vercel

vercel Bot commented Jul 29, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
evals Ignored Ignored Jul 29, 2026 9:54pm

Request Review

@mattrossman
mattrossman marked this pull request as ready for review July 29, 2026 21:55
@mattrossman
mattrossman requested a review from a team July 29, 2026 21:55
@mattrossman mattrossman changed the title feat: run GPT-5.6 Sol evals with medium reasoning feat: use medium reasoning for GPT-5.6 Sol Jul 29, 2026
@mattrossman mattrossman changed the title feat: use medium reasoning for GPT-5.6 Sol feat: medium reasoning for GPT-5.6 Sol Jul 29, 2026
@mattrossman
mattrossman merged commit f8434fe into main Jul 30, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant