Share your results: which dependency APIs does your model get wrong?

[MohammadHijjawi97]

If you ran since-cutoff on your own project, please share what it found here. Real results from different projects and models are the best way to learn where coding models go wrong, and they help decide what to support next.

Useful things to include:

• the model and its training cutoff (since-cutoff models lists them)
• the libraries and versions that were flagged (since-cutoff scan)
• if you ran since-cutoff run: the stale / wrong / deprecated counts, and whether the notes fixed the held-out tasks
• anything that looked wrong: a false positive, a change it missed, a note that did not help

The JSON and Markdown reports in .since-cutoff/ contain all of this. Please remove anything private first.

[MohammadHijjawi97]

To get this started, here is a scan of a project that pins 36 widely used Python AI libraries at their 26 September 2026 releases (examples/ai-stack), against 21 models' training cutoffs. scan only, no model calls, since-cutoff 0.2.0.

model (cutoff)	libraries with a public API break since	on a new major version
gpt-4o (2023-09)	21 of 36, and 13 did not exist yet	15
gpt-5 (2024-09)	30	18
gemini-2.5-pro (2025-01)	33	21
claude-haiku-4-5 (2025-02)	33	21
claude-opus-4-6, deepseek-v4-flash (2025-05)	33	20
gpt-5.5 (2025-12)	29	13
claude-opus-4-7 (2026-01)	29	11
gpt-6-sol (2026-04)	25	10
grok-4.7 (2026-05)	24	8
claude-opus-5-5 (2026-06)	20	6

The full table and chart are on What your model hasn't seen. Things that looked wrong while doing this, now fixed or being fixed: transformers inherited members reported as removed (fixed in 0.2.0), and noise from typing-stub packages and *args changes (0.3.1).

What I'd most like to see from others: a run on your own project with a non-Claude model (--model openai:..., openrouter:... or ollama:...), and any false positive in the diff.