Models
| Model | PR | Average generation time | Cost | Average JSON size | Max JSON size |
|---|---|---|---|---|---|
| GPT 6 Astra Pro | #152 | 25m 53.5s (1553.5s) | $34.71* | 128.53 MiB | 316.39 MiB (worldtree) |
- Estimated total cost; the provider dashboard currently reports $0 and has not yet updated.
Notes
GPT-6 Astra Pro was quite a surprise. It might not have looked all that exciting on other benchmarks, but I think this is the biggest jump we've seen from a model on MineBench so far.
Average generation time stayed roughly the same as the previous generation of GPT models, but the cost looks to be much lower. GPT-5.6 Sol cost us around $700 to benchmark; the current estimate for GPT-6 Astra Pro is $34.71, though the provider dashboard hasn't updated yet. I suspect the lack of retries is part of that. Every build was valid on the first attempt, which was pretty crazy.
I'm not sure I agree with Greg Brockman that this model is AGI, and I haven't used it enough to weigh in on that. What does stand out is how well it seems to understand what matters to a human looking at the finished build. It knows when to add extra scenery and when to focus on the object the prompt actually asked for, in a way I haven't seen from another model yet.
What's Changed
Models
- Added GPT 6 Astra Pro with native OpenAI Responses routing and an
openai/gpt-6-astra-proOpenRouter fallback, Pro mode at max reasoning effort, strict structured output, provider-default sampling, and a 128,000-token output cap. #152 - Completed the 15-build benchmark with 25m 53.5s average generation time, 128.53 MiB average JSON size, 316.39 MiB maximum JSON size, and 15 completed attempts with no rejected responses. Estimated total cost: $34.71* ($2.31* per attempt). #152
Leaderboard
- Added cost, speed, and block-count Pareto frontiers alongside rankings, with model details on hover or keyboard focus. #158
- Added sortable cost, time, and block comparisons per build or per observed prompt-score point, with estimated costs and provisional models clearly marked. #158
- Leaderboard comparisons refresh automatically as measurements and rankings become available. Improved tabs, model-name visibility, scrolling, and mobile layouts. #158
Private Lab
- Moved private checkpoint generation onto the shared durable worker and added encrypted per-model request header and body overrides while preserving generation and output-schema contracts. #147
- Added resumable per-prompt checkpoint uploads, worker-based processing, current and archived checkpoint views, and streamed payload persistence for large builds. #148
- Improved failed-upload retries, cleanup, and in-place refresh of outdated checkpoints. #148
Reliability
- Prevented conflicting output-token overrides and aligned custom request-body size limits with UTF-8 byte counts. #153 #154
- Stopped repeated OpenRouter configuration fallbacks after authentication, rate-limit, and server failures exhaust their applicable retries. #155
- Redirected expired password-reset links to the recovery form. #156
- Rejected private endpoint credentials that become empty after normalization when saving them. #157
Documentation
- Documented refreshing outdated upload checkpoints in place. #149
- Clarified export-format block-ID preservation requirements. #150
Changelog
Full Changelog: 4.2.0...4.3.0
- #147 feat(generation): move private evaluations to the shared worker @Ammaar-Alam
- #148 feat(lab): add durable checkpoint uploads @Ammaar-Alam
- #149 docs: note outdated upload checkpoints can be refreshed in place @detail-app[bot]
- #150 docs(contrib): relax export-format block-ID preservation rule @detail-app[bot]
- #152 feat(models): add GPT-6 Astra Pro benchmark support @Ammaar-Alam
- #153 fix(generate): enforce customBody size limit on UTF-8 bytes @detail-app[bot]
- #154 fix(ai): reconcile output-token aliases when merging custom request body @detail-app[bot]
- #155 fix(openrouter): abort config fallback on unhandled HTTP statuses @detail-app[bot]
- #156 fix(auth): route expired password-reset links to forgot-password @detail-app[bot]
- #157 fix(stealth): reject empty-after-normalization api key at encrypt time @detail-app[bot]
- #158 feat(leaderboard): add interactive efficiency comparisons @Ammaar-Alam
