Skip to content

3.12.0: GPT 5.6 Luna + Qwen 3.8 Max + Muse Spark 1.2 + DeepSeek V4 Flash

Latest
Latest

Choose a tag to compare

View all tags
@Ammaar-Alam Ammaar-Alam released this 07 Aug 06:31
3.12.0
ff804f6

Models

Model PR Average generation time Cost Average JSON size Max JSON size
DeepSeek V4 Flash 0731 #74 8m 46.3s (526.3s) $0.28 17.56 MiB 56.96 MiB (worldtree)
Qwen 3.8 Max #75 24m 23.0s (1463.0s; 5/15 tracked) $11.53 14.65 MiB 99.76 MiB (worldtree)
GPT 5.6 Luna #76 13m 24.5s (804.5s) $1.15 57.19 MiB 198.43 MiB (phoenix)
Muse Spark 1.2 #77 3m 57.5s (237.5s) $2.61 14.35 MiB 82.84 MiB (worldtree)

Notes

In our testing, DeepSeek V4 Flash didn’t come nearly as close to Opus 4.8 as some other benchmarks may suggest, and the quality of its outputs was rather inconsistent.

Outside of comparisons to frontier models, however, V4 Flash remains quite impressive; at just $0.28, it becomes the cheapest model we’ve benchmarked to date – a distinction previously held by Gemini 3.5 Flash-Lite at $0.38, though 3.5 Flash-Lite remains considerably faster.

Qwen 3.8 Max was the biggest outlier in this set. The model struggled to translate its reasoning into reliable spatial programs. In observed runs, it spent 95–99% of its native completion tokens on reasoning, then frequently emitted brittle JavaScript containing single-line comments that swallowed code, undefined variables, syntax errors, or programs that produced no blocks at all. Even valid outputs favored repetitive volume over coherent, recognizable geometry.

We first believed this to an issue with our implementation, and removed MineBench's forced 1.0 temperature setting, allowing the provider to use the model's documented 0.6 default value, but this did not materially improve its reliability or visual quality. Out of curiosity, we also attempted to lower the reasoning parameter as there have been cases where models produce better outputs on lower reasoning efforts, where they don't consume their entire output budget on their CoT (sometimes the case with Anthropic models). However, the results were the same on both xhigh, high, and medium reasoning efforts, with the only improvements being in latency.

GPT 5.6 Luna was the best model at this price-range, costing only $1.15 while outputting builds that are (subjectively) near the quality of Kimi K3 and Opus 4.8

Muse Spark 1.2 was infinitely better than Meta's previous attempt on MineBench (Llama 4). The model outputted very solid and consistent builds considering its low cost.

What's Changed

Models

  • Added DeepSeek V4 Flash 0731 with direct DeepSeek routing, the versioned OpenRouter fallback, a 384,000-token native output cap, max reasoning by default, and provider-specific reasoning normalization. #74
  • Added Qwen 3.8 Max through OpenRouter with a 1M-token context window, a 131,072-token output cap, xhigh reasoning, strict structured output, and provider-default sampling. #75
  • Added GPT 5.6 Luna through OpenAI Responses with an OpenRouter fallback, pro mode, max reasoning, high text verbosity, strict structured output, and a 128,000-token combined reasoning and output cap. #76
  • Added Muse Spark 1.2 with native Meta routing, an OpenRouter fallback, mandatory reasoning, strict structured output, and a 131,072-token output cap. #77
  • Added completed benchmark profiles and cohort metrics for DeepSeek V4 Flash, Qwen 3.8 Max, GPT 5.6 Luna, and Muse Spark 1.2. #74 #75 #76 #77
  • Published Qwen 3.8 Max benchmark statistics: 24m 23.0s average generation time (5/15 tracked), 14.65 MiB average JSON size, and 40 completed attempts. ff804f6

Generation reliability

  • Allowed Qwen 3.8 Max to use provider-default sampling instead of MineBench's shared temperature override. #75
  • Preserved cross-realm voxel execution failures in batch generation and local conversion so model-program errors are no longer mislabeled as provider failures. #75
  • Raised the voxel execution limit from 12 to 30 seconds and avoided serializing oversized intermediate block lists during normal generation. #76
  • Added Meta Model API credentials, request validation, admin status, Sandbox configuration, and automatic OpenRouter fallback when a direct Meta key is unavailable. #77

Leaderboard

  • Reworked the champion header into responsive champion, dataset-status, and table-control areas across desktop, tablet, and mobile layouts. #73
  • Increased the champion rank numeral, aligned the medallion with the capsule edge, and simplified narrow-screen metadata to prevent crowding. #73
  • Added model and provider search with canonical rank preservation, accessible result announcements, clear and empty states, and mobile-first placement. #73
  • Kept the Details disclosure visually stable on desktop and removed it from the mobile card layout where it has no effect. #73

Tests

  • Added focused UI regression coverage for query matching, rank preservation, search accessibility, empty-state recovery, and Details behavior. #73
  • Added provider request-shape and reasoning coverage for DeepSeek V4 Flash, Qwen 3.8 Max, GPT 5.6 Luna, and Muse Spark 1.2, plus voxel-runtime, error-diagnostic, and benchmark-profile regressions. #74 #75 #76 #77

Changelog

Full Changelog: 3.11.1...3.12.0

Assets 2
Loading