Repository navigation
Releases
mmPISA · Ling 3.0 Flash · Japanese / Spanish
Full mmPISA official human-translation evaluation of inclusionai/ling-3.0-flash through OpenRouter, Japanese (ja) and Spanish (es), low/medium/high requested reasoning effort.
| Effort | Japanese | Spanish |
|---|---|---|
| low | 24/25 (96%) | 22/25 (88%) |
| medium | 24/25 (96%) | 23/25 (92%) |
| high | 22/25 (88%) | 24/25 (96%) |
Protocol: multiple-choice-v1; one repeat; seed 42; combined reasoning/output cap 32,768 tokens. All 150 jobs completed in 150 attempts, without truncations or transport failures. One low/es answer was unparseable and scored zero, without retry. Costs are unknown in the published accounting because no pricing table was configured; this does not mean zero cost.
Verified with pnpm bench release verify. The attached manifest records checksums and provenance; public artifacts omit private API reasoning and raw SDK responses. Post-run paired comparisons use Japanese minus Spanish and are exploratory; 25 questions do not establish a reliable effort ranking. This is the Llang Gap evaluation condition, not a reproduction of the paper's prompting setup. Existing releases remain unchanged.
mmPISA: Ling 3.0 Flash · EN/RU · low/medium
First complete published Llang Gap benchmark: inclusionai/ling-3.0-flash through OpenRouter on mmPISA official human translations, English and Russian, with requested reasoning efforts low and medium.
| Requested effort | EN | RU | EN − RU (pp) | Paired 95% interval (pp) |
|---|---|---|---|---|
| low | 24/25 (96%) | 23/25 (92%) | +4 | [-8, +16] |
| medium | 24/25 (96%) | 22/25 (88%) | +8 | [0, +20] |
All 100 jobs completed in 100 attempts, with no unparseable answers, refusals, truncations, missing usage or unresolved jobs. Each condition uses all 25 questions and one repeat. Intervals include zero; these results do not establish a directional language advantage or an effort-level improvement.
This uses Llang Gap's zero-shot multiple-choice-v1 protocol, not a reproduction of the paper's prompting or model results. The output cap is 32,768 tokens including reasoning. Effort labels record the exact OpenRouter request; they do not establish separate native compute levels. OpenRouter routing is not pinned to a serving endpoint.
Pricing was not configured, so aggregate USD costs remain unknown (null). Token usage is preserved. No retrospective rates or cost estimates have been inserted.
Dataset revision: 22b1caa65980ca1fa295acc31ee5e35200e0c8c0.
Runner source: 69de84a7e4262c95f35f83fe994f684aefa2ce7e (clean at run creation).
Run: mmpisa-ling-3-flash-low-medium-en-ru-2026-09-08-98071858.
The attached manifest contains SHA-256 checksums for all other artifacts. The release passed pnpm bench release verify. To independently verify, download all assets into one directory and run that command on the directory using the recorded source revision and pinned dependencies. Raw SDK responses, API reasoning text, credentials and the private SQLite journal are not included. See ATTRIBUTION.md for dataset provenance and terms.
mmPISA · Ling 3.0 Flash · 12 languages · high / low
Complete mmPISA human-translation benchmark for inclusionai/ling-3.0-flash through OpenRouter.
- Protocol: multiple-choice-v1.
- Languages: sq, cs, zh, en, de, fr, ko, kk, ru, sr, sk, sv.
- Efforts: high and low; one repeat; all 25 questions per language.
- 600/600 responses, 600 attempts; no transport failures, refusals or truncation. Unparseable answers are retained and scored as incorrect.
- No spending or output cap was supplied; costs are unknown because token prices were not configured.
- Run: cli-2026-09-08-60fa7581. Source: 09aa411.
- Independently verified with bench release verify. Public files include checksums and reproducible prompt/response evidence; private SQLite state is excluded.
This is a new immutable release. Earlier releases remain unchanged. No paired language comparisons were requested.
The current homepage guide pins a 32,768-token output cap. This run omits that cap, so its results are available in release history but are not eligible to replace that guide's pinned scores. There are 69 unparseable answers (35 high, 34 low), all scored as incorrect.
mmPISA · Four models · ES/ZH/EN/KK/RU · high / low
Complete mmPISA benchmark using multiple-choice-v1, with Spanish, Chinese, English, Kazakh, and Russian; high and low effort; one repeat; concurrency 5.
Models: upstage/solar-pro4, qwen/qwen3.7-flash, openai/gpt-oss-120b, and inception/mercury-2.5-preview, through OpenRouter.
All 1,000 requested answers are saved and independently verified, with no truncated responses. Incorrect and unparseable completed answers remain in the scores. No language comparisons were selected for this release.
The execution ledger retains all 1,399 attempts, including HTTP 429 failures and timeouts. Following explicit operator authorization, failed requests were resumed beyond the CLI's five-attempt ceiling through the unchanged runner's resumeRun function, increasing the total attempt limit one at a time to nine and pausing 45 seconds between the final recovery passes. Completed responses were never reissued; the original configuration, dataset, source, and prior attempt history remain intact.
Pricing was not configured, so published token-based cost totals remain unknown. Private SQLite state and raw SDK responses are excluded; the attached artifacts include visible answers, scoring inputs, attempt history, provenance, attribution, and checksums.
PR evidence
Review screenshots only; not a benchmark or software release.