Show and tell: reproducible Pi vs DSH benchmark on the same local model + working oMLX/OpenAI-compat config for the headless generic route

[gltanaka]

What and why

I ran a 16-run, coding-only benchmark of dsh-v0.1.1-rc.2 (npm @deepseek-ai/dsh@0.1.1-rc.2, commit b150a551b8d465e31e418e1b2eaf5e79bbb7d28e) in the headless profile against another harness (Pi 0.73.1), with both harnesses driving the same local model: Qwen3.8-27B-MLX-oQ8e-mtp on oMLX 0.6.1, M4 Max / 128 GB. DSH never ran a DeepSeek model here — the point was to hold model, server, sampling, tasks, prompt suffix, isolation, and the 1,200 s wall cap fixed and let the harness be the only variable. 4 OpenBench tasks x 2 trials, hidden checker.sh run outside the harness, Seatbelt isolation, a body-free metering proxy, byte-locked toolchains, and full repro commands.

I'm posting mainly because getting the generic route pointed at a non-DeepSeek OpenAI-compatible endpoint took real work, and the working config may save someone else a day.

The working config

DSH's adapter docs otherwise assume OpenAI-style developer role, max_completion_tokens, and bare reasoning_effort; oMLX accepts none of those, so each needed an explicit compat: override. I also declared the exact model and capacities rather than accepting the generic route's fallbacks (262,144 context / 32,768 output, no reasoning unless declared), and set maxRetries: 0 so a DSH cell got exactly one attempt, matching Pi.

api: openai-completions
baseURL: http://127.0.0.1:<per-cell-proxy>/v1
reasoning: medium
retryPolicy: {mode: normal, maxRetries: 0}
compat:
  supportsDeveloperRole: false
  supportsReasoningEffort: false
  supportsUsageInStreaming: true
  maxTokensField: max_tokens
  thinkingFormat: qwen-chat-template
models:
  - id: Qwen3.8-27B-MLX-oQ8e-mtp
    contextWindow: 98304
    maxTokens: 32000
    reasoningEfforts: {medium: medium}

dsh --profile headless --patch <patch.yml> --dump-config was genuinely useful here — the runner resolves the composition and asserts on it before booting any inference, which is how I caught overrides that weren't landing.

Results, with the caveats up front

Pi passed 6/8 externally checked runs to DSH's 4/8 (mean external scores 0.825 vs 0.680). The entire margin comes from one fixture. Both separating pairs were taskflow, where DSH hit the cap with partial scores of 0.28 and 0.56; make-ci-green and add-feature were 2/2 for both, and webcore was 0/2 for both. Paired: 2 wins, 0 losses, 6 ties. Four fixtures and two trials — two non-tied pairs — is far too small for a general ranking, and I make no significance claim. This is a lead on the measured sample, nothing more. DSH is an RC/developer preview, and some of this plausibly reflects config-surface immaturity on a route it wasn't built for rather than the harness's steady state.

The observation likely most useful here: DSH kept 19 model-facing tool schemas per request to Pi's 4 stock built-ins, and that showed up in the numbers — +11.9% completed-response input tokens (1,130,071 vs 1,009,881), +23.8% mean time to first token (38.87 s vs 31.40 s), and +14.0% total wall time (7,356.5 s vs 6,455.5 s). I equalized the capability boundary for network, filesystem, and model concurrency, but not schema count, so this is a whole-harness comparison and not a tool-schema-matched ablation. On a small local model, that fixed per-request schema tax is not free.

Please correct me

If I misconfigured the generic route, or there's a better-supported way to point headless DSH at a local OpenAI-compatible endpoint, I'd genuinely like to know and will re-run and amend the writeup.

• PR: research: benchmark Pi versus DeepSeek Harness promptdriven/pdd#2414
• Full README (method, cell-level table, limitations, repro commands): https://github.com/promptdriven/pdd/blob/main/research/omlx-qwen38-pi-deepseek-harness-2026-08-23/README.md

Disclosure: I have no affiliation with deepseek-ai or earendil-works. Pi is the harness I use day to day, which is why it's the reference arm — treat that as a bias to check, not a neutral choice. The repo above is pdd, a third-party open-source CLI of mine; it's where I keep the benchmark code and is not an arm of either project.