nova-3-pharma ~1 s slower than nova-3-medical on prerecorded REST (reproducible, request IDs included) #1709
Unanswered
ryanshrott
asked this question in
General help
Replies: 1 comment
|
Thanks for asking your question. Please be sure to reply with as much detail as possible so the community can assist you efficiently. |
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
We are seeing consistently higher prerecorded REST latency for nova-3-pharma in DictaFlow Medical, and wanted to share a reproducible benchmark. We have moved our Medical traffic to nova-3-medical meanwhile and would switch back if Pharma latency matches it.
On October 26, 2026 (20:49-21:15 UTC), we ran the actual production transcription function from Azure Canada Central against api.deepgram.com/v1/listen. We used one persistent Python requests.Session (requests 2.33.0 / urllib3 2.8.0), sequential calls, identical synthetic 5/12/25-second mono 16 kHz FLACs within each round, and randomized model order. All 135 calls returned HTTP 200, without retries or failover.
All three models used the same options: language=en, dictation=true, smart_format=false, punctuate=true, numerals=true, filler_words=false, utterances=true, paragraphs=false, and the same 12 generic clinical keyterms. No customer dictionary was used.
Provider-function wall time, milliseconds (excludes recording, client upload to our backend, and subsequent LLM formatting):
The paced arm inserted 8-20 seconds of idle before every call; burst had no intentional idle. The paired Pharma-minus-Medical median was +992 ms (95% bootstrap CI: +438 to +1,243 ms), with Pharma slower in 33/35 matched rounds. This is not simply the difference between the marginal medians.
Instrumentation places most of the excess in waiting for HTTP response headers, not local wrapper work. It also occurs on established connections: the 21:15:03.597Z Pharma call below took 1,988 ms total, including 1,986 ms waiting for headers, with no DNS/TCP/TLS setup. We cannot distinguish inference from queueing or routing without your traces.
Production timing logs corroborate the direction against general nova-3 on each available UTC date, October 23-26, and in all 11 hour-of-day bins with at least five calls per model. Those observational logs are not a randomized comparison; no customer data is included here.
Eight synthetic-test request IDs follow. The first two pairs are paced (12s and 25s audio); the last two are burst (5s and 25s). Each pair used identical audio:
Returned model versions: pharma-nova-3 2026-06-05.22845; medical-nova-3 2026-09-30.28661; general-nova-3 2025-07-31.0.
Separately, one synthetic dosage example produced a numeric formatting error: Pharma rendered 250 milligrams as "2 50 milligrams", while Medical and general Nova-3 returned the correct number. This was a limited synthetic check, not an accuracy benchmark.
Could you confirm:
Only synthetic-test request IDs and aggregate timings are shared; no customer audio, transcripts, or user identifiers are attached. We can provide synthetic fixtures and a minimal repro privately if useful.
Ryan Shrott, Founder, DictaFlow (dictaflow.io)
All reactions