You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We publish measured transcription times at https://scenaristo.com/benchmarks, on four full length podcast episodes that are public on YouTube. The files and their checksums are published for one reason: so you can run the same test instead of taking our word for it.
This thread is where those results go. Ours cover one Mac and one Windows machine. Yours are the ones that tell us whether the numbers hold on hardware we do not own.
Download with yt-dlp, selecting the H.264 (avc1) video stream and AAC audio. Nothing gets re-encoded, trimmed or cleaned first.
Then check you have the same bytes we timed:
shasum -a 256 <file> # macOS
sha256sum <file> # Linux
certutil -hashfile <file> SHA256 # Windows
These are the four fingerprints:
Smosh, Unpredictable daf57b2985a06f1fabec26766c4fcaf11797d6f82af8f1adc95954e18be56552
Smosh, Plot Twist 21e94d714148e465167d47022355995cf14c67e76256d90800812a20243c34e1
La Ruina 205 cceaad4591dbe3b10bd72d00a3e4ca52a036ab209f5b344779f4852c071462d5
La Ruina 222 f6c4a0578949e2a9129f1014904466261b60eee0d1d921983fe3c0f4ed31d9a7
If yours does not match, you have a different encode, and the time it produces is not comparable with ours. Post it anyway and say so.
How to measure it
Do not hold a stopwatch. Scenaristo times itself.
Open Preferences and turn on Benchmark mode. It is off by default.
Import the file and transcribe. Leave your network alone: transcription does not use it, and pulling it offline before the model has downloaded turns a working run into a failed one.
When the run finishes, a Benchmark card appears in the transcript side panel: the machine, the build, and one row per pass with its audio length, its processing time, and RTFx.
Press Copy on that card.
RTFx is audio divided by processing time, so one number compares a 40 second clip with a two hour episode. RTFx 58× means it got through 58 seconds of audio for every second you waited.
Transcript and Diarization are listed as separate rows and are never summed. They are different engines with RTFx an order of magnitude apart, so adding them together would hide the thing worth seeing.
That is the whole ask. If something is missing from your card, post it anyway. A result with a gap in it is worth more than no result.
Two things worth knowing about what the card counts. Loading the model's weights is inside the processing time; downloading a model you do not have yet is not, so a first run on a cold cache is not reported as a slow one. And if you were on battery, plugged in, or had something heavy running alongside, say so in a line underneath.
What happens to them
Nothing you post here goes on the site as a measurement of ours. The published table stays what we measured ourselves, on machines we can re-run on demand, because that is what makes it re-checkable on every release.
What your results do change is what we know. A row that lands far off ours on comparable hardware is a bug report, and we would rather find it here than have the table quietly be wrong. If enough results come in on hardware classes we do not own, we will say so on the page and link back to this thread.
A note on what is not being asked
This is not a comparison against other tools. We do not publish those, because we have not run anyone else's software on this hardware with these files and so have no basis to. Post what Scenaristo did on your machine, and that is plenty.
Nothing is uploaded by the app during any of this, though do read your Machine line before you paste it: it names your model, chip, core counts, memory and OS version. If you would rather not post that publicly, hello@scenaristo.com reaches us instead.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
We publish measured transcription times at https://scenaristo.com/benchmarks, on four full length podcast episodes that are public on YouTube. The files and their checksums are published for one reason: so you can run the same test instead of taking our word for it.
This thread is where those results go. Ours cover one Mac and one Windows machine. Yours are the ones that tell us whether the numbers hold on hardware we do not own.
What to run
Any of the four episodes:
Download with yt-dlp, selecting the H.264 (avc1) video stream and AAC audio. Nothing gets re-encoded, trimmed or cleaned first.
Then check you have the same bytes we timed:
These are the four fingerprints:
If yours does not match, you have a different encode, and the time it produces is not comparable with ours. Post it anyway and say so.
How to measure it
Do not hold a stopwatch. Scenaristo times itself.
RTFx is audio divided by processing time, so one number compares a 40 second clip with a two hour episode. RTFx 58× means it got through 58 seconds of audio for every second you waited.
Transcript and Diarization are listed as separate rows and are never summed. They are different engines with RTFx an order of magnitude apart, so adding them together would hide the thing worth seeing.
What to post
Paste what Copy gave you. It looks like this:
Then add the two things the card cannot know:
That is the whole ask. If something is missing from your card, post it anyway. A result with a gap in it is worth more than no result.
Two things worth knowing about what the card counts. Loading the model's weights is inside the processing time; downloading a model you do not have yet is not, so a first run on a cold cache is not reported as a slow one. And if you were on battery, plugged in, or had something heavy running alongside, say so in a line underneath.
What happens to them
Nothing you post here goes on the site as a measurement of ours. The published table stays what we measured ourselves, on machines we can re-run on demand, because that is what makes it re-checkable on every release.
What your results do change is what we know. A row that lands far off ours on comparable hardware is a bug report, and we would rather find it here than have the table quietly be wrong. If enough results come in on hardware classes we do not own, we will say so on the page and link back to this thread.
A note on what is not being asked
This is not a comparison against other tools. We do not publish those, because we have not run anyone else's software on this hardware with these files and so have no basis to. Post what Scenaristo did on your machine, and that is plenty.
Nothing is uploaded by the app during any of this, though do read your Machine line before you paste it: it names your model, chip, core counts, memory and OS version. If you would rather not post that publicly, hello@scenaristo.com reaches us instead.
All reactions