R2T2 Plugin

[scriptease]

Claude and I implemented a plug in for the R2T2 Model and it implements streaming. So you can use a way bigger model. In my case I'm using a 2.5 gigabyte model and it is half real time. But you don't pay the costs since it's transcribing while you're speaking. If anybody is has an interest here's the GitHub project.

it's work in progress since model is very new and it still needs some audio.cpp upstream requests.

https://github.com/scriptease/typewhisper-r2t2-plugin

Is there a process to get your own plug-in into the app?

[SeoFood]

Nice work, and thanks for sharing it. Streaming a bigger local model so the latency hides behind your speech is exactly the kind of thing plugins are meant for.

Yes, there's a process. We build, sign and host community plugins ourselves, so the source has to live in this repo. A submission PR would include:

• the plugin source in TypeWhisperPluginSDK/Plugins/R2T2Plugin/ plus the SwiftPM/Xcode targets (your patch already covers this)
• a slug-to-target line in .github/workflows/plugin-release.yml
• a registry entry at PluginRegistry/community-v1/<plugin-id>.json with "source": "community" and no releases[]

After review we run the release workflow, and the plugin shows up in the Integrations tab with a Community badge. The MemPalace plugin went through the same steps (#588, #754) if you want an example.

A few things to change first:

• Please use your own ID and author (e.g. com.scriptease.r2t2). com.typewhisper.* makes it look like an official plugin, and registry validation will reject it once chore: reserve the TypeWhisper namespace for community plugins #1392 is merged.
• Add "capabilities": ["live-dictation"] to the manifest. It's new in the current daily build: without it, TypeWhisper falls back to one batch request after you stop recording when the preview is hidden, and you lose the streaming benefit.
• Make the description say clearly that a running audio.cpp server is required, since the plugin doesn't start it.

Since you're still waiting on audio.cpp upstream, it's fine to open the PR once that settles. Happy to review it then.

[scriptease]

Thanks @SeoFood, that's exactly what I needed.

I've made the changes on my branch:

• ID and author are now com.scriptease.r2t2 / scriptease, including the Xcode bundle identifier
• "capabilities": ["live-dictation"] is in the manifest
• both descriptions say a running audio.cpp server is required
• the r2t2 slug is in plugin-release.yml, and the registry entry is PluginRegistry/community-v1/com.scriptease.r2t2.json

The registry validator passes, including the stricter one from #1392. I'll open the PR once the audio.cpp side has landed.

One question for later. Would a community plugin be allowed to start the server itself, the way the Cohere plugin runs CrispASR? The idea is:

• download a pinned audio.cpp release asset (bin-macos-arm64-metal, Apache-2.0) from its GitHub release and check its SHA-256
• download the chosen GGUF from Hugging Face (Q4_K_M, Q8_0 or F16)
• run audiocpp_server as a supervised child process on a random loopback port, and stop it on unload or quit

That would turn it into "pick a model, download and run" with nothing to install separately. If running an externally built binary doesn't fit how you sign and host community plugins, I'll keep the user-run server.

[SeoFood]

Thanks for turning that around so quickly.

Yes, a community plugin can manage the server itself. The Cohere plugin is the pattern to follow, and the same rules would apply:

• pin the audio.cpp release in source: exact version, file size and SHA-256, no "latest" lookup. Updates go through a new plugin release that we review.
• pin the GGUFs to a Hugging Face revision and check their SHA-256 too. That matters here because the quants come from individual uploaders rather than NetEase.
• bind to 127.0.0.1 on a random port and make sure the server can't outlive TypeWhisper. CrispAsrServer.swift in the Cohere plugin has a small watchdog for the case where the app crashes, so you can reuse that approach.
• show the audio.cpp and model licenses in the plugin settings, like Cohere does for CrispASR.

I'd keep the user-run server as an option next to it, since some people will want their own build.

[scriptease]

Thanks, will follow!

[scriptease]

The plugin is now submitted as #1432, following the steps and rules from this thread: own ID and author, live-dictation, a registry entry and a release slug. It runs audio.cpp itself: a pinned v0.9.0 release and pinned Q4_K_M, Q8_0 and F16 GGUFs, each checked by size and SHA-256, on a random loopback port with a watchdog, and with the licenses shown in the settings. The user-run server stays an option.

Thanks @SeoFood for the guidance and the review. Closing this in favour of the PR.

[SeoFood]

#1432 is merged and the plugin is out as 0.2.0. Anyone on TypeWhisper 1.7.0 (currently RC2 or a daily) can install it from Settings > Integrations > Discover.

Thank you, @scriptease, for building this and for seeing the whole submission through.