Feature Request: Speech-To-Shell

[iSteeb]

With the implementation of the new text editor popup, it opens the door to an idea that I've been wishing for in a mobile terminal app: natural dictation. Apple's dictation model obviously doesn't cut it, but I've been playing around with chat models / REST and then third party apps and it works excellently, except for the quite heavy inconvenience of it not being native to the application I want to write in. Switching apps to record, copy and then paste is really bad for productivity, especially on the Apple Vision. Two tiers of implementation:

First, simply using the API you've already built into the app. Add a dictation button to the text editor popup that records audio and sends the request. This is what I've been playing with, to great success. Would be nice to be able to set a custom prompt, so that users could add references to their custom tooling for example.

{
    "model": "google/gemini-2.5-flash",
    "messages": [
      {
        "role": "system",
        "content": "You translate spoken natural language into a single valid POSIX shell command. Output ONLY the command. No explanations. No backticks."
      },
      {
        "role": "user",
        "content": [
          {
            "type": "input_audio",
            "input_audio": {
              "data": "base64_encoded_audio_here"
              "format": "m4a"
            }}]}]}

Second tier would be to add offline recording. Lately I've been using the Parakeet model with Superwhisper. Its lightning fast, only 450mb, and local. But this doesn't seem like a worthwhile implementation tbh, because the translation to POSIX would need to be done online anyway (I briefly tested a couple of tiny LLMs and they don't seem to be smart enough to be reliable for this purpose - with further testing and choosing the right model, I'm sure there would be a good option, but frankly I don't think its worth the effort).

I do think the relatively minimal work (considering you already have AI APIs built into the app) would be a major usability function and would take this app from top 3 terminal apps on the app store, to a clear productivity #1.

[kitknox]

I also already have experience doing on device ASR and even speech diarization.

I wrote a podcast app that does the audio work locally and then with the transcript data hands that off to a big model online for the chapter generation and ad marker generation. Optionally can then automatically skip ads.

Another interesting thing is on the iPhone you could be interacting without the virtual keyboard in the way. So many possibilities here.

A difference here is the audio durations are so much smaller than podcasts. Doing the podcast fully remote is way more expensive. For this the data to send and cost is totally different.

[iSteeb]

A command costs a fraction of a cent to transcribe and translate, even with Gemini (cheaper models out there which might be good enough and cheaper), and most people who are inclined to use an AI API are probably okay with that cost. Like you said, they’re short.

Privacy is another matter, but considering this particular use case is just writing a command… I’d be inclined to say that it’s a reasonably acceptable loss of privacy.

Fully offline would be amazing, and if you have experience and want to tackle it, I would certainly appreciate and use the feature. But I think it’s a lot more work for a pretty minimal benefit.

I like what you’re thinking about with the hidden keyboard too. Working in vim in landscape mode without missing half the screen would be nice.

[kitknox]

I am currently testing a branch with a prototype using a websocket streaming interactive audio to the Google Live Flash API. It then offer tool calls to the model allowing (with approval) for it to access your scroll back history, request a screenshot of the rendered terminal, and run commands. It’s pretty wild to have this kind of engagement where you can just interrupt it talking to give it new instructions vs a walkie-talkie type of approach where you do ASR and get text back from a model that you can optionally generate voice again from.

The problem is that the live models are not as good on the agent tasks themselves so it is a trade off.