feat: enhance voice features with Smallest.ai integration and Jarvis wake functionality: - #255
Conversation
…wake functionality: - Updated voice capture to exclusively use Smallest.ai for Speech-to-Text (STT) with new environment variables. - Introduced Text-to-Speech (TTS) capabilities using ElevenLabs, with fallback options for macOS. - Added Jarvis wake functionality to launch daily workspace apps via voice commands. - Updated documentation and example environment files to reflect new configurations and usage instructions.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
WalkthroughThe CLI now uses Smallest.ai for speech-to-text, supports macOS spoken replies, and adds gated Jarvis wake-word, workspace startup, and ChangesVoice and Jarvis features
Estimated code review effort: 4 (Complex) | ~45 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 2📝 Generate docstrings 💡
🛠️ Fix failing CI checks 💡
🧪 Generate unit tests (beta)
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 ESLint
ESLint install failed. For unrecoverable errors, disable the tool in CodeRabbit configuration. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 7
🧹 Nitpick comments (1)
apps/supercode-cli/server/src/voice/__tests__/speech.test.ts (1)
47-55: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueUse double-quoted test names.
Lines 47 and 52 use single-quoted strings. Use double-quoted strings.
As per coding guidelines, use double-quoted strings.
Proposed fix
- it('always returns "smallest" (Smallest.ai is the only STT provider)', async () => { + it("always returns \"smallest\" (Smallest.ai is the only STT provider)", async () => { ... - it('ignores STT_PROVIDER and still returns "smallest"', async () => { + it("ignores STT_PROVIDER and still returns \"smallest\"", async () => {🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@apps/supercode-cli/server/src/voice/__tests__/speech.test.ts` around lines 47 - 55, Update the two test descriptions in the speech provider tests to use double-quoted strings instead of single-quoted strings, while preserving their existing text and assertions.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@apps/supercode-cli/server/README.md`:
- Around line 47-53: Update the voice capture prerequisites section in the
README to state that `ffmpeg` is required along with either direct Smallest.ai
credentials or an authenticated `SUPERCODE_SERVER_URL` server proxy. Keep the
existing environment-variable table unchanged.
In `@apps/supercode-cli/server/src/cli/ai/chat/chat.ts`:
- Around line 1583-1587: Ensure the voice-submit state is consumed per submitted
turn rather than leaking into later responses: update the auto-submit flow
around commitInput() and the slash-command handling so /jarvis, /help, /model,
and other non-agent commands clear or otherwise isolate voiceAutoSubmitted,
while preserving voice playback for the current agent response.
In `@apps/supercode-cli/server/src/voice/__tests__/jarvis.test.ts`:
- Around line 4-8: Isolate both default-value tests from inherited environment
configuration: in apps/supercode-cli/server/src/voice/__tests__/jarvis.test.ts
lines 4-8, clear JARVIS_WAKE_PHRASE before calling getWakePhrases; in lines
37-63, clear every target override environment variable before asserting the
default GitHub, WhatsApp, Slack, Linear, Twitter, and Warp targets.
In `@apps/supercode-cli/server/src/voice/__tests__/speech.test.ts`:
- Around line 90-95: Update the isTtsAvailable test setup so VOICE_REPLY is
explicitly set to "on" before importing speech.ts and asserting the result,
ensuring the expected platform-based boolean is not affected by the external
environment.
In `@apps/supercode-cli/server/src/voice/jarvis.ts`:
- Around line 146-177: Update openUrl to be asynchronous and resolve true only
after the spawned launcher successfully starts, while resolving false on its
asynchronous error event. Replace the Windows start invocation with a
Windows-compatible executable opener that passes configured URLs without shell
reinterpretation, and update runJarvisStart and its callers to await the
Promise<boolean> result before reporting targets as opened.
In `@apps/supercode-cli/server/src/voice/speech.ts`:
- Line 4: Replace the synchronous writeFileSync usage in the speech generation
flow with awaited Bun.write(tmpFile, bytes), and remove writeFileSync from the
fs import while preserving the existing temporary-file path and audio bytes.
- Around line 20-23: Update the Smallest.ai configuration around SMALLEST_MODEL
and SMALLEST_LANGUAGE so the default pulse-pro model always uses English.
Prevent unsupported non-English language/model combinations before audio capture
or direct transcription begins, while preserving configurable languages for
models that support them.
---
Nitpick comments:
In `@apps/supercode-cli/server/src/voice/__tests__/speech.test.ts`:
- Around line 47-55: Update the two test descriptions in the speech provider
tests to use double-quoted strings instead of single-quoted strings, while
preserving their existing text and assertions.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 66c4299f-0e40-48aa-8aed-9ddae94b6d10
📒 Files selected for processing (9)
AGENTS.mdapps/supercode-cli/server/.env.exampleapps/supercode-cli/server/README.mdapps/supercode-cli/server/src/cli/ai/chat/chat.tsapps/supercode-cli/server/src/cli/commands/slashCommands/index.tsapps/supercode-cli/server/src/voice/__tests__/jarvis.test.tsapps/supercode-cli/server/src/voice/__tests__/speech.test.tsapps/supercode-cli/server/src/voice/jarvis.tsapps/supercode-cli/server/src/voice/speech.ts
| Voice capture requires `ffmpeg` and a Smallest.ai API key. | ||
|
|
||
| | Env Var | Description | Default | | ||
| |---|---|---| | ||
| | `STT_PROVIDER` | STT provider (`elevenlabs` or `groq`) | `elevenlabs` | | ||
| | `ELEVENLABS_API_KEY` | ElevenLabs API key (required for ElevenLabs STT) | — | | ||
| | `ELEVENLABS_MODEL` | ElevenLabs model ID | `scribe_v1` | | ||
| | `GROQ_API_KEY` | Groq API key (required when `STT_PROVIDER=groq`) | — | | ||
| | `STT_LANGUAGE` | Transcription language | `en` | | ||
| | `SMALLEST_API_KEY` | Smallest.ai API key (https://app.smallest.ai/dashboard/api-keys) | — | | ||
| | `SMALLEST_MODEL` | Smallest.ai STT model | `pulse-pro` | | ||
| | `SMALLEST_LANGUAGE` | Transcription language | `en` | |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Document the server-proxy transcription path.
Voice capture does not always require SMALLEST_API_KEY. canVoiceCapture() also permits an authenticated SUPERCODE_SERVER_URL proxy. Update this section to state that voice capture requires ffmpeg and either direct Smallest.ai credentials or a configured server proxy.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@apps/supercode-cli/server/README.md` around lines 47 - 53, Update the voice
capture prerequisites section in the README to state that `ffmpeg` is required
along with either direct Smallest.ai credentials or an authenticated
`SUPERCODE_SERVER_URL` server proxy. Keep the existing environment-variable
table unchanged.
| if (stdinResolve) { | ||
| // Loop is idle awaiting input — Clicky-style: auto-submit the spoken | ||
| // command so the agent actually does the thing and speaks back. | ||
| voiceAutoSubmitted = true | ||
| commitInput() |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Consume voice-submit state for non-agent slash commands.
Line 1586 sets voiceAutoSubmitted before parsing the captured input. If the input resolves to /jarvis, /help, /model, or another non-agent slash command, none of the response paths at Lines 2351, 2384, or 2402 clears the flag. The next typed agent response is then spoken unexpectedly.
Store the voice origin per submitted turn, or clear the flag in every slash-command branch that does not produce an assistant response.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@apps/supercode-cli/server/src/cli/ai/chat/chat.ts` around lines 1583 - 1587,
Ensure the voice-submit state is consumed per submitted turn rather than leaking
into later responses: update the auto-submit flow around commitInput() and the
slash-command handling so /jarvis, /help, /model, and other non-agent commands
clear or otherwise isolate voiceAutoSubmitted, while preserving voice playback
for the current agent response.
| test("default wake phrases cover bare Jarvis and wake up variants", () => { | ||
| const phrases = getWakePhrases() | ||
| expect(phrases).toContain("jarvis") | ||
| expect(phrases).toContain("jarvis wake up") | ||
| expect(phrases).toContain("jarvis wakeup") |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Isolate default-value tests from inherited JARVIS_* configuration.
Bun loads .env files automatically. These tests therefore fail whenever a developer configures a custom wake phrase or target URL locally.
apps/supercode-cli/server/src/voice/__tests__/jarvis.test.ts#L4-L8: clearJARVIS_WAKE_PHRASEbefore asserting default phrases.apps/supercode-cli/server/src/voice/__tests__/jarvis.test.ts#L37-L63: clear all target override variables before asserting the default GitHub, WhatsApp, Slack, Linear, Twitter, and Warp targets.
📍 Affects 1 file
apps/supercode-cli/server/src/voice/__tests__/jarvis.test.ts#L4-L8(this comment)apps/supercode-cli/server/src/voice/__tests__/jarvis.test.ts#L37-L63
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@apps/supercode-cli/server/src/voice/__tests__/jarvis.test.ts` around lines 4
- 8, Isolate both default-value tests from inherited environment configuration:
in apps/supercode-cli/server/src/voice/__tests__/jarvis.test.ts lines 4-8, clear
JARVIS_WAKE_PHRASE before calling getWakePhrases; in lines 37-63, clear every
target override environment variable before asserting the default GitHub,
WhatsApp, Slack, Linear, Twitter, and Warp targets.
| describe("isTtsAvailable", () => { | ||
| it("returns a boolean reflecting the platform (darwin only)", async () => { | ||
| const mod = await import("../speech.ts") | ||
| expect((mod as any).getSttProvider()).toBe("elevenlabs") | ||
| const result = (mod as any).isTtsAvailable() | ||
| expect(typeof result).toBe("boolean") | ||
| expect(result).toBe(process.platform === "darwin") |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Set VOICE_REPLY before asserting TTS availability.
isTtsAvailable() returns false when VOICE_REPLY=off, even on macOS. This test currently expects true on macOS regardless of that environment value. Set VOICE_REPLY="on" in test setup, or include it in the expected value.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@apps/supercode-cli/server/src/voice/__tests__/speech.test.ts` around lines 90
- 95, Update the isTtsAvailable test setup so VOICE_REPLY is explicitly set to
"on" before importing speech.ts and asserting the result, ensuring the expected
platform-based boolean is not affected by the external environment.
| function openUrl(url: string, app?: string, uri?: string): boolean { | ||
| if (process.platform === "darwin") { | ||
| if (uri) { | ||
| // Custom scheme deep link — `open warp://…` hands it to the handler. | ||
| try { | ||
| const proc = spawn("open", [uri], { stdio: "ignore", detached: true }) | ||
| proc.unref() | ||
| return true | ||
| } catch { | ||
| return false | ||
| } | ||
| } | ||
| // macOS: `open -a <App> <url>` opens the URL in a specific browser/app. | ||
| const browser = envOr("JARVIS_BROWSER", "Dia") | ||
| const args = app ? ["-a", app, url] : ["-a", browser, url] | ||
| try { | ||
| const proc = spawn("open", args, { stdio: "ignore", detached: true }) | ||
| proc.unref() | ||
| return true | ||
| } catch { | ||
| return false | ||
| } | ||
| } | ||
|
|
||
| const cmd = process.platform === "win32" ? "start" : "xdg-open" | ||
| try { | ||
| const proc = spawn(cmd, [uri ?? url], { stdio: "ignore", detached: true }) | ||
| proc.unref() | ||
| return true | ||
| } catch { | ||
| return false | ||
| } |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy lift
Report launch success only after the launcher succeeds.
spawn() does not throw when open, xdg-open, or another launcher fails to start. The current try blocks therefore return true before an asynchronous error event. runJarvisStart() then reports failed targets as opened.
The Windows branch also invokes start directly, but start is a cmd.exe builtin and is not an executable that spawn() can resolve. Make openUrl() asynchronous, wait for the launcher result, and use a Windows-compatible opener that does not reinterpret configured URLs through a shell.
🧰 Tools
🪛 ast-grep (0.45.0)
[warning] Importing child_process exposes a command-execution surface; ensure any command/argument built from input is validated, and prefer execFile/spawn with an argument array over exec.
Context: import { spawn } from "node:child_process"
Note: [CWE-78] Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection').
(detect-child-process-typescript)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@apps/supercode-cli/server/src/voice/jarvis.ts` around lines 146 - 177, Update
openUrl to be asynchronous and resolve true only after the spawned launcher
successfully starts, while resolving false on its asynchronous error event.
Replace the Windows start invocation with a Windows-compatible executable opener
that passes configured URLs without shell reinterpretation, and update
runJarvisStart and its callers to await the Promise<boolean> result before
reporting targets as opened.
| import { tmpdir } from "os" | ||
| import { join } from "path" | ||
| import { unlinkSync, readFileSync } from "fs" | ||
| import { unlinkSync, readFileSync, writeFileSync } from "fs" |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Repository files matching speech.ts:"
fd 'speech\.ts$' . || true
echo
echo "Relevant imports/usages:"
rg -n "writeFileSync|readFileSync|unlinkSync|Bun\.write|Bun\.file|tmpFile|speech" apps/supercode-cli/server/src/voice/speech.ts || true
echo
echo "File outline/size:"
wc -l apps/supercode-cli/server/src/voice/speech.ts || true
ast-grep outline apps/supercode-cli/server/src/voice/speech.ts || true
echo
echo "Context around imports and lines 352-354:"
sed -n '1,40p;320,370p' apps/supercode-cli/server/src/voice/speech.ts | cat -nRepository: yashdev9274/supercli
Length of output: 7779
Use Bun-native asynchronous file output.
writeFileSync blocks the TTS path while writing generated audio. Replace it with await Bun.write(tmpFile, bytes) and remove writeFileSync from the fs import.
Proposed fix
-import { unlinkSync, readFileSync, writeFileSync } from "fs"
+import { unlinkSync, readFileSync } from "fs"
...
- writeFileSync(tmpFile, bytes)
+ await Bun.write(tmpFile, bytes)🧰 Tools
🪛 ast-grep (0.45.0)
[warning] Importing child_process exposes a command-execution surface; ensure any command/argument built from input is validated, and prefer execFile/spawn with an argument array over exec.
Context: import { spawnSync, spawn, type ChildProcess } from "child_process"
Note: [CWE-78] Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection').
(detect-child-process-typescript)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@apps/supercode-cli/server/src/voice/speech.ts` at line 4, Replace the
synchronous writeFileSync usage in the speech generation flow with awaited
Bun.write(tmpFile, bytes), and remove writeFileSync from the fs import while
preserving the existing temporary-file path and audio bytes.
Source: Coding guidelines
| /* smallest.ai provider (Pulse STT) */ | ||
| const SMALLEST_URL = "https://api.smallest.ai/waves/v1/stt/" | ||
| const SMALLEST_MODEL = process.env.SMALLEST_MODEL || "pulse-pro" | ||
| const SMALLEST_LANGUAGE = process.env.SMALLEST_LANGUAGE || STT_LANGUAGE || "en" |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== locate speech.ts =="
fd -a 'speech\.ts$' . | sed 's#^\./##'
echo "== outline =="
ast-grep outline apps/supercode-cli/server/src/voice/speech.ts --view expanded || true
echo "== relevant sections =="
sed -n '1,80p' apps/supercode-cli/server/src/voice/speech.ts
echo "---"
sed -n '170,225p' apps/supercode-cli/server/src/voice/speech.ts
echo "== usages of SMALLEST_MODEL/SMALLEST_LANGUAGE =="
rg -n "SMALSET_MODEL|SMALLEST_MODEL|SMALLEST_LANGUAGE|pulse-pro|pulse" apps/supercode-cli/server/src/voice/speech.ts apps/supercode-cli/server -g '!node_modules' -g '!dist' -g '!build' || true
echo "== deterministic probe of request payload construction around transcribeSmallest =="
python3 - <<'PY'
from pathlib import Path
p = Path('apps/supercode-cli/server/src/voice/speech.ts')
s = p.read_text()
for needle in ['SMALLEST_URL', 'SMALLEST_MODEL', 'SMALLEST_LANGUAGE', 'pulse-pro', 'pulse']:
print(f"{needle}: count={s.count(needle)}")
print("contains language in smallest config:", 'language' in s or '"language"' in s or "'language'" in s)
PYRepository: yashdev9274/supercli
Length of output: 8033
🌐 Web query:
Smallest.ai Waves STT Pulse Pro language=en only
💡 Result:
Pulse Pro is a specialized Speech-to-Text (STT) model within the Smallest.ai Waves platform that is exclusively designed for English-only transcription [1][2]. It is optimized for the highest possible transcription accuracy on pre-recorded audio files [2]. Key distinctions for Pulse Pro include: - Language Limitation: Unlike the standard Pulse model, which supports multilingual transcription (38+ languages) and automatic language detection, Pulse Pro is restricted to English [1][2]. - Use Case: It is intended for pre-recorded audio where maximum accuracy is the primary requirement [2]. - Integration: While the standard Pulse model is used for both real-time streaming and batch processing, Pulse Pro is explicitly positioned for pre-recorded media [1][2]. Users requiring multilingual support or real-time streaming capabilities should use the standard Pulse model instead [1].
Citations:
- 1: https://docs.smallest.ai/models/model-cards/speech-to-text/pulse-pro
- 2: https://smallest.ai/speech-to-text
🌐 Web query:
Smallest.ai Waves documentation speech-to-text-pulse quickstart pulse-pro language
💡 Result:
Smallest.ai offers two primary speech-to-text models under the Pulse family: Pulse and Pulse Pro [1]. Both share a unified endpoint for integration [1][2]. Quickstart Overview To get started with Pulse speech-to-text, you must first obtain an API key from the Smallest AI console [3]. For pre-recorded audio, you send an HTTPS POST request to: https://api.smallest.ai/waves/v1/pulse/get_text [2] For real-time streaming, you connect to the following WebSocket endpoint: wss://api.smallest.ai/waves/v1/pulse/get_text [2] Language Support Language support differs between the two models: Pulse (Standard): Supports multilingual transcription, covering 30+ languages [4][5]. It includes support for streaming and pre-recorded audio, along with features like automatic language detection (by using language=multi) and speaker diarization [2][6][7]. Pulse Pro: Designed for the highest transcription accuracy on pre-recorded audio [8][4]. It is currently restricted to English-only transcription [8][4]. Implementation details for the language parameter involve passing ISO 639-1 language codes (e.g., 'en' for English, 'hi' for Hindi) via query parameters [2]. For scenarios requiring automatic language switching, the 'multi' code is used [2].
Citations:
- 1: https://docs.smallest.ai/models/documentation/speech-to-text-pulse/overview
- 2: https://smallest.ai/blog/speech-to-text-api-integration-guide-for-python-node-and-streaming
- 3: https://docs.smallest.ai/models/documentation/speech-to-text-pulse/quickstart
- 4: https://smallest.ai/speech-to-text
- 5: https://docs.smallest.ai/models/model-cards/speech-to-text/pulse
- 6: https://docs.smallest.ai/waves/v-4-0-0/documentation/speech-to-text-pulse/realtime-web-socket/features
- 7: https://smallest.ai/speech-to-text/audio-to-text
- 8: https://docs.smallest.ai/models/model-cards/speech-to-text/pulse-pro
Make the default Smallest.ai language match pulse-pro.
pulse-pro is English-only, but SMALLEST_LANGUAGE can change independently. A non-English language with the default model sends model=pulse-pro&language=xx to Smallest.ai and fails every direct transcription request. Pair the model with the language, reject unsupported combinations before capture starts, or keep SMALLEST_LANGUAGE fixed when using the default model.
🧰 Tools
🪛 ast-grep (0.45.0)
[warning] Importing child_process exposes a command-execution surface; ensure any command/argument built from input is validated, and prefer execFile/spawn with an argument array over exec.
Context: import { spawnSync, spawn, type ChildProcess } from "child_process"
Note: [CWE-78] Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection').
(detect-child-process-typescript)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@apps/supercode-cli/server/src/voice/speech.ts` around lines 20 - 23, Update
the Smallest.ai configuration around SMALLEST_MODEL and SMALLEST_LANGUAGE so the
default pulse-pro model always uses English. Prevent unsupported non-English
language/model combinations before audio capture or direct transcription begins,
while preserving configurable languages for models that support them.
1 participant
Description
Type of change
How Has This Been Tested?
Please describe the tests that you ran to verify your changes.
bun testpassesbun run typecheckpassesbun run lintpasses (if applicable)Checklist:
Summary by CodeRabbit
/jarvissupport, and configurable startup targets for apps and websites.