Voice Assistant Pipeline

Voice Assistant Pipeline — fully on-device

Chain the Jokobee on-device AI stack to build a private voice assistant: the user speaks, the phone answers out loud, and no audio or text ever leaves the device.

🎙️ mic ─▶ FFmpegKit ─▶ Whisper ─▶ llama-android ─▶ Android TTS ─▶ 🔊 speaker
        (decode/resample) (speech→text)  (text→answer)   (answer→speech)

Stage	Library	Job
Decode audio	FFmpegKit	any format → 16 kHz mono PCM
Speech → text	Whisper	transcribe the PCM
Text → answer	llama-android	run the LLM
Answer → speech	Android TextToSpeech	speak the reply

The glue code

import dev.ffmpegkit.llama.Llama
import dev.ffmpegkit.llama.LlamaConfig
// import your Whisper + FFmpegKit wrappers

class VoiceAssistant(context: Context) {

    private val tts = TextToSpeech(context) { /* onInit */ }
    private lateinit var llm: dev.ffmpegkit.llama.LlamaModel

    suspend fun warmUp(modelPath: String) {
        // Load the LLM once and keep it — loading is the expensive part.
        llm = Llama.loadModel(modelPath, LlamaConfig(contextSize = 2048, threads = 4))
    }

    /** recordedAudio = a file the user just spoke into (any format). */
    suspend fun handle(recordedAudio: File) {
        // 1. Decode + resample to 16 kHz mono PCM (Whisper's expected input).
        val pcm = FFmpegKit.toPcm16kMono(recordedAudio)          // your FFmpegKit call

        // 2. Speech → text.
        val question = Whisper.transcribe(pcm)                   // your Whisper call

        // 3. Text → answer, on-device.
        val answer = Llama.complete(
            llm,
            prompt = question,
            systemPrompt = "You are a helpful voice assistant. Answer in one or two short sentences.",
            maxTokens = 128,
        ).text

        // 4. Answer → speech.
        tts.speak(answer, TextToSpeech.QUEUE_FLUSH, null, "answer")
    }

    fun shutdown() {
        Llama.releaseModel(llm)
        tts.shutdown()
    }
}

Practical tips

• Load the LLM once (warmUp) and reuse it for every turn. Reloading per question is what makes naive assistants feel sluggish.
• Keep replies short (maxTokens = 96–160 + a "one or two sentences" system prompt) so the user hears an answer quickly.
• Pick a small model — a 0.5B–2B model keeps the round-trip snappy on CPU. See Choosing a Model.
• Serialize turns — one LlamaModel is not thread-safe; don't start a new complete until the previous one returns. (Concurrent sessions are a Pro feature — see Free vs Pro.)
• Multi-turn memory: the Free complete is single-turn. To keep conversation history, prepend prior turns into your prompt yourself, or use the Pro streaming / session API for managed KV-cache history.

Why on-device

• Privacy — voice and transcripts never touch a server.
• No latency floor / no network — works on a plane, in a tunnel, offline.
• No per-request cost — no cloud LLM bill, ever.

Back to Home.