Skip to content

About

Open-source on-device multimodal AI SDKs for speech, LLM, and more, fully offline.

Resources

Code of conduct

Contributing

Stars

9 stars

Watchers

0 watching

Forks

Repository files navigation

DeviceAI

On-device AI for Android & iOS — speech recognition, text-to-speech, and LLM chat. Zero cloud latency, zero privacy risk. Optional cloud backend for OTA model updates, telemetry, and device management.

Build License Maven Central Kotlin Swift Android iOS


Install

Android (Kotlin)

// build.gradle.kts
implementation("dev.deviceai:core:0.0.4")
implementation("dev.deviceai:speech:0.0.4")   // STT + TTS
implementation("dev.deviceai:llm:0.0.4")      // LLM + RAG

iOS / macOS (Swift Package Manager)

Add the DeviceAI package to your Xcode project or Package.swift:

// Package.swift
dependencies: [
    .package(url: "https://github.com/deviceai-labs/deviceai", from: "0.0.1")
]

Then add the modules you need:

.target(
    name: "YourApp",
    dependencies: [
        .product(name: "DeviceAI", package: "deviceai"),
        .product(name: "DeviceAISpeech", package: "deviceai"),   // STT + TTS
        .product(name: "DeviceAILLM", package: "deviceai"),      // LLM + RAG
    ]
)

Or in Xcode: File → Add Package Dependencies → paste https://github.com/deviceai-labs/deviceai → select the modules you need.


Initialize

Android

class MyApp : Application() {
    override fun onCreate() {
        super.onCreate()
        PlatformStorage.initialize(this)
        DeviceAI.initialize(context = this)
    }
}

iOS / macOS

import DeviceAI

// Local mode — no cloud, fully offline
DeviceAI.initialize()

// With cloud backend (optional)
DeviceAI.initialize(apiKey: "<YOUR_API_KEY>") {
    $0.telemetry = .minimal
}

That's it. The SDK runs fully on-device with no backend required.

With cloud backend (optional)

Android:

DeviceAI.initialize(context = this, apiKey = "<YOUR_API_KEY>") {
    telemetry = TelemetryLevel.Minimal
    appVersion = BuildConfig.VERSION_NAME
}

iOS:

DeviceAI.initialize(apiKey: "<YOUR_API_KEY>") {
    $0.telemetry = .minimal
    $0.appVersion = Bundle.main.infoDictionary?["CFBundleShortVersionString"] as? String
}

The API key connects the SDK to the DeviceAI cloud backend. Device hardware (RAM, CPU, SoC) is detected automatically — no manual configuration needed.


Speech-to-Text

Android

SpeechBridge.initStt(modelPath, SttConfig(language = "en", useGpu = true))

// From raw audio samples
val text = SpeechBridge.transcribeAudio(samples)  // FloatArray, 16kHz mono

// From a WAV file
val textFromFile = SpeechBridge.transcribe("/path/to/audio.wav")

SpeechBridge.shutdownStt()

iOS

let engine = try await SttEngine(modelPath: path, config: .init(language: "en"))

// From raw audio samples
let text = try await engine.transcribe(samples: audioBuffer)  // [Float], 16kHz mono

// From a WAV file
let textFromFile = try await engine.transcribe(audioPath: "/path/to/audio.wav")

engine.shutdown()

Powered by whisper.cpp. Runs 7× faster than real-time on mid-range hardware.

Text-to-Speech

Voices are Piper models run by sherpa-onnx. Get them from the SDK catalog — it downloads sherpa's packaged tarball (model + tokens.txt + espeak-ng-data), verifies its SHA-256, and extracts it. Loose .onnx files from elsewhere are missing the metadata the engine needs and will not load.

Android

ModelRegistry.initialize()
val voice = ModelRegistry.getTtsTarballVoices(languageCode = "en").first()   // ~67 MB, once
val local = ModelRegistry.download(voice) { progress -> /* … */ }.getOrThrow()

val ok = SpeechBridge.initTts(
    modelPath  = local.modelPath,
    tokensPath = local.configPath!!,
    config     = TtsConfig(
        speechRate = 1.0f,
        dataDir    = local.ttsDataDir(),      // espeak-ng-data, required for Piper
        voicesPath = local.ttsVoicesPath(),   // "" for Piper; voices.bin for Kokoro
    ),
)
check(ok) { "TTS engine failed to initialise" }

val pcm: ShortArray = SpeechBridge.synthesize("Hello from DeviceAI.")
val sampleRate = SpeechBridge.ttsSampleRate()   // 22050 for Piper — configure AudioTrack with this

SpeechBridge.shutdownTts()

Piper medium voices synthesize at roughly 3× real time on a mid-range phone CPU (RTF ~0.36 on a Raspberry Pi 4).

iOS

let tts = try await TtsEngine(modelPath: path, tokensPath: tokens)
let audio = try await tts.synthesize("Hello from DeviceAI")
tts.shutdown()

// Or use Apple's built-in voices (zero setup, no model download):
let systemTts = SystemTTSEngine()
try await systemTts.speak("Hello from DeviceAI")

Powered by sherpa-onnx. Supports VITS and Kokoro voice models.

LLM Chat

Android

val session = DeviceAI.llm.chat("/path/to/model.gguf") {
    systemPrompt = "You are a helpful assistant."
    maxTokens = 512
    temperature = 0.7f
    useGpu = true
}

// Streaming (recommended for UI)
session.send("What is Kotlin?").collect { token -> print(token) }

// Multi-turn — history managed automatically
session.send("Give me an example.").collect { print(it) }

// Lifecycle
session.cancel()        // abort generation
session.clearHistory()  // fresh conversation
session.close()         // unload model

iOS

let session = try await ChatSession(modelPath: path) {
    $0.systemPrompt = "You are a helpful assistant."
    $0.maxTokens = 512
    $0.temperature = 0.7
}

// Streaming
for try await token in try session.send("What is Swift?") {
    print(token, terminator: "")
}

// Multi-turn — history managed automatically
for try await token in try session.send("Give me an example.") {
    print(token, terminator: "")
}

// Lifecycle
session.cancel()        // abort generation
session.clearHistory()  // fresh conversation
session.close()         // unload model

Powered by llama.cpp. Supports any GGUF model. Vulkan GPU on Android, Metal GPU on iOS.

Offline RAG

Android

val store = BM25RagStore(rawChunks = listOf(
    "DeviceAI supports Android and iOS.",
    "LLM inference uses llama.cpp with Vulkan GPU."
))
val session = DeviceAI.llm.chat("/path/to/model.gguf") { ragStore = store }
session.send("What GPU does DeviceAI use?").collect { print(it) }

iOS

let store = BM25RagStore(chunks: [
    "DeviceAI supports Android and iOS.",
    "LLM inference uses llama.cpp with Metal GPU."
])
let session = try await ChatSession(modelPath: path) {
    $0.ragStore = store
}
for try await token in try session.send("What GPU does DeviceAI use?") {
    print(token, terminator: "")
}

No embedding model needed — BM25 keyword retrieval runs entirely on-device.


Telemetry

When telemetry is enabled, the SDK automatically tracks performance metrics for all modules:

What's collected

Module Metrics
STT Model load time, transcription latency, audio duration (input_length_ms)
TTS Model load time, synthesis latency, text length (output_chars)
LLM Model load time, inference latency, time-to-first-token, tokens/sec, token counts, finish reason

What's NEVER collected

  • Prompt or response text content
  • Audio recordings or transcript content
  • PII by default

Apps should avoid putting PII in appAttributes, since developer-provided attributes are sent in the capability profile.

Telemetry levels

Android:

DeviceAI.initialize(context = this, apiKey = "<YOUR_API_KEY>") {
    telemetry = TelemetryLevel.Off      // default — nothing sent
    telemetry = TelemetryLevel.Minimal  // model load/unload + inference metrics
    telemetry = TelemetryLevel.Full     // includes OTA downloads + manifest syncs
}

iOS:

DeviceAI.initialize(apiKey: "<YOUR_API_KEY>") {
    $0.telemetry = .off      // default — nothing sent
    $0.telemetry = .minimal  // model load/unload + inference metrics
    $0.telemetry = .full     // includes OTA downloads + manifest syncs
}

Events are batched on-device and delivered efficiently — respects Wi-Fi preference, data-saver mode, and flushes automatically when the app goes to background.

Custom telemetry sink

Route events to your own analytics instead of the DeviceAI backend:

Android:

DeviceAI.initialize(context = this, apiKey = "<YOUR_API_KEY>") {
    telemetry = TelemetryLevel.Minimal
    telemetrySink = object : TelemetrySink {
        override suspend fun ingest(events: List<TelemetryEvent>) {
            myAnalytics.track(events)
        }
    }
}

iOS:

DeviceAI.initialize(apiKey: "<YOUR_API_KEY>") {
    $0.telemetry = .minimal
    $0.telemetrySink = MyAnalyticsSink()  // conforms to TelemetrySink protocol
}

Cloud Backend

The SDK optionally connects to a cloud control plane. When an API key is provided:

Feature What happens
Device registration Automatic — hardware profile sent, capability tier assigned
Model manifest Backend assigns the right model for each device tier, synced every 6h
OTA updates Push new models with canary rollouts and instant kill-switch
Telemetry Performance metrics batched and delivered (when enabled)
Device identity Stable across reinstalls — same device always gets the same ID

No cloud calls are made without an API key. Local mode works fully offline.


Models

Whisper (STT)

Model Size Speed Best for
ggml-tiny.en.bin 75 MB 7× real-time English, mobile-first
ggml-base.bin 142 MB Fast Multilingual, balanced
ggml-small.bin 466 MB Medium Higher accuracy

LLM (GGUF via llama.cpp)

Model Size Best for
SmolLM2-360M-Instruct (Q4) ~220 MB Fastest, mobile-first
Qwen2.5-0.5B-Instruct (Q4) ~400 MB Multilingual, compact
Llama-3.2-1B-Instruct (Q4) ~700 MB Strong reasoning
SmolLM2-1.7B-Instruct (Q4) ~1 GB Balanced

Browse LLM models with LlmCatalog. Download Whisper/TTS models via ModelRegistry.


Features

Feature Android iOS
Speech-to-Text (whisper.cpp) ✅ ✅
Text-to-Speech (sherpa-onnx VITS / Kokoro) ✅ ✅
System TTS (Apple AVSpeechSynthesizer) — ✅
Voice Activity Detection ✅ ✅
LLM inference (llama.cpp, GGUF) ✅ ✅
Streaming generation ✅ ✅
Stateful multi-turn chat ✅ ✅
Offline RAG (BM25) ✅ ✅
Auto model download (HuggingFace) ✅ 🗓
GPU acceleration ✅ Vulkan ✅ Metal
Cloud backend (registration, manifest, telemetry) ✅ ✅
Auto hardware detection ✅ ✅
Stable device identity (survives reinstall) ✅ ✅
Telemetry (STT/TTS/LLM) ✅ ✅
Custom telemetry sink ✅ ✅
OTA model rollouts + kill switch ✅ ✅
Flutter plugin 🗓 🗓
React Native module 🗓 🗓

Platform support

Platform STT TTS LLM Version Status
Android (API 26+) ✅ ✅ ✅ 0.0.4 Available
iOS 17+ / macOS 14+ ✅ ✅ ✅ 0.0.1 Available
Flutter — — — — Planned
React Native — — — — Planned

Benchmarks

Device SoC Model Audio Inference RTF
Redmi Note 9 Pro Snapdragon 720G whisper-tiny.en 5.4s 746ms 0.14x

RTF < 1.0 = faster than real-time. 0.14x = ~7× faster than real-time.


Building from source

Android

git clone https://github.com/deviceai-labs/deviceai.git
cd deviceai
make setup
./gradlew :kotlin:core:compileDebugKotlinAndroid
./gradlew :kotlin:speech:compileDebugKotlinAndroid
./gradlew :kotlin:llm:compileDebugKotlinAndroid

iOS (Swift)

git clone https://github.com/deviceai-labs/deviceai.git
cd deviceai

# Build XCFrameworks (requires Xcode + CMake)
./sdk/deviceai-commons/scripts/build-xcframeworks.sh

# Build the Swift package
cd swift
swift build

Sample App

# Android: Open samples/androidApp/ in Android Studio and run on device/emulator
# iOS: Open samples/iosApp/ in Xcode

Contributing

Issues and PRs welcome. Platform SDK contributions (flutter/, react-native/) are especially welcome.


License

Apache 2.0 — see LICENSE.

About

Open-source on-device multimodal AI SDKs for speech, LLM, and more, fully offline.

Resources

Code of conduct

Contributing

Stars

9 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages