Real-time AI-powered sound discovery and mixing on your phone
SSYNC listens to the world through your phone's microphone, identifies environmental sounds using spectral AI analysis, and lets you control the volume of each discovered sound category in real-time — like a DJ mixer for reality.
- No hardcoded categories — the AI discovers sounds dynamically
- Real-time spectral analysis identifies: Speech, Singing, Music, Traffic, Bird Song, Wind, Engine Rumble, and more
- New sliders automatically appear when sounds are detected
- Sliders fade out after 8 seconds of silence
- Individual volume control for each discovered sound
- Touch-to-set horizontal sliders with per-category gain
- Glassmorphism dark UI with neon color coding:
- 🎙️ Cyan — Voice (Speech, Talking, Whispering)
- 🎵 Purple — Music (Keyboard, Singing)
- 🏙️ Orange — Urban (Traffic, Engine Rumble, Impact)
- 🌿 Green — Nature (Bird Song, Wind)
- Oboe audio engine with
SharingMode::Exclusive+PerformanceMode::LowLatency - 48kHz phone mic for maximum AI accuracy
- C++ DSP pipeline — zero-allocation, lock-free audio processing
- ~64ms classification interval with 300ms consensus voting
- Source Separation (4-stem TFLite model) — separates Voice, Music, Traffic, Nature
- Spectral Classifier — ZCR, spectral centroid, high-frequency ratio analysis
- Consensus Voting — 4-frame window, 50% threshold to prevent false detections
- 5x Digital Pre-Amp for enhanced microphone sensitivity
┌─────────────┐ ┌──────────────────────────────────────┐
│ Phone Mic │────▶│ Oboe (48kHz, Exclusive, LowLatency) │
│ (Built-in) │ └──────────────┬───────────────────────┘
└─────────────┘ │
▼
┌──────────────────────────┐
│ AudioEngine (C++) │
│ │
│ ┌─ 5x Pre-Amp ──────┐ │
│ │ processBlock() │ │
│ └──────┬────────────┘ │
│ │ │
│ ┌──────▼────────────┐ │
│ │ Source Separation │ │
│ │ (TFLite 4-stem) │ │
│ └──────┬────────────┘ │
│ │ │
│ ┌──────▼────────────┐ │
│ │ Spectral Features │ │
│ │ ZCR + Centroid │ │
│ └──────┬────────────┘ │
│ │ │
│ ┌──────▼────────────┐ │
│ │ Heuristic Classify │ │
│ │ + Voting Window │ │
│ └──────┬────────────┘ │
│ │ │
│ ┌──────▼────────────┐ │
│ │ Discovery Map │ │
│ │ (mutex-guarded) │ │
│ └───────────────────┘ │
└────────────┬──────────────┘
│ JNI Bridge
┌────────────▼──────────────┐
│ AudioModule (Java) │
│ React Native Bridge │
└────────────┬──────────────┘
│ JSON
┌────────────▼──────────────┐
│ App.tsx (React Native) │
│ Dynamic FlatList UI │
│ Glassmorphism + Neon │
└───────────────────────────┘
SSYNC/
├── App.tsx # React Native UI (Discovery Mode)
├── index.js # Entry point
├── package.json
├── android/
│ ├── app/
│ │ ├── build.gradle
│ │ └── src/main/
│ │ ├── AndroidManifest.xml
│ │ ├── assets/
│ │ │ └── auramix_model.tflite # ML model
│ │ ├── java/com/auramixapp/
│ │ │ ├── AudioModule.java # JNI ↔ React bridge
│ │ │ ├── AudioPackage.java # RN package registration
│ │ │ ├── MainActivity.kt
│ │ │ └── MainApplication.kt
│ │ ├── cpp/ # C++ Audio Engine
│ │ │ ├── AudioEngine.h # Engine header
│ │ │ ├── AudioEngine.cpp # DSP + AI classifier
│ │ │ ├── AudioBridge.cpp # JNI bridge
│ │ │ ├── OboeAudioStream.h # Oboe header
│ │ │ └── OboeAudioStream.cpp # Low-latency audio I/O
│ │ └── jni/
│ │ ├── CMakeLists.txt # Build config
│ │ └── OnLoad.cpp
│ └── gradle.properties
└── ml/
└── train_and_export.py # TFLite model training
- Node.js 18+
- Java JDK 17
- Android SDK (API 33+)
- Android NDK (for C++ compilation)
- Python 3.10+ (for model training only)
- An Android phone with USB debugging enabled
# 1. Clone the repository
git clone https://github.com/EmirhanEtem/SSYNC.git
cd SSYNC
# 2. Install Node dependencies
npm install
# 3. Build and run on device
npx react-native run-android# Bundle JavaScript into the APK
npx react-native bundle \
--platform android --dev false \
--entry-file index.js \
--bundle-output android/app/src/main/assets/index.android.bundle \
--assets-dest android/app/src/main/res/
# Build release APK
cd android && ./gradlew assembleRelease
# Output: android/app/build/outputs/apk/release/app-release.apkcd ml
pip install tensorflow numpy
python train_and_export.py
# Output: auramix_model.tflite → copy to android/app/src/main/assets/| Constant | Default | Description |
|---|---|---|
kSampleRate |
48000 | Microphone sample rate (Hz) |
kInputPreAmp |
5.0 | Digital input gain multiplier |
kClassifyEveryN |
8 | Classify every N audio callbacks |
kVotingWindowSize |
4 | Frames for consensus voting |
kConsensusThreshold |
0.50 | Min vote ratio to confirm detection |
kConfidenceThreshold |
0.35 | Min confidence per detection |
kCooldownMs |
8000 | Time before removing inactive slider |
# View real-time AI detections in Logcat
adb logcat -s AuraMixAI
# Example output:
# AuraMixAI: Detected [Speech: 0.90, Voice: 0.70] | inputDb=-15.2 zcr=0.230
# AuraMixAI: NEW DISCOVERY: [Speech] category=voice confidence=100% votes=4/4| Layer | Technology |
|---|---|
| Frontend | React Native 0.84 (New Architecture) |
| Bridge | JNI (Java ↔ C++) + NativeModules (JS ↔ Java) |
| Audio I/O | Google Oboe (AAudio/OpenSL ES) |
| DSP Engine | Custom C++17, lock-free processing |
| ML Inference | TensorFlow Lite C API |
| Classification | Spectral heuristics (ZCR, Centroid, HFR) |
| Build System | Gradle + CMake + Metro |
Contributions are welcome! Please feel free to submit a Pull Request.
- Fork the repository
- Create your feature branch (
git checkout -b feature/amazing-feature) - Commit your changes (
git commit -m 'Add amazing feature') - Push to the branch (
git push origin feature/amazing-feature) - Open a Pull Request
This project is licensed under the MIT License — see the LICENSE file for details.
Emirhan Etem
- GitHub: @EmirhanEtem
Built with ❤️ and C++ for real-time audio