Skip to content
BranchesTags

Repository files navigation

ChromaSound

A Real-Time Camera-Driven Music & Painting Instrument

SIGGRAPH 2026 Poster Platform Python License: MIT Stars

A camera-based real-time interaction system that turns hand gestures into music composition, brush painting, and particle effects, with vision–audio closed-loop coordination between Python and Processing over OSC.

Overview · Quick Start · Architecture · Citation · Acknowledgements

✨ Teaser

Place your teaser GIF here: assets/teaser.gif (recommended 1200×800, ≤ 8 MB) Capture the running program with S to start, perform with one hand, press S again to save the WAV + screenshot.

📖 Overview

ChromaSound is a real-time multimodal interactive system. A single webcam captures the user's hand; MediaPipe Hand Landmarker extracts 21 keypoints, and a Python backend synthesizes and plays music in response to the hand's position, speed, and pinch state. OSC streams the same state to a Processing front-end, which renders the canvas, brush strokes, particles, and glow effects — closing the loop between gesture, sound, and vision.
🎨 Painting Index-finger trajectory as brush; three-color cycle (red / blue / green)
🎵 Music Horizontal motion → instrument; vertical motion → chord; speed → volume
✨ Visuals Particles & glow synced to every triggered note / chord
🤏 Pinch Index + thumb = switch color → switch instrument group

Pipeline at a Glance

Webcam ─▶ MediaPipe ─▶ Python backend ──OSC──▶ Processing front-end
                       │  (audio + state)              (canvas + FX)
                       └─▶ pygame / numpy synth ◀─── intro_state ───┘

📰 Updates

  • [2026-07] Refactored main loop: full try / except + structured logging (chroma_boot.log, chroma_crash.log).
  • [2026-07] Hand-tracking stability: switched to RunningMode.VIDEO + monotonic timestamps; tuned detection thresholds to 0.3.
  • [2026-07] EMA smoothing α raised 0.35 → 0.55; velocity-based extrapolation up to 8 frames on missed detections.
  • [2026-06] Intro-state timeout fallback so a crashed Processing side no longer leaves Python muted forever.
  • [2026-06] OSC protocol stabilized: /cursor /mode /color /note /chord (Python → Processing) and /intro_state (Processing → Python).
  • [2026-05] 5 synthesized timbres (piano / pluck / bell / glass / warm_pad) with ADSR, low-pass IIR, Schroeder reverb, soft clipping.
  • [2026-05] First public release.

🔨 Quick Start

Recommended: run proj.py directly. start.bat / start.sh are optional helpers that may fail on non-ASCII usernames or paths with spaces.

1. System Requirements
OS Windows 11 (also works on macOS / Linux with minor tweaks)
Python 3.11+
Hardware Webcam (required), audio output device (required), microphone not required
Optional Processing 4+ for the visual front-end

2. Install Dependencies

git clone https://github.com/ling775/ChromaSound.git
cd ChromaSound
pip install -r requirements.txt

3. Run the Python Backend (Required)

python proj.py

The program will:

  1. Open the webcam (small preview window in the bottom-right of the main canvas)
  2. Start MediaPipe hand tracking
  3. Initialize the pygame audio engine
  4. Start the OSC server on 127.0.0.1:8001 and OSC client to 127.0.0.1:8000
  5. Render the HUD overlay; press S to record, Esc / Q to quit

The Python side runs fine without the Processing front-end — OSC sends will silently drop and the HUD will show OSC:OFF.

If python proj.py ever raises IndentationError, run the auto-fix script once and retry:

python _indent_fix3.py
python proj.py

4. Run the Processing Front-End (Optional, for Visuals)

  • Open ChromaSound/ChromaSound.pde in Processing IDE and press ▶, or
  • Run a pre-built processing/ChromaSound.exe (Windows) / ChromaSound.app (macOS)

⌨️ Keyboard Shortcuts

Key Action
S Start / stop recording (saves WAV + screenshot)
Esc / Q Quit

🏗️ Architecture

System Overview

┌────────────────────┐         OSC (UDP)         ┌────────────────────┐
│   Processing 端    │ ◀──────────────────────▶  │     Python 端     │
│   (visual front)   │                           │   (compute back)  │
├────────────────────┤                           ├────────────────────┤
│ · Intro page       │   client → 8000           │ · Hand tracking   │
│ · Canvas + brush   │   /cursor /mode /color    │ · Audio synthesis │
│ · Particles / glow │   /note /chord            │ · Rhythm & scale  │
│ · Recording view   │                           │ · Webcam preview  │
│                    │   ←  server 8001          │                    │
│                    │     /intro_state          │                    │
└────────────────────┘                           └────────────────────┘
Side Responsibility Entry
Processing All visuals: intro page, canvas, brush, particles, glow ChromaSound.pde
Python All compute: hand tracking, audio synthesis, rhythm, scale proj.py

Module Map (Python side)

File Role
proj.py Main loop, OSC bridge, webcam + HUD, keyboard
audio.py pygame mixer, note / chord playback, WAV recording
music_engine.py Music logic (chord progression, volume curve)
gesture.py Finger-up / pinch detection
motion.py Speed calculation, EMA smoothing
trajectory.py Trajectory analysis (acceleration, direction)
rhythm.py Beat / rhythm control
scale.py Scale & pitch mapping
state.py Shared program state
osc_send.py Standalone OSC test sender

OSC Protocol

Python → Processing (client, port 8000)

Address Args Meaning
/cursor [x, y] Index-finger position in canvas coords (1200×800)
/mode int 0 = IDLE, 1 = DRAW, 2 = PINCH
/color [r, g, b] Current brush color (RGB)
/note [midi, volume] Horizontal-motion melody note + volume
/chord [midi_1, ..., volume] Vertical-motion chord MIDI list + volume

Processing → Python (server, port 8001)

Address Args Meaning
/intro_state int (0 / 1) 1 = on intro page, suppress audio; 0 = resume

The bidirectional channel makes both sides feel like one app: while the Processing intro page is on, Python holds back /note and /chord to avoid stray brush or particles over the intro background.

🎵 Audio System

5 built-in timbres, all synthesized with numpy — no external soundfont required:

Timbre Algorithm Character
piano Additive synthesis (6 harmonics) Multi-harmonic + ADSR
pluck Karplus-Strong Physical model: noise loop with decay
bell Inharmonic partials (2.76 / 5.4 / 8.2 × f₀) Bell-like
glass High partials + short attack Crystal-clear
warm_pad Multi-octave + detune + low-pass Warm pad

Every timbre goes through:

  • ADSR envelope (exponential Attack / Decay / Sustain / Release)
  • 1st-order IIR low-pass (smooth high end)
  • Schroeder-style multi-tap reverb (spatial depth)
  • Peak normalization (PEAK_LIMIT = 0.45) + tanh soft-clip
  • Chord energy normalization (1/√N decay, secondary target 0.7)

→ No clipping, no harshness, no external assets.

🧪 Hand-Tracking Stability (Engineering Notes)

Fast hand motion is the #1 failure mode for single-frame detection. Mitigations in proj.py:

  • RunningMode.VIDEO + monotonic per-frame timestamps — uses temporal context between frames, much more robust than IMAGE mode
  • Lowered thresholds — min_hand_detection_confidence / min_hand_presence_confidence / min_tracking_confidence all set to 0.3
  • EMA α = 0.55 (up from 0.35) — brush follows the hand more closely
  • Velocity-based extrapolation — when a frame misses the hand, predict position from last velocity for up to 8 frames (~ 0.4 s) with 0.7 per-frame decay, then zero out
  • Late clear — only after 8 consecutive misses does smooth_x / y go to None

📁 Project Structure

ChromaSound/
├── proj.py                 # Main entry: tracking + audio + OSC + webcam
├── audio.py                # Playback, note / chord, WAV recording
├── music_engine.py         # Music logic
├── gesture.py              # Finger-up / pinch detection
├── motion.py               # Speed & smoothing
├── trajectory.py           # Trajectory analysis
├── particles.py            # (legacy particles, kept for fallback)
├── rhythm.py               # Rhythm control
├── scale.py                # Scale & pitch
├── state.py                # Shared state
├── osc_send.py             # Standalone OSC test sender
│
├── start.bat               # Windows helper (optional)
├── start.sh                # macOS / Linux helper (optional)
│
├── hand_landmarker.task    # MediaPipe model
├── requirements.txt
├── LICENSE
└── README.md

⚙️ System Compatibility

Tested on Status
Windows 11, Python 3.11.9 ✅ Verified
macOS 14 (M-series) ⚠️ Likely works; webcam index may need adjustment
Linux (Ubuntu 22.04) ⚠️ Likely works; pygame audio backend may need SDL config

Other OSes may need tweaks to camera index and audio device.

🐛 Troubleshooting

Symptom Fix
IndentationError on launch python _indent_fix3.py then retry
Webcam window is black Check no other app is using the camera; try cv2.VideoCapture(0)
No sound Verify system audio output device; pygame needs a default sink
HUD shows OSC:OFF Expected when Processing front-end is not running — not an error
start.bat fails immediately Run python proj.py directly instead
Recording produces no file Confirm write permission in the project root

🖋️ Citation

If you use this work in academic / artistic contexts, please cite:

@inproceedings{ling2026chromasound,
  title     = {ChromaSound: A Real-Time Camera-Driven Music \& Painting Instrument},
  author    = {Ling, ????},
  booktitle = {SIGGRAPH 2026 Posters},
  year      = {2026},
  url       = {https://github.com/ling775/ChromaSound}
}

Replace ???? with your actual name. SIGGRAPH poster format accepts single-author entries.

🤍 Acknowledgements

Inspired by the long lineage of camera-as-instrument artworks, including Myron Krueger's Videoplace and the spirit of the Reactable.

📄 License

This project is released under the MIT License — see LICENSE for details.

Copyright © 2026 ling775

About

Gesture-Driven Synesthesia Canvas with Real-Time Audio Feedback

Resources

Readme
MIT license
Activity

Stars

1 star

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages