AudioVisual VJ Lite

Desktop-friendly live audio-reactive visualizer built with TypeScript, Vite, React, Web Audio API, and Three.js.

Tech Notes

• Rendering path: Three.js + custom ShaderMaterial scenes.
• Audio analysis: Web Audio AnalyserNode (fftSize=2048) + rolling beat detector (energy + spectral flux blend).
• Input layer: mouse position, velocity, click/drag states, double-click lock.

Run

npm install
npm run dev
npm run build

Live Operation

1. Open app in browser (npm run dev).
2. In Audio section, choose source:

• Microphone
• Device Input + specific input selector
• Sine + Noise Test
• Load Audio File (looped playback)

3. Click Start Audio (required by browser autoplay policy).
4. Switch scenes with 1/2/3 and tune intensity/sensitivity.
5. Use F for fullscreen and H to hide/show UI.

Audio Device Selection

• Device Input mode lists audio interfaces from navigator.mediaDevices.enumerateDevices().
• If a device disconnects, the app surfaces an error and allows re-selection/restart.

Scene Presets

1. Fluid Warp

• Audio drives warp flow and color movement.
• Cursor controls local swirl center + distortion force.

2. Particle Field

• Bass drives velocity and spread.
• Beat triggers bursts.
• Cursor acts as magnet/repulsor (stronger while mouse button is held).

3. Tunnel / Kaleidoscope

• Beat creates pulse/zoom kick.
• Cursor steers center/rotation axis.

Scene switches crossfade for smooth transitions.

Cursor Control

Emits:

• x, y normalized
• vx, vy smoothed velocity
• speed
• down
• dragDistance
• double-click toggles cursor influence lock

UI control: Cursor Influence = Off / Subtle / Strong.

Controls and Shortcuts

• 1 / 2 / 3: switch scenes
• F: fullscreen toggle
• B: blackout toggle
• Space: freeze/unfreeze animation
• H: hide/show UI
• R: reset active scene

Performance Features

• Render scale options: 0.5, 0.75, 1.0.
• If FPS drops below 45 for ~3s, app auto-reduces render scale one step and shows a notice.
• Projection mode hides UI and attempts pointer lock for focused control.

Live Tips

• Use Render Scale 1.0 first, then drop to 0.75 if thermals rise.
• For aggressive environments, use Cursor Influence: Subtle to avoid oversteer.
• Keep Beat Sensitivity around 1.3–1.6 as a starting point.
• Use Blackout for quick stage transitions while audio analysis continues.

Project Structure

src/
  main.tsx
  app/App.tsx
  audio/
    AudioEngine.ts
    AudioFeatures.ts
    BeatDetector.ts
  input/
    CursorController.ts
  visuals/
    VisualEngine.ts
    SceneTypes.ts
    scenes/
      FluidWarpScene.ts
      ParticleFieldScene.ts
      TunnelScene.ts
    shaders/
      fluidWarp.frag
      tunnel.frag
      common.vert
  ui/
    Overlay.tsx
    ControlsPanel.tsx
  utils/
    math.ts
    smoothing.ts
    fps.ts
  styles/
    main.css