Skip to content

Music

Jump to bottom
Jdaie edited this page Apr 10, 2026 · 1 revision

Music Playback

Whisplay AI Chatbot supports local music playback through built-in LLM tools. After you configure a local music library, you can ask the device to play a song by name or start random playback by voice.

This feature is designed for simple hands-free music playback on the device itself. It does not require a separate music service provider.

What It Does

  • Plays a specific local song by fuzzy matching the spoken title
  • Supports random playback from the local music library
  • Continues with random songs automatically after the requested song finishes
  • Shows the current track title and playback progress on the Whisplay display or Web Display
  • Stops music immediately when you press the button to talk again

Music playback is exposed to the LLM as built-in tools:

  • playMusic: find and play a specific song from the local library
  • playMusicRandom: start random playback from the local library

Configure the Music Library

Add the following settings to your .env file:

## Local Music Tool (LLM function calling)
# Comma-separated local folders for music search. Absolute paths are recommended.
MUSIC_LIBRARY_DIRS=/home/pi/Music

# Optional: supported file extensions for local music search.
MUSIC_FILE_EXTENSIONS=mp3,wav,flac,m4a,aac,ogg

# Optional: fuzzy-match score threshold (0.0-1.0). Higher value means stricter matching.
MUSIC_FUZZY_MIN_SCORE=0.35

# Optional: rescan interval for music library index in seconds.
MUSIC_RESCAN_SECONDS=60

Notes:

  • MUSIC_LIBRARY_DIRS is required. If it is not configured, the music tools are not registered.
  • Multiple folders are supported. Separate them with commas.
  • Hidden files and folders are ignored during indexing.
  • The player scans recursively, so subfolders under your music directory are supported.

Example:

MUSIC_LIBRARY_DIRS=/home/pi/Music,/mnt/usb/music

Supported Audio Formats

By default, the local music index includes these file types:

  • mp3
  • wav
  • flac
  • m4a
  • aac
  • ogg

mp3 files are played through mpg123. Other supported formats are played through sox.

How to Use It

After the music library is configured and the chatbot is running, just ask naturally, for example:

  • "Play Hotel California"
  • "Play some music"
  • "Play a random song"
  • "Play Jay Chou"

The LLM decides when to call the music tools. When a matching song is found:

  1. The chatbot finishes the current spoken response.
  2. It enters a dedicated music state.
  3. The selected track starts playing.
  4. When that track ends, playback continues with random songs until you interrupt it.

To stop music and start a new conversation, press the hardware button. The device stops playback immediately and returns to listening mode.

Display Behavior

While music is playing, Whisplay shows:

  • the music status
  • the current track title
  • a playback progress bar
  • elapsed time and total time when duration metadata is available

The same progress information is also forwarded to the optional Web Display.

Matching Behavior

Song lookup uses fuzzy matching on the file name without the extension.

Examples:

  • Hotel California.mp3 can match "hotel california"
  • Jay_Chou_Blue_And_White_Porcelain.flac can match "jay chou blue and white porcelain"

If the match is too weak, playback will not start and the tool returns a "No matching track found" message.

You can tune the minimum matching threshold with MUSIC_FUZZY_MIN_SCORE:

  • lower values: more tolerant, but may pick the wrong song
  • higher values: stricter, but may reject approximate titles

Audio Coexistence

The music player is designed to coexist with Whisplay's TTS audio path.

  • Music uses the ALSA dmixed device for playback
  • The persistent TTS player is released before local playback starts
  • The persistent TTS player is restored after playback stops

There is also a test script for validating this behavior on device:

bash python/test/test_music_playback.sh

Troubleshooting

Music commands do nothing

Check these first:

  • MUSIC_LIBRARY_DIRS is set in .env
  • the configured directory exists on the device
  • the directory contains supported audio files
  • the chatbot process was restarted after changing .env

At startup, the logs should show a line similar to:

[Music] Indexed <N> track(s)

The wrong song is selected

Raise MUSIC_FUZZY_MIN_SCORE to make matching stricter.

No files are found

Use absolute paths in MUSIC_LIBRARY_DIRS and verify file permissions for the user running the chatbot service.

Playback fails on Raspberry Pi audio

Verify that:

  • mpg123 and sox are installed
  • your ALSA output path is working normally for chatbot audio
  • the sound card is configured correctly for Whisplay

If needed, run the playback coexistence test:

bash python/test/test_music_playback.sh

Current Scope

The current implementation is intentionally simple:

  • local library playback only
  • no playlist management UI
  • no pause, resume, next, or previous voice commands yet
  • no streaming service integration

If you need more advanced controls, implement additional LLM tools or a plugin on top of the current music player.

Clone this wiki locally