CL-Streamer

Overview

CL-Streamer is a standalone Common Lisp audio streaming library. It provides HTTP audio streaming with ICY metadata, multi-format encoding (MP3 and AAC), and a real-time audio pipeline via Harmony and cl-mixed — all in a single Lisp process.

Use it to build internet radio stations, live streaming applications, or any system that needs to serve encoded audio over HTTP with metadata.

Key Features

• HTTP streaming server with ICY metadata protocol (iolib sockets)
• MP3 encoding via LAME FFI
• AAC encoding via FDK-AAC (with C shim for SBCL signal compatibility)
• Harmony audio backend — decode FLAC/MP3/OGG, crossfade, mix, queue
• CLOS protocol layer — generic functions for server, pipeline, encoder
• Declarative pipeline DSL — make-pipeline creates server, mounts, encoders, and wiring from a single spec
• Hook system — pipeline-add-hook for event callbacks (track change, playlist change)
• Broadcast ring buffer — single-producer, multi-consumer with burst-on-connect for fast playback start
• Socket options — SO_KEEPALIVE, TCP_NODELAY, SO_SNDTIMEO for robust client handling

Architecture

┌──────────────────────────────────────────────────────────┐
│                    Lisp Process                          │
│                                                          │
│  ┌─────────┐    ┌──────────────┐    ┌─────────────────┐ │
│  │ Harmony  │───▶│ streaming-   │───▶│ MP3 Encoder     │ │
│  │ (decode, │    │ drain        │    │ (LAME FFI)      │─┼──▶ /stream.mp3
│  │  mix,    │    │ (float→s16   │    └─────────────────┘ │
│  │  effects)│    │  conversion) │    ┌─────────────────┐ │
│  └─────────┘    └──────────────┘───▶│ AAC Encoder     │ │
│       ▲                              │ (FDK-AAC shim)  │─┼──▶ /stream.aac
│       │                              └─────────────────┘ │
│  ┌─────────┐                         ┌─────────────────┐ │
│  │ Pipeline │    ICY metadata ──────▶│ HTTP Server     │ │
│  │ (queue,  │    Listener stats ◀────│ (iolib)         │ │
│  │  hooks)  │                         └─────────────────┘ │
│  └─────────┘                                              │
└──────────────────────────────────────────────────────────┘

Harmony Integration

Harmony is Shinmera’s Common Lisp audio framework built on top of cl-mixed. It handles audio decoding (FLAC, MP3, OGG, etc.), sample rate conversion, mixing, and effects processing.

CL-Streamer connects to Harmony by replacing the default audio output drain with a custom streaming-drain that intercepts the mixed audio data. Instead of sending PCM to a sound card, we:

1. Read interleaved IEEE 754 single-float samples from Harmony’s pack buffer
2. Convert to signed 16-bit PCM
3. Feed to all registered encoders (MP3 via LAME, AAC via FDK-AAC)
4. Write encoded bytes to per-mount broadcast ring buffers
5. HTTP clients read from these buffers with burst-on-connect

Both voices (e.g., during crossfade) play through the same Harmony mixer and are automatically summed before reaching the drain — the encoders see a single mixed signal.

The FDK-AAC C Shim

SBCL’s signal handlers conflict with FDK-AAC’s internal memory access patterns. When FDK-AAC touches certain memory during aacEncOpen or aacEncEncode, SBCL’s SIGSEGV handler intercepts it before FDK-AAC’s own handler can run, causing a recursive signal fault.

The solution is a thin C shim (fdkaac-shim.c, compiled to libfdkaac-shim.so) that wraps all FDK-AAC calls:

• fdkaac_open_and_init — opens the encoder, sets all parameters (AOT, sample rate, channels, bitrate, transport format, afterburner), and runs the initialisation call, all from C
• fdkaac_encode — sets up the buffer descriptors (AACENC_BufDesc) and calls aacEncEncode, returning encoded ADTS frames
• fdkaac_close — closes the encoder handle

The Lisp side calls these via CFFI and never touches FDK-AAC directly. This avoids the signal handler conflict entirely.

Note: one subtle bug was that FDK-AAC’s OUT_BITSTREAM_DATA constant is 3, not 1. Getting this wrong causes aacEncEncode to return error 96 with no useful error message. The fix was to use the proper enum constants from aacenc_lib.h rather than hardcoded integers.

Additionally, the AAC encoder uses a PCM accumulation buffer to feed FDK-AAC exact frameLength-sized chunks (1024 frames for AAC-LC). Feeding arbitrary chunk sizes from Harmony’s real-time callback produces audio artefacts.

To rebuild the shim:

gcc -shared -fPIC -o libfdkaac-shim.so fdkaac-shim.c -lfdk-aac

Broadcast Buffer

Each mount point has a ring buffer (broadcast-buffer) that acts as a single-producer, multi-consumer queue. The encoder writes encoded audio in, and each connected client reads from its own position.

• Never blocks the producer — slow clients lose data rather than stalling the encoder
• Burst-on-connect — new clients receive ~4 seconds of recent audio immediately for fast playback start
• Condition variable signalling for efficient client wakeup

ICY Metadata Protocol

The server implements the SHOUTcast/Icecast ICY metadata protocol:

• Responds to Icy-MetaData: 1 requests with metadata-interleaved streams
• Injects metadata blocks at the configured metaint byte interval
• set-now-playing updates the metadata for a mount, picked up by all connected clients on their next metadata interval

Protocol Layer

CL-Streamer defines a CLOS protocol with generic functions that decouple the API from the implementation. You can specialize these on your own classes to provide alternative backends.

Pipeline Protocol

Generic	Description
pipeline-start	Start the audio pipeline
pipeline-stop	Stop (cleans up encoders and server)
pipeline-running-p	Is the pipeline running?
pipeline-play-file	Play a single audio file
pipeline-play-list	Play a list of files with crossfade
pipeline-skip	Skip current track
pipeline-queue-files	Queue files for later playback
pipeline-get-queue	Get the current queue
pipeline-clear-queue	Clear the queue
pipeline-current-track	Get current track info
pipeline-listener-count	Get connected listener count
pipeline-update-metadata	Update ICY metadata on all mounts
pipeline-add-hook	Register an event callback
pipeline-remove-hook	Remove an event callback

Encoder Protocol

Generic	Description
encoder-encode	Encode PCM samples to compressed audio
encoder-flush	Flush any buffered data
encoder-close	Close the encoder and release resources

Hook Events

Event	Arguments	Fired when
:track-change	pipeline track-info	A new track starts playing
:playlist-change	pipeline path	The playlist is switched

Current Status

Working and in production at Asteroid Radio:

• [X] HTTP streaming server with multiple mount points (iolib sockets)
• [X] MP3 encoding via LAME (128kbps, configurable)
• [X] AAC encoding via FDK-AAC with C shim (128kbps ADTS, configurable)
• [X] Harmony audio backend with custom streaming drain
• [X] Real-time float→s16 PCM conversion and dual-encoder output
• [X] ICY metadata protocol (set-now-playing on track change)
• [X] Broadcast ring buffer with burst-on-connect
• [X] Sequential playlist playback with reliable track-end detection
• [X] Crossfade between tracks (configurable overlap and fade durations)
• [X] Multi-format simultaneous output (MP3 + AAC from same source)
• [X] CLOS protocol layer with generic functions
• [X] Declarative make-pipeline DSL
• [X] Hook system for event callbacks
• [X] File metadata reading via cl-taglib (artist, title, album)
• [X] SO_KEEPALIVE, TCP_NODELAY, write timeouts for robust connections
• [X] Queue system with file queueing and playlist switching
• [X] Live DJ input via Harmony mixer

Future work:

• [ ] Stream quality variants (low bitrate, multiple quality levels)
• [ ] Robustness: auto-restart on encoder errors, watchdog
• [ ] Flush AAC accumulation buffer at track boundaries

File Structure

File	Purpose
cl-streamer.asd	ASDF system definitions (core, harmony, encoder, aac)
package.lisp	Package and exports
protocol.lisp	CLOS protocol: generic functions for server, pipeline, encoder
cl-streamer.lisp	Top-level API (start, stop, add-mount, etc.)
stream-server.lisp	HTTP server (iolib), client connections, ICY responses
buffer.lisp	Broadcast ring buffer (single-producer, multi-consumer)
icy-protocol.lisp	ICY metadata encoding and injection
conditions.lisp	Error condition types
harmony-backend.lisp	Harmony integration: pipeline DSL, streaming-drain, crossfade, playlist, hooks
encoder.lisp	MP3 encoder (LAME wrapper)
lame-ffi.lisp	CFFI bindings for libmp3lame
aac-encoder.lisp	AAC encoder with frame accumulation buffer
fdkaac-ffi.lisp	CFFI bindings for FDK-AAC (via shim)
fdkaac-shim.c	C shim for FDK-AAC (avoids SBCL signal conflicts)
libfdkaac-shim.so	Compiled shim shared library
test-stream.lisp	End-to-end test: playlist with MP3 + AAC output

Dependencies

Lisp Libraries

• harmony — audio framework (decode, mix, effects)
• cl-mixed — low-level audio mixing
• cl-mixed-flac — FLAC decoding
• cl-mixed-mpg123 — MP3 decoding
• cffi — C foreign function interface
• iolib — I/O library (sockets with SO_KEEPALIVE, TCP_NODELAY, write timeouts)
• flexi-streams — flexible stream types
• log4cl — logging
• alexandria — utility library
• bordeaux-threads — portable threading
• taglib — audio file metadata reading (optional, for harmony backend)

C Libraries

• libmp3lame — MP3 encoding
• libfdk-aac — AAC encoding (via C shim)
• libfixposix — required by iolib

Quick Start

;; Load the systems
(ql:quickload '(:cl-streamer :cl-streamer/encoder
                :cl-streamer/aac-encoder :cl-streamer/harmony))

;; Create a complete pipeline from a declarative spec.
;; This creates the HTTP server, mount points, encoders, and wiring.
(defvar *pipeline*
  (cl-streamer/harmony:make-pipeline
   :port 8000
   :outputs '((:format :mp3 :mount "/radio.mp3" :bitrate 128
               :name "My Radio MP3")
              (:format :aac :mount "/radio.aac" :bitrate 128
               :name "My Radio AAC"))))

;; Register hooks for events
(cl-streamer:pipeline-add-hook *pipeline* :track-change
  (lambda (pipeline track-info)
    (format t "Now playing: ~A~%" (getf track-info :title))))

(cl-streamer:pipeline-add-hook *pipeline* :playlist-change
  (lambda (pipeline playlist-path)
    (format t "Playlist changed to: ~A~%" playlist-path)))

;; Start the audio pipeline
(cl-streamer:pipeline-start *pipeline*)

;; Play a playlist with crossfade
(cl-streamer:pipeline-play-list *pipeline*
  '("/path/to/track1.flac" "/path/to/track2.flac")
  :crossfade-duration 3.0
  :fade-in 2.0
  :fade-out 2.0)

;; Or play individual files
(cl-streamer:pipeline-play-file *pipeline* "/path/to/track.flac")

;; Queue tracks for later
(cl-streamer:pipeline-queue-files *pipeline*
  '("/path/to/next1.flac" "/path/to/next2.flac"))

;; Skip current track
(cl-streamer:pipeline-skip *pipeline*)

;; Check listeners
(cl-streamer:pipeline-listener-count *pipeline*)

;; Stop everything (pipeline cleans up encoders and server automatically)
(cl-streamer:pipeline-stop *pipeline*)

License

AGPL-3.0