An experiment in next-generation streaming audio infrastructure for radio stations.
CoffeeScript Other
Permalink
Failed to load latest commit information.
config Bugfix: apiVersion for ES in analytics module May 22, 2017
js Try to recover gracefully from a bulk insert fail May 30, 2017
src/streammachine Try to recover gracefully from a bulk insert fail May 30, 2017
test Updates package.json to use 13.0.0-beta2 May 25, 2017
util Forgot several references to the 'icecast' module Nov 11, 2015
.gitignore Add example configuration files in config/examples Jun 1, 2013
.npmignore WIP on better AAC support Feb 7, 2014
.travis.yml Travis: Env settings need to be on one line to not create a matrix Jan 2, 2016
ANALYTICS.md First pass at analytics intent May 5, 2014
Gruntfile.coffee Add `streammachine-util` command; Add utils to compiled JS Aug 13, 2015
LICENSE Tweak years in LICENSE file Jan 28, 2016
README.md Admin API has been removed Apr 5, 2016
ROADMAP.md Tweaks to README and ROADMAP Aug 13, 2015
RPC.md Rebuild Master <-> Slave Communications Oct 30, 2014
coffee.js Runner: Add `--coffee` and `--dir` arguments (Addresses #67) Nov 23, 2015
index.js Runner: Add `--coffee` and `--dir` arguments (Addresses #67) Nov 23, 2015
load_test.sh Try to track down a memory leak Apr 30, 2012
package.json Updates package.json to use 13.0.0-beta2 May 25, 2017
runner.coffee Runner: Add `--coffee` and `--dir` arguments (Addresses #67) Nov 23, 2015
streamer.coffee Improve dereferencing during stream deletion (Fixes #76) Jan 15, 2016
streammachine-cmd Add compiled JS codebase; Make coffeescript a dev-only dependency Aug 13, 2015
streammachine-runner-cmd Implement runner script, replacing Lifeguard Aug 13, 2015
streammachine-util-cmd Add `streammachine-util` command; Add utils to compiled JS Aug 13, 2015
util.coffee Add `streammachine-util` command; Add utils to compiled JS Aug 13, 2015

README.md

StreamMachine

StreamMachine is an open-source streaming audio server aimed at pushing innovation for radio stations that have spent too many years running old technology like Shoutcast and Icecast.

The project has two goals: emulating the traditional streaming experience and building support for new features that push the radio listening experience forward. Currently StreamMachine supports traditional Shoutcast-style streaming and HTTP Live Streaming.

StreamMachine is being developed by Eric Richardson (e@ewr.is) and Southern California Public Radio. SCPR has run StreamMachine in production since 2012.

Build Status

The Rewind Buffer

StreamMachine's big idea is keeping a big buffer of audio data in memory, allowing clients to connect to "nearly-live" radio. Because audio data is relatively small, a station can easily keep hours of audio in memory. For instance, a 64k MP3 stream will store 8 hours of audio data in just a little over 200MB of memory.

Using the Rewind Buffer, players can add support for concepts like "Play the current program from its start", or "Play the 9am broadcast." Unlike podcasts, these functions are available immediately and keep the user connected to the station's live stream.

You can read more about the implementation of the RewindBuffer in the wiki.

Analytics

StreamMachine includes support for writing analytics into Elasticsearch. Output includes listening events—chunks of listening that reflect a segment of duration delivered—and rolled-up sessions.

Unlike traditional streaming log outputs, StreamMachine's listening events allow deep realtime analysis of user beahvior, including playhead position and in-flight session durations.

Source Transcoding

With the new Source Mounts support in 0.7, StreamMachine now supports taking one input source and transcoding it into the variety of output formats you may want to offer your listeners.

Architecture

StreamMachine is a Node.js application. It can run on one server, or in a master-slave configuration for load-balancing. StreamMachine is designed for Node 0.10 and higher.

Configuration

Static configuration can be done via a JSON configuration file. Configuration changes will not be written back to the file.

For a dynamic configuration, StreamMachine can store its stream information on a Redis server.

Incoming Audio

Two modes are supported for incoming audio:

  • Icecast Source: The SourceIn module emulates the Icecast server's source handling, allowing source clients to connect to a mount point and provide a source password.

  • Proxy: The ProxyRoom module can connect to an existing Icecast server and proxy its stream through the StreamMachine infrastructure.

Supported Formats

Currently, MP3 and AAC streams are supported.

Operation Modes

StreamMachine can operate as a single process or in a master-slave configuration.

  • Standalone: One process manages both incoming and outgoing audio. For development and small installations.

  • Master: Master handles source connections and configuration. Provides the admin UI and centralizes logging from slave processes. Handles no client traffic.

  • Slave: Slave handles all client requests. Stream audio is proxied from the master over a single socket connection.

For more, see the Mode documentation in the wiki.

Outputs

  • Raw Audio: Realtime audio stream output appropriate for HTML5 audio tags and other web players

  • Shoutcast: Realtime audio stream that injects Shoutcast-style metadata at intervals

  • HTTP Live Streaming: Apple's HTTP Live Streaming protocol uses playlists and audio segments to replace long-lived connections. StreamMachine supports segment chunking and can produce HLS playlists for individual streams and adaptive stream groups.

  • Pumper: Allows chunks of audio to be downloaded as quickly as possible

Alerts

Alerts are intended to provide a heads-up when something's going wrong in the streaming process. A similar alert will fire when the condition is ended. Where logging is intended to signal an event, alerts are about signalling that a condition exists.

Alerts can be sent via email or Pagerduty.

Configuration

StreamMachine uses nconf to load configuration settings from a JSON file, from environment variables and from the command line.

For more, see the full documentation of configuration options.

Handoff Restarts

Traditional streaming is done using long-lived connections that can make life painful when you need to deploy a new code release. StreamMachine is able to accomplish live restarts by doing a "handoff" where the old process passes all clients to the new process before shutting down.

To manage this process, StreamMachine ships with the "streammachine-runner" command. The runner is a lightweight process that will manage the StreamMachine process, initiating a handoff based on either a SIGHUP or a change to a watched trigger file.

Getting Started

StreamMachine isn't the most user-friendly piece of software to install at the moment, but there are two options for quickly getting something running that you can play with:

Running Locally (Repo)

To run StreamMachine locally with no service dependencies, try the included config/standalone.json. To do so:

  • Install Node.js
  • Download the StreamMachine repository
  • Run npm install to install the Node modules that we depend on
  • Run ./streammachine-cmd --config ./config/standalone.json to start the StreamMachine service

The included configuration has an MP3 stream configured at /test, with the source password "testing". The source input listener is on port 8002, and that is where the broadcast should be pointed. To connect and stream audio, you can use a variety of input stream providers:

Streammachine Test Utility
./streammachine-util-cmd source --port 8002 --stream test \
--password testing ./test/files/mp3/mp3-44100-128-s.mp3
VLC or Liquidsoap

To stream via VLC or Liquidsoap for supporting playlists and multiple mp3 files, please refer to the documentation.

Listening

To listen, point an audio player to http://127.0.0.1:8001/test.

Vagrant

You can also use the Vagrantfile included in streammachine-cookbook to install a standalone configuration in a virtual machine.