JavaScript HTML CSS Other
Switch branches/tags
Nothing to show
Permalink
Failed to load latest commit information.
client add missing infos when WebGL 2 is not supported Jan 1, 2018
documentation
editor
fas @ 023746a add instagram link to front page Jan 1, 2018
fas_relay fix fas_relay granular/subtractive/PM compatibility Dec 31, 2017
fsdb add session name validation to frontpage May 17, 2017
fss add slice synthesis type (wip) Aug 15, 2017
fsws add external editor support May 1, 2017
osc_relay add spectral synthesis option Sep 8, 2017
supercollider add OSC bundle output of frames data Sep 2, 2017
www add instagram link to front page Jan 1, 2018
.gitmodules initial commit Jan 22, 2017
LICENSE Initial commit Jan 14, 2017
README.md add showdown library Dec 22, 2017
common.js initial commit Jan 22, 2017
main.js
package.json initial commit Jan 22, 2017
prod add session name validation to frontpage May 17, 2017
prod_files add session name validation to frontpage May 17, 2017
setup fix origin problems Mar 27, 2017

README.md

Fragment

Fragment - The Collaborative Spectral Synthesizer

Source code repository for the Fragment app. which can be found at : https://www.fsynth.com

Table of Contents

About Fragment

Fragment is a collaborative audiovisual live coding web. environment with pixels based (image-synth) live additive/spectral/granular synthesis and sequencing, the sound synthesis is powered by pixels data generated from live GLSL code

Many videos of most features are available on YouTube

Fragment has only one fragment shader which has the particularity to be shared between the users of an online session, it update and compile on-the-fly as you or other peoples type, some settings are also synchronized between users such as slices and some global settings.

Fragment has many features making it a bliss to produce any kind of sounds associated (or not) with visuals, it is aimed at artists seeking a creative environment with few limitations to experiment with, a programmable noise-of-all-kinds software.

With WebGL 2 capable web browser, audio can be produced independently from the visuals with the synthOutput vec4 uniform, with WebGL 1 you can use the default pixels output (gl_FragColor or fragColor) to produce sounds and visuals, use the monophonic mode if you want independent audio & visuals with WebGL 1.

For any questions, a message board is available here

You can support this project through Patreon

Requirement

  • Recent web browser such as Chrome, Opera, Safari or Firefox (MIDI is not supported by Firefox at the moment)
  • Mid-range GPU, this app. was made and tested with a GeForce GTX 970
  • Mid-range multi-core CPU (a dual core should be ok with the FAS, a beefy CPU is needed if you use more than one output channel)
  • Not necessary but a MIDI device such as a MIDI keyboard is recommended

Note on performances : Fragment has excellent performances with a modern multi-core system and a browser such as Chrome, if you experience crackles or need advanced audio features, it is recommended that you use the sound server and the independent code editor, the things that may cause poor performances is due to the browser reflow (UI side)

Fragment is also able to do real-time distributed sound synthesis with the sound server, it support any number of machines over the wire and multicore support, this feature also need the fas_relay to work (see below)

Features

  • Additive, spectral, granular, subtractive, phase modulation synthesizer powered by WebAudio oscillators (additive only, work best in Chrome), a Wavetable (additive only, slow/worst) OR a C native audio server (fastest/best)
  • Live coding/JIT compilation of shader code
  • Real-time, collaborative app.
  • Distributed sound synthesis, multi-machines/multi-core support (Fragment Audio Server with fas_relay)
  • Stereophonic or monaural
  • Polyphonic
  • Multitimbral
  • MIDI inputs support with Web MIDI compatible web browsers, MPE supported
  • Adjustable audio output channel per slices
  • Real-time frames by frames recording with export as an image or Fragment input (export back into itself, this can be used to build complex brushes for drawing canvas inputs)
  • .wav export (without FAS : additive synthesis only)
  • WebGL 2.0 and GLSL 3.0 support when compatibility is detected
  • RGBA Live visuals with stereophonic sound generation (WebGL 2.0) or monophonic sound generation (WebGL 1)
  • Synthesis data processed in 32-bit precision (WebGL 2.0 & EXT_color_buffer_float extension) or 8-bit precision
  • Slices can be added/deleted anywhere on the canvas without limits, move left or right automatically and have independent pitch offset
  • Feedback via framebuffer (for fx like reverb, delay, spectral distortion etc)
  • OSC in/out support (a SuperCollider port of the additive synthesis engine use this)
  • Shader inputs:
    • Webcam (no audio support)
    • Images
    • Videos with rate & rewind & loop settings (can import the audio track as an image)
    • Audio files (translated to images)
    • Drawing canvas with drawing and compositing operations which use images Fragment input as brushes, Fragment is bundled with 20 high-quality brushes, a pack of 969 high-quality brushes is also available as a separate download
  • Uniform controllers via OSC Open Stage Control is recommended
  • Per-sessions discussion system
  • Global and per sessions settings smart save/load; make use of localStorage
  • No authentifications needed, anonymous (make use of localStorage and is sessions based)

Sound Synthesis

Fragment capture pixels data (1px wide slices) from a WebGL drawing surface at the browser display refresh rate and translate the RGBA pixels value to notes, the notes are then interpreted and played by one or more synthesis method in real-time.

Common to all synthesis methods, the canvas represent frequencies (exponential mapping) on the vertical axis and time on the horizontal axis.

The Fragment audio server is necessary for very fast sound synthesis and other features such as different output channels per slices.

With the web browser, Fragment is limited to additive synthesis and two output channels.

Slices data can be sent via OSC bundles to extend the possibilities of this synthesizer.

Additive Synthesis

Fragment default sound synthesis is additive, with additive synthesis, Fragment become a fully capable spectral synthesizer able to do re-synthesis based on imported videos or audio files, an image-synth where any number of partials can be played, there is no limits except the canvas height and the computing resources available.

With additive synthesis and stereo mode, the Red channel is the left oscillator amplitude and the Green channel is the right oscillator amplitude.

With FAS (Fragment Audio Server), the Blue channel value is used to add bandwidth noise to the oscillator which may enhance the sound by adding a noise component to the oscillator.

In monophonic mode, the alpha value is used for both L&R oscillator amplitude.

Granular Synthesis (FAS only)

FAS load all samples in a "grains" folder and try to guess their pitch to map it on the canvas so that it match the canvas freq. mapping.

Fragment let you manipulate granular synthesis parameters, there is two type of parameters, channels based and dynamic parameters from the fragment shader output.

Channels based parameters :

  • Granular envelope
    • sine
    • hann
    • hamming
    • tukey
    • gaussian
    • confined gaussian
    • trapezoidal
    • blackman
    • blackman harris
    • parzen
    • nutall
    • flattop
    • kaiser
  • Minimum grain length
  • Maximum grain length

Dynamics parameters

  • Red = Left amplitude
  • Green = Right amplitude
  • Blue = Sample index bounded to [0, 1] (cyclic) and grains density when > 2
  • Alpha = Grains start index bounded [0, 1] (cyclic), grains start index random [0, 1] factor when > 1, play the grain backward when negative

Sampler (FAS only)

Granular synthesis with grain start index of 0 and min/max duration of 1/1 can be used to trigger samples as-is like a regular sampler.

PM (FAS only)

Phase modulation (PM) is a mean to generate sounds by modulating the phase of an oscillator (carrier) from another oscillator (modulator), it is very similar to frequency modulation (FM).

PM synthesis in Fragment work by giving an oscillator index (based on image-height) to the note, this oscillator will be used as a modulator.

PM synthesis use a high quality low-pass filter (Moog type).

Dynamics parameters

  • Red = Amplitude value of the LEFT channel
  • Green = Amplitude value of the RIGHT channel
  • Blue = Moog filter cutoff multiplier; the cutoff is set to the fundamental frequency, 1.0 = cutoff at fundamental frequency
  • Alpha = Moog filter resonance [0, 1] & oscillator selection on integral part (0.x, 1.x, 2.x etc), cyclic

Subtractive (FAS only)

Subtractive synthesis start from harmonically rich waveforms which are then filtered.

It is somewhat slow due to additive synthesized waveforms and there is only one high quality low-pass filter (Moog type) implemented.

There is three type of band-limited (no aliasing!) waveforms : sawtooth, square, triangle

This type of synthesis may improve gradually with more waveforms (and a faster way to generate them) and more filters.

Note : The waveforms are constitued of a maximum of 64 partials

Dynamics parameters

  • Red = Amplitude value of the LEFT channel
  • Green = Amplitude value of the RIGHT channel
  • Blue = Moog filter cutoff multiplier; the cutoff is set to the fundamental frequency, 1.0 = cutoff at fundamental frequency
  • Alpha = Moog filter resonance [0, 1] & waveform selection on integral part (0.x, 1.x, 2.x etc)

MIDI

Fragment support MIDI inputs.

Features
  • MIDI keyboard support, list of MIDI parameters accessible in the fragment shader :
    • note-on/note-off messages
    • note frequency
    • velocity / release velocity
    • aftertouch
    • pitch bend
    • CC74
    • MIDI channel
    • elapsed time since the key was pressed
  • Polyphony is automatically detected from the GPU capabilities (704 notes with a GeForce GTX 970 GPU, 16 notes is the minimum, maximum notes depend on the GPU capability/shader complexity)
  • Multidimensional Polyphonic Expression (MPE) support
  • Hot plugging of MIDI devices

If you need to control more parameters, see OSC below.

OSC

Fragment support OSC input and output, an OSC relay which translate WebSockets data to UDP packets should be used for this feature to work.

Fragment uniforms can be defined through OSC with two methods :

  • Message with an address starting with i such as /iarr
    • This will create/update a float array uniform, the message should contain an array with index to update at index 0 and the value at index 1
    • If the array does not exist, it will create it and grow the array as needed
  • Update whole float array with message starting with a address such as /aarr
    • This will create/update a whole **float **array uniform, the message should just contain all the array values

You can send a message to the /clear address to clear all OSC defined uniforms

Open Stage Control can be used to control partials or more parameters through OSC via faders etc.

Tools

Many tools are available to enhance Fragment.

Notes

  • WebAudio oscillators and the WebAudio Wavetable mode can only have two output channels (L/R) due to performances issues (this may change in the future!)

  • One of the main limitation of Fragment may be the events granularity caused by the monitor refresh rate (60 or 120 FPS), this can be solved by running the browser without VSYNC, example for Chrome with the command-line parameter --disable-gpu-vsync

Tips and tricks

  • If you enable the monophonic setting, you have the RGB output for live coding visuals which can be fully synchronized with the synthesized sounds, sounds will be synthesized by using the alpha channel
  • Pressing F11 in the GLSL code editor make the editor fullscreen (as an overlay)
  • You can feed the display content of any apps on your desktop (such as GIMP or Krita) by streaming your desktop as a camera (v4l2loopback and ffmpeg is useful to pull of this on Linux)

Project organization

  • client - main application
  • www - landing page
  • fss - main server (discuss. system, slices)
  • fsdb - sharedb server (collaborative features)
  • fsws - web. server (only used for development or local installation)
  • osc_relay - an OSC relay which use the osc.js library (must be launched to use OSC features)
  • fas_relay - distributed multi-machines/multi-core realtime sound synthesis
  • editor - external GLSL code editor
  • supercollider - the SuperCollider port of the additive synthesis engine (fed through OSC)
  • documentation - MkDocs documentation and previous MAML (Minimalist Anubis Markup Language) documentation with the latest HTML and PDF doc.
  • main.js - Electron app. file
  • common.js - Server config. file

All servers are clustered for scalability and smooth updates.

Build system

Fragment is built with a custom build system scanning for changes in real-time and which include files when it read /*#include file*/, it execute several programs on the output files such as code minifier, the build system was made with the functional Anubis programming language, a programming language based on cartesian closed category theory.

Since the Anubis language is relatively unknown, a simplified (without live check & build) Python port of the build system is available, check out pyNut

If you want to build it by yourself, install pyNut script somewhere in your PATH then call pynutbuild shell script in the client root directory.

_app_fs_ and _app_cm_ are the entry point files used by the build system to produce a single file and a production ready file in the dist directory.

The Anubis build system can be found here and this build system is called by the shell script named nutbuild (root folder)

How to setup your own server

Fragment make use of NodeJS, NPM, MongoDB and Redis database, once those are installed, it is easy to run it locally:

  • clone this repository
  • cd fss & npm install & node fss
  • cd fsdb & npm install & node fsdb
  • cd fsws & npm install & node fsws
  • point your browser to http://127.0.0.1:3000

If you just want to try it out without the collaborative feature and GLSL code save, you don't need MongoDB and Redis, you just need "fsws" then point your browser to http://127.0.0.1:3000

If you want to use it with an OSC app like the SuperCollider fs.sc file or Open Stage Control, please look at the osc_relay directory.

To use the OSC relay :

  • cd osc_relay & npm install & node osc_relay

To use the FAS relay :

  • cd fas_relay & npm install & node fas_relay

Prod. system

  • prod_files contain a list of files and directories that will be copied to the production system
  • prod is a shell script which produce an archive from prod_files list, perform additional cleanup and unarchive over SSH
  • setup is a script which is executed on the server after everything has been uploaded, this configure Fragment for the production system

The future

Maybe a VST/LV2 plugin for accessibility and of course many new features are coming soon. ;)

A native app. will be done soon but with a totally different paradigm, it may be extremely simple technically and very flexible/accessible, it will also fix the main limitation of Fragment by allowing > 60 FPS capture (configurable so not limited to the display refresh rate...) which mean basically unlimited granularity as the hardware get faster.

Stuff used to make this

Client :

Papers :

Servers :

Utilities :

Data :

The repository for the early proof of concept can be found here.

Fragment on social medias

YouTube

Twitter

Instagram

Facebook

SoundCloud

License

Simplified BSD license

Credits

The main inspiration for all of this is Alexander Zolotov Virtual ANS software, thank to him.

Heavily inspired by Shadertoy as well.