TL;DR A musical instrument completely controlled by your face.
Smile to begin - click your face to change to a different instrument.
Or...
An augmented reality, real-time motion-capture, machine-learning powered, holographic, accessible general MIDI controller and synthesizer (that comes without a manual).
learn more about the project at photosynth.lol
An exended-reality accessible musical synthesizer - completely controlled by your face.
By piggybacking onto a person's own natural movements, we reduce the complexity required to learn a new instrument to mere moments whilst retaining full expression and precise control.
We can convert smiles into musical notes and winks into audio controls.
This virtual interface has no moving parts, requires no extra hardware, and can run on most modern computers and mobile phones.
We can use a person's own expressions and movements to control and manipulate hardware, software and data to create music.
I have been researching and creating accessible technology for over twenty years and for the past 10 have been helping run a monthly workshop alongside Drake Music Charity that brings together people who struggle to play traditional instruments, and technology focussed individuals who know how to build and make things. The charity work directly alongside schools and individuals offering real world use cases and genuine feedback - indispensable when simplicity is the aim. Together we have created a whole number of accessible musical instruments, some that are repurposing of existing instruments and some entirely new. My focus for the past few years has been to try and create musical instruments & music creation tools with the lowest barriers to entry - intuitive tools that everybody can play instinctively, that are easy to obtain and fun to play too! Currently the only requirement for this one to play is a mouth, but that could be swapped for eyes or eyebrows, the technology is quite broad!
Ultimately, this tool allows all sorts of people to create their own sounds and music but the technology can be expanded to control anything.
Add more musicality, expression and to create an app version that would allow it to run smoother on mobile devices. I think this would be key to getting it onto everybody's devices and the app stores also offers retail opportunities.
Already there are plans for a recording looper and effects but ideally the full magnitude of the project would be suggested by user cases in an attempt to streamline the simplicity and minimize development costs whilst still offering a viable product.
Can be used as a controller for other hardware MIDI devices and in duet-mode allows 2 people to independently control individual MIDI channels in realtime together.
In single player mode, the MIDI events currently get broadcast to all channels.
Set the global lights query option to select the physical Plasma LED order. ?lights=default uses the standard Picade Max layout, ?lights=table uses the tabletop layout, and ?lights=picade uses the six-light a,b,lb,rb,y,x Picade order.
Custom mappings are comma-separated physical light slots ordered left-to-right across the panel. Prefix every button with its one-based player number, for example ?lights=1x,2rb,1y,2lb,1a,2b; the first item is light 1, the second is light 2, and so on. Join controls with + to assign several buttons to one light, such as ?lights=1a+1b,-,-,-,-,1x; use - for an unassigned physical light. Both raw + and encoded %2B separators are accepted in the query string. Supported button names are a, b, x, y, lb, rb, lt, rt, start, select, ls, rs, dup, ddown, dleft, and dright; l1, r1, l2, r2, up, and down are accepted aliases. Use uniquely indexed system lights such as s1, s2, and s3 for programmable lights with no gamepad button, for example ?lights=1a,1b,1lb,1rb,1y,1x,s1; target one with setSystemLight('s1', colour). The table preset has fifteen physical lights, including s1 at light 8.
I would also like to develop certain different modes - as well as the expressive mode which works in direct relation with facial movements (as it currently works) would be a more fun but equally satisfying pop music factory mode where the mouth controls the playback and amplitude of the next phrase in a sequence (loaded from a MIDI file) whilst a backing track supports you. I think that would be a lot of fun and smooths over the latency issues.
Given enough time it would be possible to save MIDI files directly from the app too, turning it into a rudimentary face controlled DAW.
- If your MIDI device is not found, try connecting it directly to a USB port, rather than through a hub or USB-C convertor
- If your face is not found, try and improve the brightness in the room by turning on some lights
- If the website does not load and shows "loading" forever, hard refresh your browser
- After installing the app it doesn't load, uninstall and re-run it
- Install NodeJS v16.16.0+ and Yarn / NPM
- Download the source code and enter the directory
- open a terminal / cli / bash / command prompt
- enter
yarn install - then
yarn build - Your interFACE will be built into the dist folder
All commands are run from the root of the project, from a terminal:
| Command | Action |
|---|---|
yarn install |
Installs dependencies |
yarn reset && parcel serve source/*.pug -p 909 |
freshstart |
parcel serve source/*.pug -p 909 |
start |
node actions/clean.mjs |
clean out the dist/ folder |
node actions/changelog.mjs |
create change log and changes.md |
jsdoc -d dist/docs --configure .jsdoc.json source/index.js |
create documentation |
yarn dev |
Starts local dev server at localhost:909 |
yarn build |
Build production site to ./dist/ |
yarn test |
Ensure the environment is set up to build |
release:* |
patch/minor/major : increment version and deploy |
- Better musicality and expression
- Upgraded synth engine
- MIDI phrase support
- Percussion that sounds beefier!
- Hand remote Controls
- Full Body version for beats
- Saving "Personas" and using face ID loading them back in
- Vocoder mode using microphone
- Improved timing
- MIDI 2.0 (MPE) Support
- Better WAM2 support and implementation
- Favouriting instruments so they go to the top of the list per person
- Adding new instruments!
- Face
- Video camera or web cam
- A screen with dimensions greater than 320 x 280 pixels
- A relatively modern computer (GPU prefered)
- Speakers or headphones or MIDI equipment
- Optional: GamePAD / Mouse / Keyboard / finger to change settings
- Requires Yarn 3+
- Drake Music for inspiration and support
- Font is Oxanium by Severin Meyer
- Yoshimi Synthesizer ported by Jari Kleimola, Alan Calvert and maintained by Will J Godfrey
- TensorFlowJS is maintained by Google
- Gawain Hewitt
- Becky Morris Knight
- Tim Yates
- Andrew McPherson
- Darren Southea
- Michel Buffa
- Peter Salomonsen
- Jean-Baptiste Thiebaut
- Severin Meyer
- Will J Godfrey
-
Suite of none touch instruments / tools amic museum
- Accessible - high contrast colours to remove visibility barriers, keeping an interface streamlined to remove complexity barriers
- Authentic - co-created with disabled folk
- Affordable - free mode for all courtesy of github
- Available - online, no password, installable directly, total privacy
- Adaptable - configurable and mappable
- augmented reality
- extended reality
- pose piggybacking
- natural interface
- musical instrument
- synthesizer
- MIDI hardware controller
- head angle causes issues with the calculations
- enhanced model precision
- recording performances as videos
- analysis of voice commands!
- recognition of faces and preserving state
-
[Sentiment-based robot sonification can enhance comprehension, emotional meaning and engagement in the fairy tale listening: A preliminary study
Yeaji Lee a, Sophie Thompsen a, Ziming Fang b, Myounghoon Jeon a b](https://doi.org/10.1016/j.ijhcs.2026.103805)
https://www.sciencedirect.com/science/article/abs/pii/S1071581926000807?via%3Dihub