Talking to the outside: MIDI, OSC, sensors, other instances and a shared clock

[bonuoq]

The idea

Let Kairotic talk to the outside: MIDI and OSC first, then phones, controllers, other Kairotic instances and a shared clock. Kairotic is static and runs in the browser, so everything goes through what a page can reach (Web MIDI, WebSocket, WebRTC, sensor and device APIs) and, where a page cannot (UDP), through the relay, which becomes Kairotic's network bridge as well as its GUN relay.

1. One message format before any transport

Everything that comes in or goes out is an EDN event map in the same shape as pattern events, so the patch never knows where a message came from:

{:src :midi :port "IAC Bus 1" :ch 2 :freq 440.0 :amp 0.7 :gate true}
{:src :midi :ch 1 :cc 74 :value 0.42}
{:src :osc  :address "/patch/faust~1/cutoff" :args [1200]}
{:src :motion :alpha 12.3 :beta -4.1 :gamma 30.0}

• Notes carry :freq (Hz, not note numbers), :amp (0–1) and :gate; the MIDI note number travels as :note for whoever wants it.
• midiin can record straight into a freqt; a pbind / player can go out through midiout or oscout unchanged.
• Silences and note-offs follow the pattern rules (:gate false, :rest).

2. Transports, in order

#	Transport	Where it works	Notes
1	Web MIDI	Chrome, Edge, Firefox; not Safari / iOS	no server; permission prompt; MPE so microtonal freqt / Scala tunings reach hardware without rounding to semitones; MIDI clock
2	Phone sensors + Gamepad	everywhere (motion needs a gesture-granted permission on iOS)	every phone in the room is a controller — fits Release/Perform on mobile
3	OSC	through the relay (WebSocket ⇄ UDP, osc.js)	local bridge (relay on the same machine as SC/Max/Pd) or remote; later OSCQuery so tools discover the patch's parameters
4	WebRTC data channels	all browsers	ephemeral, low-latency gestures between participants; the relay does signalling
5	Shared clock	via relay / WebRTC	NTP-style offset estimation; pulse and phase shared, Ableton Link through the relay
6	Web Serial, OSCQuery, Link	Serial: Chromium, Firefox 151+, not Safari	Arduino / ESP32 / home-made sensors

3. State vs stream

GUN keeps state (the history). Gestures are a stream: a knob moved from a phone must not become a hundred history events. So two channels:

• state (GUN) — the "important changes" of Compose/Perform mode #5, part of the tree;
• stream (WebRTC / relay) — ephemeral, enters the history only through a snapshot/keyframe or a recording into a buffer (with its recipe, as in get inspiration for EDN musical tree from smoge/Rastrum #17).

4. Components

@kairotic/io

• midiin / midiout — notes, CC, bend, aftertouch ⇄ event maps; MPE mode where midiout takes the bend per note from the Y grid's tuning
• oscin / oscout — address patterns, EDN args, relay as argument; OSC address ⇄ box path (/patch/faust~1/cutoff)
• motion, gamepad, serial

@kairotic/net (or with the collaboration code)

• send / receive across the room — named channels, flag ephemeral / into the history
• clock — shared clock with estimated offset, feeding a reader's or player's position
• link — Ableton Link pulse and phase through the relay

Glue

• map — range, curves, smoothing, debounce
• learn mode in the interface (not a box): click a parameter, move a controller

First step (open)

• Study 34 · io: the common event format, midiin / midiout with Web MIDI (MPE included), a virtual input for browsers without Web MIDI, recording into a freqt
• motion and gamepad
• OSC bridge in the relay package + oscin / oscout
• WebRTC stream channel and clock
• Serial, Link, OSCQuery

Questions: one package @kairotic/io or several by transport? Does the stream channel belong to the core (beside GUN) or to @kairotic/net?