Controller audio: mic + speaker over the Satellite protocol - #178
Merged
Conversation
The client half of controller audio starts where every capability does: the model that decides whether an emulated pad may claim a microphone and a speaker at all, and the surfaces that tell the user about it. Nothing captures or plays yet; the later waves fill the pipelines behind these caps. Two new features, MIC (SEND, slug "mic") and SPEAKER (RECEIVE, slug "speaker"), with descriptor bits CAP_MIC 0x0040 and CAP_SPEAKER 0x0080 mirrored off satellite core/types.h and named booleans in the descriptor JSON. They are independent directions: a client can play a pad's audio without sourcing its microphone, and either can be advertised alone. Where they come from, layer by layer: - Input. The virtual pad claims both unconditionally, because the phone IS the actuator: its own microphone and speaker stand in for the emulated pad's endpoints, so neither needs a hardware probe. A Direct-claimed physical pad claims only what Android actually routes: we take the HID interface and leave the pad's USB-audio function with the OS, so the model tables are the wrong source and the new PadAudioRoutes table is the right one. Its resolver lands with the playback wave; until then it reports no routes and physical pads advertise neither surface, which is the honest answer for a pad whose audio function the OS never enumerated. - Type. The catalog's mic/speaker slugs flow through the existing slug-driven mapper. The bundled offline fallback carries them on the two Sony types only, since those are the identities a host can materialize with audio endpoints. The legacy v1 translator explicitly does not: v1 is a fixed historical shape, and a satellite still serving it predates controller audio. - Host. This is the one runtime-switched fact, so unlike the other per-type surfaces the host layer really does gate it. The per-backend `audio` flag on GET /api/server/capabilities folds the host's controllerAudio setting into the backend's own ability, and only an AVAILABLE backend reporting it opens the gate. It is opt-in: an unprobed satellite (or one predating the field) offers no audio, so we never cost a user a permission prompt for a host that cannot land it. Because no other document carries `audio`, the probe merges it in on its own and a later catalog write carries it forward. - Transport. Satellite only. The Moonlight control protocol has no controller-audio message and no microphone channel, so its profile is untouched. The two toggles ride the binding screen next to motion and rumble, with the same per-slot persistence, and gate the wire caps: a slot that will not capture must not have the host accept mic frames or send it a mute lamp, and one that will not play must not be sent audio at all. Mic defaults OFF (it is opt-in, and it needs a runtime grant), speaker defaults ON like rumble. When RECORD_AUDIO is missing the mic row says so and offers the ask rather than being a switch that silently does nothing; MicPermissionGate publishes the grant and the request seam is one function in the activity for the capture wave to fill. The rest is description: Microphone and Controller sound rows on the setup capability tables (relevance-filtered like Battery and Light bar), mic/speaker chips on the destination cards and review flows, capability pills on the dashboard card, two new 24dp stroke icons matching the existing feature set, and strings in all six languages.
The client's half of the two audio streams, from the socket up to a JNI surface the capture and playback waves can call. Nothing captures or plays yet; what lands here is the transport, the codec, and the ordering rules both ends of a stream have to agree on. Descriptor caps first, because it turned out none of this could have reached the wire. SatelliteConnection built every descriptor as a hardcoded base (analog triggers + rumble) OR'd with a motion bit, while CapabilityResolver.wireCaps had been computing the whole word since protocol 2 and the manager was already watching it for change detection. So CAP_LIGHTBAR, CAP_TRIGGER_EFFECTS and CAP_PLAYER_LEDS were resolved and then dropped on the floor, and CAP_MIC/CAP_SPEAKER would have joined them. The connection now takes a per-slot caps provider and uses it at all three sites; the composer projects it through wireCapsFor. The "Android has no controller-LED API" comment that justified the old base goes with it: it stopped being true when the feedback return paths landed. The wire, mirroring satellite core/types.h byte for byte: MSG_MIC_AUDIO 0x0012 up, MSG_SPEAKER_AUDIO 0x0013 down, MSG_MIC_LED 0x0014 down, the two stream messages sharing one ctrlIdx + big-endian seq header ahead of exactly one 20 ms Opus packet. The datagram ceilings move with them: a whole Opus packet does not fit the old 256-byte inner payload or the old 128-byte receive buffer, so both are now derived from the contract's 1500-byte MTU rather than from what today's senders happen to emit, with static asserts tying the buffers to that arithmetic. Opus comes from a pinned release tarball through FetchContent, the same shape as libsodium, and costs ~400 KB per ABI in a release-grade build. The wrapper pins the formats the wire fixes (mic mono VOIP 32 kbps with in-band FEC, speaker stereo 48 kHz) and refuses a window that is not exactly 20 ms, because a mis-framed packet is one the far end cannot place in its timeline. audio_jitter.h is a deliberate copy of satellite's reorder window rather than a reimplementation of it: both ends decide what counts as lost, what counts as late, and how long a hole is worth concealing, and a divergence there is audible rather than testable. Its suite mirrors satellite's case for case for the same reason. Speaker frames get their own queue and thread instead of riding the input bridge or upcalling straight from receiveAck like the other return paths: an audio frame must never evict a gamepad report, and the sink at the far end is an AudioTrack whose write blocks. So the receive thread copies the packet and returns, and reorder, decode and concealment all happen off the socket. Mic capture encodes on the caller's own thread, which is where the capture wave wants the work anyway. wButtons 0x0800 is exposed on both sides as the mic-mute bit with nothing setting it yet. The sweep in GamepadButtonLayoutsTest is the interesting half: every other bit of the XINPUT-shaped word is reachable from a real button and this one is reachable from none, which is what makes it free to spend. One gate fix rides along. CMake 4.x names the scratch file its post-build gtest discovery writes after a hash of an argument it never passes, so every target in the host test build shared one file; with seven targets building in parallel that failed outright, and before that it had been quietly listing one target's tests under another (the run that reported 315 tests for 263 real ones). Discovery moves to PRE_TEST, which is serial by construction and generator-independent.
The wave that makes a microphone real, and the one that owes the user a promise about it: while muted, or switched off, or ungranted, or not streaming, ZERO MSG_MIC_AUDIO packets leave the device. Not silence on the wire, no packets, and the enforcement is arranged so that is provable rather than asserted. Four facts decide it, and any one of them going false has to stop the microphone: the slot is bound to a live satellite, the whole capability path carries a microphone and the user switched it on, RECORD_AUDIO is granted, and the slot is not muted. They move independently at runtime, so they are combined into one plan by MicCaptureComposer and reduced by one pure rule in MicCapturePolicy, rather than being checked as four scattered guards that each have a window where they disagree. Its suite walks all sixteen rows of the matrix; exactly one of them captures. MicEngine turns that plan into an AudioRecord: VOICE_COMMUNICATION for the free echo cancellation a phone pointed at its own speaker badly needs, 48 kHz mono, one 20 ms window at a time on a dedicated URGENT_AUDIO thread, straight into the native Opus encoder the previous wave landed. The gate is applied twice on purpose. The eligible target set is published before anything starts and cleared before anything stops, so the ineligible state is the one in effect during the transition; and it is re-read for every window AFTER that window was recorded, so a mute landing mid-window drops the window instead of shipping it. That is what bounds mute latency to one frame, and it is what the integration test measures against a real session. One recorder, fanned out to every eligible slot, because the phone has one microphone and two emulated pads asking for it are asking for the same sound. Per-route capture is the playback wave's problem, when a pad's own headset becomes a preferred device. A device that refuses VOICE_COMMUNICATION falls back to plain MIC; there is deliberately no sample-rate fallback and no resampler, since 48 kHz mono is the platform's native capture format and inventing a converter for a case that does not arise would put untested arithmetic in the one path where a bug is audible on somebody else's PC. Mute has two controls and one state. The on-screen DualSense grows the mute button the real one has, under the PS button, as a local-only bit (1 shl 12, following the touchpad click at 1 shl 11) whose PRESS toggles the state; the wire's WBUTTON_MIC_MUTE carries the state itself, held on every frame including resends, so the host reads the pad as muted rather than seeing blips. A Direct-claimed DualSense's own button does the same thing from the other end: the report decoder owns that latch, because the pad's input report is built and sent entirely on the USB reader thread and a latch behind a JNI call would put the JVM in that path. The resulting state comes back up through MicMuteBridge into the same per-slot holder, so the capture engine and the lamp read one place whichever button was pressed. The mute lamp lights locally the moment you press, and a host MSG_MIC_LED overrides it afterwards, last writer wins, which is what the hardware does. The foreground service gains the microphone type, and only while a mic-enabled binding is streaming. It follows arming and not delivery on purpose: a while-in-use type can only start from the foreground, so dropping it on every mute toggle would be a coin flip on getting it back, and zero packets while muted is enforced where it belongs. A denied type falls back to the one the session can always hold rather than taking the session down. The store paperwork changes with the code, because it has to. PRIVACY.md loses the "we do not request microphone" line and gains the exact conditions capture runs under; the Data Safety form declares audio as collected, transmitted off-device to the user's own paired PC, processed ephemerally, optional and user-initiated; the permissions justification explains both RECORD_AUDIO and the service type; and the content rating stops claiming no user-to-user communication, since a voice that reaches the PC reaches whatever is running on it.
The last client wave, and the one that makes the return direction real: what a game plays through the emulated pad's speaker now comes out of something, what a pad's own endpoints can carry is decided by what Android actually routes, and the mute lamp reaches the hardware that has one. Playback. Frames arrive from the native reorder thread already in stream order, already whole, and already concealed where a packet never came, so the only decision left per frame is which output it belongs to. That is why the plan is keyed by (session handle, controller index): the delivery path is one hash lookup on a volatile snapshot, no lock, no allocation, no walk of the connection map, on a thread every stream shares. Resolving that pair to a slot is the same job FeedbackRouter does for the lamp, done once per plan instead of once per frame. The write is non-blocking rather than handed to a per-slot writer thread. One native dispatch thread carries every stream, so a sink that waits does not stall its own pad's audio, it stalls everybody's; AudioTrack's WRITE_NON_BLOCKING returns a short count instead, and a short count is a live stream running ahead of playback, which dropping the newest window is exactly the cure for. The dropped samples are counted rather than swallowed. A writer thread would have added a queue, a copy and a thread per slot to reach the same place. Playback starts after two windows are buffered, not on the first, because a track started empty plays one window and then underruns, and every underrun is a click; that is the same 40 ms the reorder window upstream already costs, so it adds no latency the stream did not have. The session's lock is not for throughput but for lifetime: writes come from the dispatch thread and closes from the collector, and releasing an AudioTrack out from under a write in progress is a use-after-free. Routing. There is no public API that puts a vendor:product on an AudioDeviceInfo, so the pad-to-endpoint table matches on the one field both sides genuinely share, which is the USB device's own iProduct string, and every ambiguity resolves to no route rather than to a guess: the pad must carry a USB Audio Class interface of its own, the name must identify exactly one attached device, and at most one endpoint per direction. Two DualSenses, or a DualSense beside a DualShock 4, both answer to "Wireless Controller" and are therefore unmatchable, because routing a slot to the wrong pad's speaker is worse than routing it nowhere. The resolver listens on the audio side, not the USB side: the table is a statement about endpoints, and a pad whose audio function the OS never enumerated has no route however long its cable has been in. Publishing re-runs the capability composition, so caps appear and vanish with the endpoint. Capture grows a route split to match. An AudioRecord's preferred device is fixed when it is built, so the phone's microphone and a pad's headset cannot share one: the engine now holds one recorder per distinct endpoint, with the delivering set grouped the same way, and the route table rides its upstream because a pad's endpoint appearing moves a slot between recorders without changing the plan at all. The privacy invariant is unchanged and enforced in the same two places, now per recorder. The mute lamp on a Direct-claimed DualSense. Report 0x02 carries the lamp at byte 9 and power_save at byte 10 in this file's convention (report id at out[0], so hid-playstation's stripped 8 and 9 plus one, the shift the player LEDs already carry). The power-save mic-mute bit rides with it because the lamp and the microphone amplifier are one thing on that pad: a lit mute lamp over a live microphone is the failure this whole feature exists to prevent, and pulse counts as lit for the same reason. Every other 0x02 report we build re-asserts the lamp from a shadow in FeedbackState, since each builder starts from a fresh memset and the firmware applies whatever the valid flags claim, so a colour change that flagged the lamp field and left it zeroed would turn the lamp off as a side effect. Nothing changes for a host that never drives it. On the on-screen pad, pulse breathes the accent ring the adaptive-trigger effects already use rather than introducing a second treatment, and it is the only lamp state that costs frames.
Two instrumented tests failed on the CI emulator while the rest passed. Both were real, and only one of them was a test problem. The host's controller-audio verdict never arrived. `audio` rides the per-backend entries of GET /api/server/capabilities and nothing else carries it, and the only caller of the repository that reads it was ConfigureBindingsViewModel. So a session that reached Live without anyone opening the binding screen never learned the verdict, and unknown is opt-OUT by design: the host layer withheld MIC and SPEAKER, the caps never went live, and the playback engine correctly opened nothing. No timeout would have helped, because it never converged. That is a user-facing bug and not a harness gap: startup auto-reconnect binds and streams without visiting that screen, so a returning user would have had controller audio silently off until they happened to open it once. HostCapabilitiesProbe reads the document when a link goes Live, on the same trigger and in the same shape as CatalogPrewarmer, watching the raw connection states for the same reason. It dedupes on the session HANDLE rather than on a Live transition, because a transition is the wrong question twice over: a heartbeat blip is one session recovering and reads as a fresh Live, while a drop and reconnect the collector only ever sees as a blip is a genuinely new session and reads as none. The handle tells them apart. Its watcher is cancelled when the connection leaves the map, so a forgotten host leaves no collector behind. The microphone one was a test that could not see what it was asserting. It took the engine's word that it had stopped, joined a thread on a fixed timeout, cleared the fake's received list and slept. Neither step is a barrier: a capture body told to stop is still inside a blocking read, and delivery is UDP with the fake reading on a thread of its own, so a datagram sent BEFORE the stop can arrive after the clear. On a loaded emulator it did. So the engine now says when it has actually stopped rather than only when it was told to. `quiescent` is true when no capture body is executing, which is the property the privacy claim is about; MicCaptureState.Idle lands with the plan and is a different question. The test waits for that, then drains the wire with a marker window it sends itself: loopback delivery is ordered, so once the marker lands every earlier datagram has landed and is cleared with it, and anything arriving afterwards can only come from a capture still running. The mute test gets the same quiescence wait before it asserts the recorder was released, and the speaker test now waits for the capability model to carry SPEAKER before it waits for the engine, so a future failure names which half broke.
emir-hasanbegovic
marked this pull request as ready for review
September 1, 2026 14:32
The mic stream paid full price for a quiet room. A live microphone never goes digitally silent -- room tone, breath and the preamp floor are all real samples -- so nothing downstream can collapse it and the only thing that can is the encoder's own VAD. OPUS_SET_DTX(1) turns it on: measured on libopus 1.6.1, 123 of 250 frames gated at -50 dBFS after speech, 30.0 -> 16.4 kbps, and digital silence from a closed capture path drops 8.4 -> 1.1 kbps. The speaker encoder deliberately does not get it. That gate cuts anything ~26-30 dB below the recent peak, which on game audio replaces a reverb tail or quiet ambience with comfort noise at -2.3 dB SNR. The satellite suppresses exact digital silence on that stream instead, which cannot touch anything a listener could hear. This client only decodes speaker anyway; the setting is here so the two ends stay one file's worth of the same decision. Muting is unaffected and remains the stricter thing: it stops delivery outright, which DTX cannot do and must not be confused with. Correcting the comment while here, in the same terms as satellite's: the expected-loss hint, not the application, is what picks the mode. It forces SILK in, so both streams encode as Hybrid fullband and both really do carry in-band FEC. The old text said VOIP was what bought FEC and implied the speaker had none; anyone acting on that by dropping the hint would have reached CELT and silently deleted working FEC from both streams. Verified nothing chokes on the 1-byte packets DTX emits: the JNI send path refuses only a zero-length encode, the seq still advances per packet (a DTX packet is sent, unlike the satellite's speaker suppression which sends nothing), header+1 meets AUDIO_WIRE_MIN_PAYLOAD_BYTES, and the reorder window accepts one. 310 native tests pass, 306 before.
The satellite now sends nothing at all for a digitally silent speaker window. The reorder window handles that perfectly -- it has no clock, so a stall is not even representable in its state, and a resumption at a contiguous seq takes the fast path as if no time had passed -- but the AudioTrack under it drains while the quiet lasts, and that part was not handled. The two-window start threshold exists because a track started empty plays one window and then underruns, and every underrun is an audible click. It was latched: playing was set once and the cushion was never rebuilt. That was fine while the stream was continuous, which it was until the host learned to suppress silence. Now the first quiet stretch destroys the cushion and the stream runs at nearly zero occupancy afterwards, so every scheduling hiccup on the shared dispatch thread is the exact click the threshold was written to prevent. Refill with silence rather than pausing to re-prime. Withholding windows until the threshold is met again would strand any sound shorter than the cushion: a lone 20 ms blip would sit unplayed until the next one arrived, which is worse than the click. Writing two windows of zeros delays the resumed audio by the same 40 ms the first start pays and can never swallow it. The trigger is AudioTrack's own underrun counter, not a clock and not a frame-position subtraction, which keeps wrapping arithmetic out of the one path this file warns is audible when it is wrong. The decision itself is a pure function next to SpeakerPlayoutPolicy so it can be tested without a device; the JVM suite stubs android.media, so nothing else about the track is reachable from a unit test. Also corrects the libopus attribution in the DTX comments: the 1.6.1 figures are satellite's, and this repo pins 1.5.2, where the suite re-proves the collapse holds rather than inheriting the number.
The satellite split its one controllerAudio switch into a microphone and a speaker switch, and publishes both in a new top-level controllerAudio block on GET /api/server/capabilities. This reads it. HostFeatureSet carried one boolean that fanned out to both MIC and SPEAKER, which cannot express what the host now reports. Split into controllerMic and controllerSpeaker: a single verdict would have a speaker-only host offering a microphone whose frames it drops on the floor, after charging the user a permission prompt for it. Everything downstream follows without changing -- capture and playout already read Feature.MIC/SPEAKER off SlotCapabilities.live, and the host layer is where that is decided. The DTO field is NULLABLE, like motion and unlike host, because an absent block is UNKNOWN rather than off. A satellite that predates it still carries audio and still reports it on the per-backend audio flag, so reading a missing block as two falses would mute every older host. Absent falls back to that flag for both directions; present wins outright, since it is the only place the directions are reported apart. Left the advertised wire caps alone. SatelliteConnectionManager projects wireCaps through distinctUntilChanged specifically so host and runtime changes that do not move the descriptor raise no wire traffic, and the contract lets the host flip these two under a live stream -- feeding them in would turn every host-side toggle into a mid-stream descriptor re-declare on every bound slot, which is the churn that projection exists to prevent. Nothing leaks by leaving it: the phone stops sourcing either way, because the plans read .live. 2408 JVM tests both flavors, 2397 before; 310 native; ktlint and detekt clean.
The user's report was "the mute pill did nothing; the mic would only stay on". Two real bugs, both fixed here: 1. The pill's visual state was VirtualPadFeedbackStore.micLedState, which was last-writer-wins between the local toggle and the host's MSG_MIC_LED. Any host software driving the emulated DS5's lamp repainted the pill right after every local toggle, so the button looked dead even while MicMuteStore toggled and the recorder really closed. Worse than cosmetic: the wire's WBUTTON_MIC_MUTE carries the mute STATE as a held bit, so host software that treats it as a held BUTTON toggles its own mute per press edge and drifts out of phase with the client, after which the lamp can assert the exact opposite of local truth indefinitely. Fix: decouple the two visual channels. The pill's face (glyph + ground) now renders the LOCAL MicMuteStore state unconditionally, via a new micMuted property fed from the overlay's existing store collector: slashed glyph over a warning wash while muted, so no host message can make a muted microphone look live. The host lamp keeps only the accent ring (and pulse breath) as secondary "what the host thinks" info. setLocalMicMute is retired; micLedState now has exactly one writer, MSG_MIC_LED. 2. GamepadLayout clamps the pill up on short screens until it overlaps the home button's pickup halo (smallBtnRadius x 1.5), and the recognizer hit-tested home BEFORE the pill, so presses inside the pill became BTN_HOME presses. The pill is now hit-tested before the three centre circles: a drawn rect beats an invisible forgiveness zone, and home loses only halo, never its own disc. The rest of the press-to-recorder chain (edge detection, composer keying, MicEngine's per-window mute re-read) was audited and is sound. The pressed/muted/idle ground choice is a pure function (micMutePillFace) so it is pinned without a view, alongside a clamped-geometry regression test for the hit order and rewritten store/router suites for the single-writer lamp.
The mute pill only exists on the gamepad overlay, but the microphone keeps capturing wherever the user navigates, and a mic the user cannot see or silence from the current screen is a mic they cannot account for. Following the capture pipeline's shape (pure rule -> fold -> store -> UI): - MicIndicatorState (HIDDEN / LIVE / MUTED) with MicIndicatorPolicy, a pure rule over MicCapturePlan. It reads only the plan, because the composer already folds MicMuteStore into `delivering`; reading the store again would race that fold. toggleAll is the one control: all-or-nothing over the armed slot set, so the single device-wide state its surfaces show converges instead of ping-ponging per slot. - MicIndicatorCoordinator, the shared read/write point: maps the plan to the indicator and applies a toggle-all order to MicMuteStore, so the chip and the notification action mean exactly the same thing. - A floating mic chip (overlay_mic_chip) on every screen while state is not HIDDEN: red live mic glyph reading "mic is hot", grey slashed glyph reading muted, distinct labels and content descriptions, one tap toggles everything. Installed once in attachGamepadHost, the same scaffolding that binds the low-power chrome, and gated by the StreamingScreenCoverageTest include check so a new screen cannot forget it. The gamepad overlay suppresses the binding: its pad already renders the mute pill from the same store, and a floating target over a full-screen control surface would steal pad touches. - The streaming notification carries the mic state as subtext and a Mute mic / Unmute mic action while a mic is armed, so the control works from the shade outside the app. The action rides a new ACTION_TOGGLE_MIC through a pure command mapping, where an unknown action stays the re-assert no-op rather than ever toggling a mic. Strings in all six locales; chip paint, indicator matrix, toggle-all convergence and notification mappings pinned in dedicated suites.
Contributor
Author
|
Two follow-up commits from live testing feedback:
Net +26 tests; 2433 JVM + 310 native green, lint/ktlint/detekt/format/translation gates clean. |
The accept loop registers each connection handler in the shared thread list, so a connection landing while close() walks that list for joins throws ConcurrentModificationException and fails whichever test closed the fake (seen on tofu_rejectsADifferentCertOnTheSameIdentity in CI). Register and snapshot under one lock; a thread registered after the snapshot only ever sees closed sockets, so skipping its join keeps the existing best-effort shutdown semantics.
This was referenced Sep 2, 2026
emir-hasanbegovic
added a commit
to TinkerNorth/dish-linux
that referenced
this pull request
Sep 2, 2026
Ports controller audio (protocol 2 extension; satellite TinkerNorth/satellite#88, dish-android TinkerNorth/dish-android#178, dish-windows TinkerNorth/dish-windows#66) to the Linux client. Physical Direct-claimed pads only; no virtual controller exists here. This is the file-for-file port of both Windows waves in one commit: wire (MIC_AUDIO 0x0012 up / SPEAKER_AUDIO 0x0013 down / MIC_LED 0x0014 down, caps mic 0x0040 / speaker 0x0080, receive buffer 256 to 1500, 1472 send guard), the capability fold with first-time consumption of GET /api/server/capabilities (probed per session PUT, conservative-false), the Opus codec and shared 2-frame jitter mirror, SDL audio capture/playout engines with the zero-packets-while-muted invariant, product-string pad-to-endpoint matching (ambiguity publishes nothing), the DualSense mute button and wButtons 0x0800 mute state, MIC_LED actuation with the FeedbackState lamp shadow, and slot-card mute controls showing local truth. Linux-specific deltas: the pad string comes from the USB product attribute (iProduct) with HID_NAME as fallback, since pipewire/alsa names derive from iProduct while HID_NAME prepends the manufacturer; opus arrives via pkg-config; Moonlight cannot see 0x0800 structurally (the explicit button map has no such row, pinned by test); packaging grows libopus across CI, deb/rpm (via shlibdeps), AppImage (SDL built with audio + libpulse backend), and both Flatpak manifests gain --socket=pulseaudio. Of the 33 new source files, 29 are byte-identical to dish-windows; the four that differ are three cross-platform comment variants and the POSIX loopback test. 12 new test suites plus 10 extended ones. This box has no Linux toolchain, so beyond syntax/format/translation gates the proof is this PR's CI: the compile+ctest lane, the stricter clang-tidy, TSan over the engines, and the package job's shlibdeps.
emir-hasanbegovic
added a commit
to TinkerNorth/dish-windows
that referenced
this pull request
Sep 2, 2026
Ports the controller-audio half of protocol 2 (satellite PR TinkerNorth/satellite#88, dish-android PR TinkerNorth/dish-android#178) to the Windows client. Physical Direct-claimed pads only; there is no virtual controller here. This first commit lands the wire, the capability model, the host verdict, and the codec cores. A second wave adds the SDL audio engines, pad-to-audio-device routing, the DualSense mute button, and MIC_LED actuation; until then no slot advertises an audio cap (pinned by test). **Wire additions** (protocol stays 2, everything caps-gated): | Op | Dir | Payload | |---|---|---| | MSG_MIC_AUDIO 0x0012 | c to s | ctrlIdx + seq u16 BE + one 20 ms Opus packet (mono 48 kHz, VOIP, ~32 kbps, FEC, DTX) | | MSG_SPEAKER_AUDIO 0x0013 | s to c | same header, stereo 48 kHz, AUDIO, ~96 kbps, FEC | | MSG_MIC_LED 0x0014 | s to c | ctrlIdx + state (0 off / 1 on / 2 pulse), coalesced | Caps mic 0x0040 / speaker 0x0080; wButtons 0x0800 reserved as the DualSense mute-state bit (never set yet, never leaks through the Moonlight mapping). Datagram ceiling raised to 1500 both directions: the receive buffer was 256 bytes and would have truncated every audio frame into an AEAD failure, and sendEncrypted now refuses inner payloads over 1472 instead of emitting fragments. **First-time host-verdict wiring**: GET /api/server/capabilities had zero callers; it is now probed after every successful session PUT (open and rekey) and folded per direction (controllerAudio block, per-backend audio flags) into the capability solver, conservative-false until a probe says yes. **New cores**: AudioJitter.h (third mirror of satellite/android's 2-frame reorder window, edit together), Opus codec wrappers pinned to the contract formats, HostAudioVerdict fold, per-binding MicEnabledStore (default off) / SpeakerEnabledStore (default on). 95 new tests (+6 solver, +3 routing, +3 models extensions), 2025 total green locally with clean format/tidy/qml/translation gates. Six locales updated. --------- Co-authored-by: Emir Hasanbegovic <1190336+emir-hasanbegovic@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Controller audio over the Satellite protocol: the phone (virtual pad) or a plugged DualSense / DualShock 4 v2 supplies the emulated pad's microphone, and plays the emulated pad's speaker/headset audio. Extends the unreleased protocol 2 in place (no version bump). Server counterpart: TinkerNorth/satellite PR #88.
Scope: the controller's OWN audio endpoints, never host game audio. The Moonlight path is untouched (that protocol has no controller-audio or mic channel).
What this adds
mic0x0040 /speaker0x0080; datagram ceiling 1500 (inner payload 1472); Opus 48 kHz 20 ms both ways (mic mono VOIP 32 kbps with in-band FEC, speaker stereo 96 kbps), a 2-frame reorder window with PLC, no acks or retransmits.audioverdict), binding-screen toggles (mic defaults off), setup rows, destination card pills, six locales.Pre-existing protocol-2 bugs found and fixed along the way
Tests: 2389 JVM (+240 over main), 306 native (+55), 10 instrumented on-device integration tests including the full mic eligibility matrix and real-codec wire round trips. All CI green including the emulator job.
Following the satellite's controller-audio changes
Three changes that pair with satellite #88, plus one regression that pairing created.
DTX on the mic encoder
The mic stream paid full price for a quiet room. A live microphone never goes digitally silent — room tone, breath and the preamp floor are all real samples — so nothing downstream can collapse it and the only thing that can is the encoder's own VAD.
OPUS_SET_DTX(1)turns it on.This is the encoder that actually ships: the satellite's
IAudioCodecFactoryexposes onlymakeMicDecoderandmakeSpeakerEncoder, so its mic encoder is test-only and the bandwidth was always going to be saved here.docs/contract.mdhad already mandated it (| Silence | Opus DTX on (encoder-side VAD) |); the client was simply non-conformant.The speaker encoder deliberately does not get it — that gate cuts anything ~26–30 dB below the recent peak, which on game audio replaces a reverb tail with comfort noise. The satellite suppresses exact digital silence on that stream instead, which cannot touch anything a listener could hear.
Muting is unaffected and remains the stricter thing: it stops delivery outright, which DTX cannot do.
Verified nothing chokes on the 1-byte packets DTX emits — the JNI send path refuses only a zero-length encode,
seqstill advances per packet,header + 1is exactlyAUDIO_WIRE_MIN_PAYLOAD_BYTES, and the reorder window accepts one.FakeSatellitealready re-implemented the host's guard for this case.Corrects a wrong comment while here, in the same terms as the host: the expected-loss hint, not the application, picks the mode. It forces SILK in, so both streams are Hybrid fullband and both really do carry in-band FEC. The old text said VOIP was what bought FEC and implied the speaker had none; anyone acting on that by dropping the hint would have reached CELT and silently deleted working FEC from both streams.
The playout cushion, after the host learned to go quiet
The satellite now sends nothing at all for a digitally silent speaker window. The reorder window handles that perfectly — it has no clock, so a stall is not representable in its state, and a resumption at a contiguous
seqtakes the fast path as if no time had passed. Confirmed by reading rather than assuming: no<chrono>, no timeout, no watchdog, and the dispatch thread useswait, notwait_for.But the
AudioTrackunder it drains while the quiet lasts, and that part was not handled. The two-window start threshold exists because "a track started empty plays the first window and then underruns, and every underrun is an audible click" — and it was latched.playingwas set once and the cushion never rebuilt. Fine while the stream was continuous, which it was until the host stopped sending silence; now the first quiet stretch destroys it and the stream runs at nearly zero occupancy afterwards.Refilled with silence rather than a pause-and-re-prime: withholding windows until the threshold is met again would strand any sound shorter than the cushion, leaving a lone 20 ms blip unplayed until the next one arrived — worse than the click. The trigger is
AudioTrack's own underrun counter, which keeps wrapping frame arithmetic out of the one path this file warns is audible when wrong. The decision is a pure function besideSpeakerPlayoutPolicy, because the JVM suite stubsandroid.mediaand nothing else about the track is reachable from a unit test.The host's per-direction verdict
HostFeatureSetcarried one boolean that fanned out to bothMICandSPEAKER, which cannot express what the host now reports. Split in two. Everything downstream follows unchanged — capture and playout already read the features offSlotCapabilities.live, and the host layer is where that is decided.The new DTO field is nullable, like
motionand unlikehost, because an absent block is unknown rather than off: a satellite predating it still carries audio and still reports it on the per-backendaudioflag, so reading a missing block as two falses would mute every older host. Absent falls back to that flag for both directions; present wins outright.The advertised wire caps are deliberately left alone.
SatelliteConnectionManagerprojectswireCapsthroughdistinctUntilChangedprecisely so host and runtime changes that do not move the descriptor raise no wire traffic, and the contract lets the host flip these two under a live stream — feeding them in would turn every host-side toggle into a mid-stream descriptor re-declare on every bound slot. Nothing leaks by leaving it: the phone stops sourcing either way.Verification
2408 JVM tests on both flavors (2389 before), 310 native (306 before), ktlint / detekt /
clang-format22.1.4 all clean.Not verified without hardware: whether post-stall clicking was audible on a real device in the first place, and whether the refill removes it. The logic is unit-tested; the audibility is not. It needs a device: play, let the host go quiet for ~5 s, resume, and watch
dumpsys media.audio_flingerunderrun counts.Two things worth knowing, neither fixed here
A pre-existing replug bug that this change makes worse. On a controller replug within a live session the host resets
speakerSeqto 0 — deliberately, since a carried-over sequence number "would make the new stream look like a 40 ms reorder of the old one." The client never resets its matching reorder window, so every frame comes backAccept::Lateand is silently dropped until the host's counter climbs back past where the client left off. That used to self-heal at 50 frames/second because silence still advancedseq; now only real audio does, so a mostly-quiet game recovers far more slowly. Fixing it needs a JNI reset entry point the client does not have today.Mid-stream host toggles do not reach the client. The capabilities document is probed once per session handle and otherwise only re-read from the binding screen. The host may now flip
mic/speakerunder a live stream, and nothing polls or pushes, so a change lands only at the next reconnect or configure-screen visit.