fix(linux): Hyprland cursor telemetry + anchor recording start to the first real frame - #157
Conversation
Documents the investigation (ruled out app-level cursor constraint, hardware-cursor-plane theory, EGL renegotiation noise, useSystemPicker) and the actual root cause: Electron's desktopCapturer reuses one cursor-less PipeWire session across picker + recording, and xdg-desktop-portal-hyprland lacks Metadata cursor mode entirely. Fix sources cursor position from Hyprland's own IPC socket instead of the broken screen.getCursorScreenPoint(), feeding the existing provider:"none" generic-cursor overlay pipeline. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The Linux capture path used the legacy chromeMediaSource constraints, which have no cursor toggle at all (unlike the cursor constraint available via getDisplayMedia). Mirrors the Windows path: system audio still goes through the legacy desktop-audio API since Electron's loopback audio isn't supported on Linux. This alone doesn't fix cursor visibility on Hyprland (see the design spec for the real root cause and fix), but it's the correct constraint to request regardless of compositor. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ay cursor The editor's cursor overlay (src/lib/cursor/nativeCursor.ts) always draws a generic cursor for Linux recordings, since Linux cursor sessions always produce provider:"none" position telemetry. Requesting cursor:"always" from getDisplayMedia was based on the (incorrect, now-superseded) assumption that Linux had no overlay at all — on any setup where that constraint actually bakes the cursor into the raw video (X11, or a Wayland compositor with Embedded cursor mode support, unlike Hyprland's portal), it would double up with the overlay. Requesting "never" matches how Windows already avoids this in editable-overlay mode. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
hasEditableCursorRecording gated the whole cursor-overlay feature (preview and export) to win32/darwin only, regardless of whether valid recording data existed. This predates the Hyprland telemetry fix and made sense while Linux's position samples were always frozen garbage -- but it also means the newly-fixed, real position data silently never got drawn. Linux never captures real cursor assets either way (provider is always "none"), same as the existing macOS telemetry fallback, so it renders the same default arrow asset from position samples alone. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
captureSample stamped timeMs with Date.now() after awaiting the socket query, so every sample's timestamp was biased later than the moment the cursor was actually at that position by one IPC round-trip. That reads as a constant lag between the drawn cursor and the recording during playback. Stamp it right before the query goes out instead. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
recordingId.current (Date.now() right after acquiring the track) was used as cursor telemetry's zero-point, but on Linux/PipeWire the track can still be muted at that moment -- mid DMA-BUF modifier renegotiation (see main.ts) or renegotiating again from applyConstraints just above. Every telemetry sample ends up timestamped slightly before the frame that should show that position actually exists, which plays back as the cursor lagging the recording, confirmed even scrubbing to a single paused frame (so it wasn't a live-playback rendering artifact). Starts listening for the track's unmute event right after track acquisition (resolves immediately if already unmuted) and uses that resolved time -- instead of recordingId.current -- as the cursor session's startTimeMs. Fire-and-forget like the original call, so recorder.start() and the recording UI state are unaffected; only the cursor telemetry's anchor point changes. recordingId.current is left untouched for file naming. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
MediaRecorder's internal frame-0 timeline starts whenever the track delivers its first real frame, not when recorder.start() is called. On Linux/PipeWire the desktop-capture portal can still be negotiating (DMA-BUF modifier renegotiation) for several seconds after the track reports unmuted, so anchoring duration accounting and cursor telemetry to Date.now() at button-press time let both drift out of sync with the actual video content: declared duration exceeded the real content span (playback jumped to the end early) and the cursor overlay lagged the picture. waitForFirstVideoFrame() now blocks on an actual rendered frame (via requestVideoFrameCallback, capped at 3s) before recorder.start() is ever called, so recordingId, segmentStartedAt, and the cursor-telemetry start time all anchor to the same verified wall-clock instant.
📝 WalkthroughWalkthroughAdds Hyprland Unix-socket cursor telemetry, a Linux recording session using normalized cursor overlays, Hyprland-specific session routing, Linux editable-overlay support, and first-video-frame-anchored recording timing with platform-specific capture handling. ChangesHyprland cursor recording
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant startRecording
participant MediaStream
participant waitForFirstVideoFrame
participant HyprlandCursorRecordingSession
participant HyprlandIPC
startRecording->>MediaStream: capture platform-specific recording tracks
startRecording->>waitForFirstVideoFrame: await first decodable video frame
waitForFirstVideoFrame-->>startRecording: return firstFrameAtMs
startRecording->>HyprlandCursorRecordingSession: start cursor telemetry at firstFrameAtMs
HyprlandCursorRecordingSession->>HyprlandIPC: query j/cursorpos
HyprlandIPC-->>HyprlandCursorRecordingSession: return cursor x/y
HyprlandCursorRecordingSession-->>startRecording: provide normalized cursor samples
Possibly related issues
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 2 | ❌ 3❌ Failed checks (3 warnings)
✅ Passed checks (2 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Fechando para reabrir com histórico limpo (removendo trailers de co-author). |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 6
🧹 Nitpick comments (1)
src/hooks/useScreenRecorder.ts (1)
41-94: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd unit coverage for
waitForFirstVideoFrame's branches.The function is a good self-contained fix, but there's no accompanying test for its three code paths (rVFC fires, rVFC unsupported, timeout fallback). It's easily testable with fake timers and a mocked
requestVideoFrameCallback.As per coding guidelines, "Add a test for every new behavior in the same package as the code under test."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/hooks/useScreenRecorder.ts` around lines 41 - 94, Add unit tests for waitForFirstVideoFrame covering all three branches: resolving with the requestVideoFrameCallback timestamp path, immediate fallback when requestVideoFrameCallback is unavailable, and Date.now() resolution after FIRST_FRAME_TIMEOUT_MS. Mock video creation and requestVideoFrameCallback, use fake timers, and verify each promise resolves with the expected timing while preserving cleanup behavior.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/superpowers/plans/2026-07-22-hyprland-cursor-position.md`:
- Around line 247-248: Add same-package Vitest coverage for
HyprlandCursorRecordingSession, mocking Electron and IPC dependencies to verify
interval sampling, coordinate normalization, fallback positions, overlap
prevention, retention, timestamps, and stop/start lifecycle behavior. Update
docs/superpowers/plans/2026-07-22-hyprland-cursor-position.md at lines 247-248
to replace the no-test exception with this test strategy; implement the suite
alongside
electron/native-bridge/cursor/recording/hyprlandCursorRecordingSession.ts.
In `@docs/superpowers/specs/2026-07-22-hyprland-cursor-position-design.md`:
- Around line 13-18: Update the design document’s capture and editor-behavior
references to match the implementation: describe the Linux capture path in
useScreenRecorder.ts as requesting cursor "always", and revise the “No
editor/rendering changes are needed” statements to reflect the implemented Linux
editor eligibility work. Apply the same corrections to both referenced sections
while preserving the documented distinction between capture changes and the
original cursor-pixel bug.
In `@electron/native-bridge/cursor/recording/hyprlandCursorIpc.ts`:
- Around line 30-34: The IPC handling around socket.setTimeout in
hyprlandCursorIpc.ts must use an absolute wall-clock deadline instead of relying
solely on inactivity, enforce a maximum response byte limit before appending to
chunks, and finish safely on either limit while preserving existing cleanup
behavior. Add coverage in hyprlandCursorIpc.test.ts for a peer that periodically
writes without closing, plus assertions for oversized responses. Update the
corresponding implementation plan in
docs/superpowers/plans/2026-07-22-hyprland-cursor-position.md to document the
deadline, response limit, and test coverage.
In `@electron/native-bridge/cursor/recording/hyprlandCursorRecordingSession.ts`:
- Around line 41-53: Update the recording session methods around stop() and
captureSample() to track in-flight capture promises and a session
generation/cancellation token. In stop(), invalidate the active generation
before stopping, await any pending capture completion, and return a detached
immutable snapshot of samples so resumed IPC work cannot mutate the returned or
subsequent recording.
In `@src/components/video-editor/VideoEditor.tsx`:
- Around line 326-335: Update the hasEditableCursorRecording eligibility logic
to avoid enabling Linux overlays solely from nativePlatform. Require a stronger
cursor-data signal for Linux, such as a valid provider/quality marker or
non-degenerate samples, while preserving the existing Windows and macOS behavior
and the editable-overlay mode check.
In `@src/hooks/useScreenRecorder.ts`:
- Around line 1250-1318: Update the Linux system-audio branch in the
screen-recording flow around getDisplayMedia and the legacy getUserMedia call so
Linux does not attempt the unsupported chromeMediaSource desktop-audio fallback.
Gate systemAudioEnabled accordingly, preserving video-only capture on Linux
unless real capability detection confirms supported audio capture; do not rely
on the current audioOnlyStream result.
---
Nitpick comments:
In `@src/hooks/useScreenRecorder.ts`:
- Around line 41-94: Add unit tests for waitForFirstVideoFrame covering all
three branches: resolving with the requestVideoFrameCallback timestamp path,
immediate fallback when requestVideoFrameCallback is unavailable, and Date.now()
resolution after FIRST_FRAME_TIMEOUT_MS. Mock video creation and
requestVideoFrameCallback, use fake timers, and verify each promise resolves
with the expected timing while preserving cleanup behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: bc2f135d-b503-4332-be34-0ec0031d786f
📒 Files selected for processing (8)
docs/superpowers/plans/2026-07-22-hyprland-cursor-position.mddocs/superpowers/specs/2026-07-22-hyprland-cursor-position-design.mdelectron/native-bridge/cursor/recording/factory.tselectron/native-bridge/cursor/recording/hyprlandCursorIpc.test.tselectron/native-bridge/cursor/recording/hyprlandCursorIpc.tselectron/native-bridge/cursor/recording/hyprlandCursorRecordingSession.tssrc/components/video-editor/VideoEditor.tsxsrc/hooks/useScreenRecorder.ts
| This class **does** import Electron's `screen` module (for the same `getDisplayNearestPoint` fallback `TelemetryRecordingSession` already uses), which is why — matching the existing convention for this family of session classes (none of `macNativeCursorRecordingSession.ts`, `windowsNativeRecordingSession.ts`, or `telemetryRecordingSession.ts` have unit tests, since Electron's modules can't be imported outside a running Electron process) — this task has no automated test. It's covered by the manual verification in Task 4. | ||
|
|
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟠 Major | 🏗️ Heavy lift
Add same-package coverage for the new recording session. The plan explicitly exempts this core behavior from tests, leaving interval sampling, normalization, fallback positions, overlap prevention, retention, timestamps, and stop/start lifecycle behavior unverified.
docs/superpowers/plans/2026-07-22-hyprland-cursor-position.md#L247-L248: replace the no-test exception with a test strategy using mocked Electron and IPC dependencies.electron/native-bridge/cursor/recording/hyprlandCursorRecordingSession.ts#L19-L91: add a Vitest suite in this package covering the session’s observable behavior.
As per coding guidelines, “Add a test for every new behavior in the same package as the code under test.”
📍 Affects 2 files
docs/superpowers/plans/2026-07-22-hyprland-cursor-position.md#L247-L248(this comment)electron/native-bridge/cursor/recording/hyprlandCursorRecordingSession.ts#L19-L91
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/superpowers/plans/2026-07-22-hyprland-cursor-position.md` around lines
247 - 248, Add same-package Vitest coverage for HyprlandCursorRecordingSession,
mocking Electron and IPC dependencies to verify interval sampling, coordinate
normalization, fallback positions, overlap prevention, retention, timestamps,
and stop/start lifecycle behavior. Update
docs/superpowers/plans/2026-07-22-hyprland-cursor-position.md at lines 247-248
to replace the no-test exception with this test strategy; implement the suite
alongside
electron/native-bridge/cursor/recording/hyprlandCursorRecordingSession.ts.
Source: Coding guidelines
| - **App-level cursor request**: `useScreenRecorder.ts`'s Linux capture path used the legacy | ||
| `chromeMediaSource` constraints, which have no cursor toggle at all (unlike the `cursor` | ||
| constraint available via `getDisplayMedia`). This was fixed (Linux now requests | ||
| `cursor: "always"` via `getDisplayMedia`, mirroring the Windows path) and is a real, | ||
| independent improvement, but it did **not** fix the reported bug — frame extraction from | ||
| real recordings, before and after, showed zero cursor pixels either way. |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Align the design with the implemented capture and editor behavior.
The PR requests cursor: "never" to avoid duplicate cursors and includes Linux editor eligibility work. Update the references to cursor: "always" and “No editor/rendering changes are needed” so this remains an accurate implementation record.
Also applies to: 56-61
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/superpowers/specs/2026-07-22-hyprland-cursor-position-design.md` around
lines 13 - 18, Update the design document’s capture and editor-behavior
references to match the implementation: describe the Linux capture path in
useScreenRecorder.ts as requesting cursor "always", and revise the “No
editor/rendering changes are needed” statements to reflect the implemented Linux
editor eligibility work. Apply the same corrections to both referenced sections
while preserving the documented distinction between capture changes and the
original cursor-pixel bug.
| socket.setTimeout(SOCKET_TIMEOUT_MS, () => finish(null)); | ||
| socket.on("error", () => finish(null)); | ||
| socket.on("connect", () => socket.write("j/cursorpos")); | ||
| socket.on("data", (chunk) => chunks.push(chunk)); | ||
| socket.on("close", () => { |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
Make the IPC deadline absolute and bounded. socket.setTimeout() only fires after inactivity, so a peer that continuously streams data without closing can keep cursor sampling stuck and allows unbounded response buffering. Use a wall-clock deadline plus a max-response byte limit, and add coverage for a non-closing peer that writes data periodically.
📍 Affects 3 files
electron/native-bridge/cursor/recording/hyprlandCursorIpc.ts#L30-L34(this comment)electron/native-bridge/cursor/recording/hyprlandCursorIpc.test.ts#L77-L116docs/superpowers/plans/2026-07-22-hyprland-cursor-position.md#L135-L154
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@electron/native-bridge/cursor/recording/hyprlandCursorIpc.ts` around lines 30
- 34, The IPC handling around socket.setTimeout in hyprlandCursorIpc.ts must use
an absolute wall-clock deadline instead of relying solely on inactivity, enforce
a maximum response byte limit before appending to chunks, and finish safely on
either limit while preserving existing cleanup behavior. Add coverage in
hyprlandCursorIpc.test.ts for a peer that periodically writes without closing,
plus assertions for oversized responses. Update the corresponding implementation
plan in docs/superpowers/plans/2026-07-22-hyprland-cursor-position.md to
document the deadline, response limit, and test coverage.
| async stop(): Promise<CursorRecordingData> { | ||
| if (this.interval) { | ||
| clearInterval(this.interval); | ||
| this.interval = null; | ||
| } | ||
|
|
||
| return { | ||
| version: 2, | ||
| provider: "none", | ||
| samples: this.samples, | ||
| assets: [], | ||
| }; | ||
| } |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Prevent in-flight samples from mutating stopped recordings.
stop() can return while captureSample() is awaiting IPC. When it resumes, it appends to the returned array; a quick subsequent start() can even place that old sample into the next recording. Track an in-flight promise plus a session generation/cancellation token, invalidate it before stopping, and return an immutable snapshot.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@electron/native-bridge/cursor/recording/hyprlandCursorRecordingSession.ts`
around lines 41 - 53, Update the recording session methods around stop() and
captureSample() to track in-flight capture promises and a session
generation/cancellation token. In stop(), invalidate the active generation
before stopping, await any pending capture completion, and return a detached
immutable snapshot of samples so resumed IPC work cannot mutate the returned or
subsequent recording.
| // Windows recordings include captured cursor assets. macOS hides the system | ||
| // cursor in ScreenCaptureKit and renders telemetry samples with OpenScreen's | ||
| // default arrow asset for the editable overlay. | ||
| // default arrow asset for the editable overlay. Linux never captures real | ||
| // cursor assets either (provider is always "none"), same as the macOS | ||
| // telemetry fallback, so it renders the same default arrow asset from | ||
| // position samples alone. | ||
| const hasEditableCursorRecording = | ||
| recordingCursorCaptureMode === "editable-overlay" && | ||
| (nativePlatform === "win32" || nativePlatform === "darwin") && | ||
| (nativePlatform === "win32" || nativePlatform === "darwin" || nativePlatform === "linux") && | ||
| hasNativeCursorRecordingData(cursorRecordingData); |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== files =="
git ls-files | rg '(^|/)VideoEditor\.tsx$|factory\.(ts|tsx)$|.*cursor.*|.*telemetry.*' | sed -n '1,200p'
echo "== VideoEditor outline =="
ast-grep outline src/components/video-editor/VideoEditor.tsx --view expanded 2>/dev/null | sed -n '1,220p' || true
echo "== relevant VideoEditor lines =="
sed -n '300,360p' src/components/video-editor/VideoEditor.tsx
echo "== search cursor/session/factory references =="
rg -n "hasEditableCursorRecording|hasNativeCursorRecordingData|recordingCursorCaptureMode|HyprlandCursorRecordingSession|TelemetryRecordingSession|screen\.getCursorScreenPoint|cursorCapture|cursorRecording" . -S --glob '!node_modules' --glob '!dist' --glob '!build' | sed -n '1,240p'Repository: getopenscreen/openscreen
Length of output: 28654
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== nativeCursor predicate =="
sed -n '230,290p' src/lib/cursor/nativeCursor.ts
echo "== telemetry session =="
cat -n electron/native-bridge/cursor/recording/telemetryRecordingSession.ts | sed -n '1,90p'
echo "== hyprland recording session =="
cat -n electron/native-bridge/cursor/recording/hyprlandCursorRecordingSession.ts | sed -n '1,80p'
echo "== useScreenRecorder comments =="
sed -n '1240,1275p' src/hooks/useScreenRecorder.ts
echo "== nativeCursor tests around predicate =="
sed -n '1,90p' src/lib/cursor/nativeCursor.test.ts
echo "== factory spec comments =="
sed -n '30,56p' electron/native-bridge/cursor/recording/factory.ts
echo "== static check: metadata source/provider markers around recording data creation =="
rg -n "cursorRecordingData|provider|sessionType|cursorCaptureMode|data:|stop\\(\\)" src lib electron/native-bridge/cursor/recording electron/ipc/handlers.ts -S | sed -n '1,260p'Repository: getopenscreen/openscreen
Length of output: 33070
Don’t gate Linux cursor eligibility on platform alone.
Non-Hyprland Linux Wayland compositors still use TelemetryRecordingSession, which records position samples even though screen.getCursorScreenPoint() is broken there. Once those samples exist, hasNativeCursorRecordingData() passes because provider === "none" and samples.length > 0, so this blanket nativePlatform === "linux" check lets stale/frozen cursor overlays into the editor. Gate on a stronger signal than platform, such as a cursor provider/quality marker or a degenerate-sample check, before enabling the overlay.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/components/video-editor/VideoEditor.tsx` around lines 326 - 335, Update
the hasEditableCursorRecording eligibility logic to avoid enabling Linux
overlays solely from nativePlatform. Require a stronger cursor-data signal for
Linux, such as a valid provider/quality marker or non-degenerate samples, while
preserving the existing Windows and macOS behavior and the editable-overlay mode
check.
| if (platform === "win32" || platform === "linux") { | ||
| // getDisplayMedia + setDisplayMediaRequestHandler (main.ts) supplies the | ||
| // pre-selected source. Editable cursor mode excludes the system cursor so | ||
| // the editor can render a replacement; system mode bakes it into the video. | ||
| // pre-selected source. On Windows, editable cursor mode excludes the system | ||
| // cursor so the editor can render a replacement; system mode bakes it into | ||
| // the video. On Linux, the editor's cursor overlay (see | ||
| // src/lib/cursor/nativeCursor.ts's provider:"none" handling) always runs — | ||
| // createCursorRecordingSession's Linux sessions (HyprlandCursorRecordingSession | ||
| // on Hyprland, TelemetryRecordingSession elsewhere) always produce position | ||
| // telemetry the overlay draws a generic cursor from. So Linux always requests | ||
| // "never" here too: on setups where the system cursor *does* end up baked into | ||
| // the raw frames (e.g. X11, or a Wayland compositor whose portal supports | ||
| // Embedded cursor mode), requesting "always" would double it up with the | ||
| // overlay. On Hyprland specifically this constraint has no effect either way | ||
| // (the portal never bakes the cursor in regardless of what's requested), so the | ||
| // overlay remains the only cursor source there. | ||
| const cursor: "always" | "never" = | ||
| platform === "linux" || cursorCaptureMode === "editable-overlay" ? "never" : "always"; | ||
|
|
||
| if (platform === "linux" && systemAudioEnabled) { | ||
| // Electron's getDisplayMedia loopback audio (main.ts's | ||
| // setDisplayMediaRequestHandler) only supports win32/macOS, so system | ||
| // audio on Linux still has to go through the legacy desktop-audio | ||
| // constraints and get merged with the getDisplayMedia video track. | ||
| const [videoOnlyStream, audioOnlyStream] = await Promise.all([ | ||
| navigator.mediaDevices.getDisplayMedia({ | ||
| video: { | ||
| cursor, | ||
| width: { max: TARGET_WIDTH }, | ||
| height: { max: TARGET_HEIGHT }, | ||
| frameRate: { ideal: TARGET_FRAME_RATE }, | ||
| } as MediaTrackConstraints, | ||
| audio: false, | ||
| } as DisplayMediaStreamOptions), | ||
| navigator.mediaDevices | ||
| .getUserMedia({ | ||
| audio: { | ||
| mandatory: { | ||
| chromeMediaSource: CHROME_MEDIA_SOURCE, | ||
| chromeMediaSourceId: selectedSource.id, | ||
| }, | ||
| }, | ||
| video: false, | ||
| } as unknown as MediaStreamConstraints) | ||
| .catch((audioErr) => { | ||
| console.warn( | ||
| "System audio capture failed, falling back to video-only:", | ||
| audioErr, | ||
| ); | ||
| toast.error(t("recording.systemAudioUnavailable")); | ||
| return null; | ||
| }), | ||
| ]); | ||
|
|
||
| const combined = new MediaStream(); | ||
| for (const track of videoOnlyStream.getVideoTracks()) combined.addTrack(track); | ||
| if (audioOnlyStream) { | ||
| for (const track of audioOnlyStream.getAudioTracks()) combined.addTrack(track); | ||
| } | ||
| return combined; | ||
| } | ||
|
|
||
| return navigator.mediaDevices.getDisplayMedia({ | ||
| video: { | ||
| cursor: cursorCaptureMode === "editable-overlay" ? "never" : "always", | ||
| cursor, | ||
| width: { max: TARGET_WIDTH }, | ||
| height: { max: TARGET_HEIGHT }, | ||
| frameRate: { ideal: TARGET_FRAME_RATE }, | ||
| } as MediaTrackConstraints, | ||
| audio: systemAudioEnabled, | ||
| audio: platform === "win32" ? systemAudioEnabled : false, |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== files =="
git ls-files | rg '(^|/)useScreenRecorder\.ts$|(^|/)main\.(ts|js)$|package\.json$|electron' || true
echo "== outline hook =="
if [ -f src/hooks/useScreenRecorder.ts ]; then
wc -l src/hooks/useScreenRecorder.ts
ast-grep outline src/hooks/useScreenRecorder.ts --view condensed | sed -n '1,220p' || true
echo "== relevant lines =="
sed -n '1200,1345p' src/hooks/useScreenRecorder.ts | cat -n
fi
echo "== search chromeMediaSource usages and docs refs =="
rg -n 'chromeMediaSource|getDisplayMedia|systemAudioEnabled|createCursorRecordingSession|Desktop|desktop audio|audio capture' -S . --glob '*.{ts,tsx,js,jsx,json,md}' | sed -n '1,240p'Repository: getopenscreen/openscreen
Length of output: 17129
🌐 Web query:
Electron getDisplayMedia desktop audio chromeMediaSource Linux macOS Windows unsupported
💡 Result:
In Electron, capturing desktop audio using navigator.mediaDevices.getUserMedia with chromeMediaSource: 'desktop' is a complex, platform-dependent feature that often faces issues such as NotSupportedError, DOMException: Could not start audio source, or renderer process crashes [1][2][3][4]. Platform status: macOS: - Support exists for capturing system audio on macOS 13 (Ventura) and later using the CoreAudio Tap API, which Chromium/Electron now utilizes [5][6][7]. - Requirements: You must include the NSAudioCaptureUsageDescription key in your application's Info.plist file [8][6][7]. Failing to do so—or running on older versions of macOS (12.7.6 and prior)—will result in failure to capture audio [8][7]. On versions prior to macOS 13, system audio capture was fundamentally limited and typically required third-party virtual audio drivers like BlackHole or Soundflower [8][7]. Windows: - Desktop audio capture via getUserMedia with chromeMediaSource: 'desktop' is highly unstable and prone to crashing the renderer process (often reported as Error 263/Bad IPC) or throwing DOMException errors [1][3]. - Issues are frequently linked to specific hardware drivers (e.g., certain USB audio devices) or software permission conflicts [2][4]. There is currently no guaranteed, stable native method within the desktopCapturer API to capture system audio on Windows; developers often encounter terminal errors when attempting to use the standard constraints [1][3]. Linux: - Support depends heavily on the underlying desktop environment and audio subsystem. Electron's desktopCapturer.getSources is known to have limitations when using PipeWire, often returning only a single source regardless of requested types [8]. General Recommendations: - The chromeMediaSource constraint approach is considered a legacy mechanism [5][6]. - For robust cross-platform desktop audio capture, many developers rely on third-party native modules or OS-level virtual audio device drivers (e.g., BlackHole on macOS, or virtual cables on Windows) that present as standard microphone inputs, which getUserMedia can then capture without requiring chromeMediaSource hacks [8][7]. - Always check the official Electron desktopCapturer documentation for the latest specific constraints and required Info.plist permissions, as these are frequently updated to track upstream Chromium changes [8][9][6].
Citations:
- 1: [Bug]: getUserMedia to get system-wide or window audio capture on Windows errors or terminates renderer electron/electron#42765
- 2: [Bug]: DesktopCapturer API cannot start audio source electron/electron#41697
- 3: Renderer Crashes (Error 263) using getUserMedia with chromeMediaSourceId for Desktop Audio Capture on Windows 11 with bad ipc electron/electron#46369
- 4: desktopCapturer not capturing audio when USB speakers are Windows output electron/electron#15829
- 5: [Bug]: Broken Desktop Audio Capture electron/electron#49607
- 6: fix: address upstream Chromium shift to enable
CoreAudio Tap APIfor audio capture used in electron'sdesktopCapturer(🍏 macOS). electron/electron#49717 - 7: https://github.com/electron/electron/blob/v41.2.0/docs/api/desktop-capturer.md
- 8: https://github.com/electron/electron/blob/main/docs/api/desktop-capturer.md
- 9: https://electronjs.org/docs/latest/api/desktop-capturer
🌐 Web query:
Chromium getUserMedia chromeMediaSource desktop Linux audio not listed monitor unsupported
💡 Result:
In Chromium-based browsers on Linux, using the legacy getUserMedia with chromeMediaSource: 'desktop' for audio is generally unsupported and often results in a "Could not start audio source" error [1][2][3]. Chromium historically restricts access to "monitor" or system audio inputs on Linux for design and user-experience reasons, as these are not standard microphone inputs [4]. To capture system audio on Linux in modern Chromium environments, the recommended approach is to use the getDisplayMedia API instead of getUserMedia [5]. When using getDisplayMedia, you can request system audio by setting the audio constraint to true or { systemAudio: 'include' } [5][6]. If you are encountering issues with getDisplayMedia not capturing audio on Linux, consider the following points: 1. Use getDisplayMedia: Switch from getUserMedia to navigator.mediaDevices.getDisplayMedia({ video: true, audio: true }) [5][7]. 2. Browser UI Prompt: When the browser's screen-sharing picker appears, the user must explicitly check the "Share system audio" or "Share tab audio" checkbox [5][6]. If this is not checked, the resulting MediaStream will not contain an audio track [5]. 3. Experimental Flags: In some Linux configurations, you may need to ensure the browser is configured to support loopback audio. For instance, launching Chromium with the flag --enable-features=PulseaudioLoopbackForScreenShare has been noted in community solutions to facilitate system audio capture via getDisplayMedia [6]. 4. Alternative Workarounds: If direct browser capture remains insufficient, developers often use OS-level tools like PulseAudio loopback modules (pactl load-module module-remap-source) to create a "Virtual Microphone" that maps the system audio monitor to a standard input device, which the browser can then capture as a regular microphone [8][3]. The limitation you are seeing is an intentional design choice in the Chromium engine regarding how it exposes non-microphone audio sources, and it is not a bug that can be fixed via standard getUserMedia constraints [4]. Switching to getDisplayMedia is the modern, compliant way to handle this requirement [5][6].
Citations:
- 1: Desktop audio capture does not work on Mac and Linux electron/electron#10515
- 2: https://stackoverflow.com/questions/62317022/cannot-capture-system-audio-output-with-electron-desktopcapturer-in-ubuntu
- 3: [Bug]: Could not start audio source error on mediaDevices.getUserMedia electron/electron#32299
- 4: Implementers must not refuse to open sources set as default at the machine: "DOMException: Could not start audio source" is not in the specification w3c/mediacapture-main#708
- 5: https://dev.to/flo152121063061/i-tried-to-capture-system-audio-in-the-browser-heres-what-i-learned-1f99
- 6: https://gist.github.com/guest271314/baaa0b8d4b034ff4e9352af4f2bbf42c
- 7: OpenWhispr/openwhispr@ae14ab9
- 8: https://github.com/guest271314/captureSystemAudio/blob/master/README.md
🌐 Web query:
Electron setDisplayMediaRequestHandler window.webContents.setDisplayMediaRequestHandler Linux audio capture
💡 Result:
To capture system audio on Linux in Electron using setDisplayMediaRequestHandler, you must configure the audio property of the callback object to loopback or loopbackWithMute [1][2]. When implementing this, use the following pattern within your main process: session.defaultSession.setDisplayMediaRequestHandler(async (request, callback) => { // Option 1: Capture both video and system audio const sources = await desktopCapturer.getSources({ types: ['screen'] }); callback({ video: sources, audio: 'loopback' }); // Option 2: Capture audio only (if video is not required) // callback({ audio: 'loopback' }); }); Technical Considerations and Known Issues: 1. System Audio Loopback Support: Setting audio to 'loopback' or 'loopbackWithMute' is explicitly supported for Linux (and Windows) [1]. On modern Linux distributions using PipeWire and Wayland, this functionality is generally functional [3]. 2. DesktopCapturer Integration: Historically, it was common to call desktopCapturer.getSources inside the handler to provide a video source [1][4]. However, if your application only requires audio, you may be able to omit the video source, which can prevent unnecessary "screen share" permission dialogs triggered by desktopCapturer on some Linux environments [3]. 3. Platform-Specific Behavior: - PipeWire/Wayland: Applications may encounter issues with persistent permissions or repeated portal dialogs if desktopCapturer is called frequently [3]. - Reliability: Users have reported intermittent issues with audio capture in certain Electron versions, often linked to upstream Chromium changes or interaction with other media processes [5][6]. If audio capture fails, ensure that no other processes have locked the audio devices and check for known regressions in the specific Electron version you are using [5][6]. - API Limitations: The setDisplayMediaRequestHandler API is strictly for responding to navigator.mediaDevices.getDisplayMedia requests [1]. Older methods of using getUserMedia with chromeMediaSource for system audio are deprecated or non-functional on Linux [6]. 4. Troubleshooting: If the stream contains no audio, verify that your Linux distribution's audio backend (PulseAudio or PipeWire) is correctly routing system audio to the capture stream [6][3]. Persistent silence may indicate a regression in the underlying Chromium media pipeline, which should be checked against the official Electron issue tracker [5].
Citations:
- 1: feat: session.setDisplayMediaRequestHandler electron/electron#30702
- 2: https://github.com/atom/electron/blob/master/docs/api/desktop-capturer.md
- 3: fix(linux): remove unnecessary screen capture from audio loopback handler jeffvli/feishin#1926
- 4: https://www.todesktop.com/builder/docs/application/capture-screen-contents
- 5: [Bug]: Broken Desktop Audio Capture electron/electron#49607
- 6: https://stackoverflow.com/questions/62317022/cannot-capture-system-audio-output-with-electron-desktopcapturer-in-ubuntu
Gate Linux system audio out of this legacy getUserMedia fallback.
This branch keeps getDisplayMedia audio as false on Linux and uses getUserMedia({ mandatory: { chromeMediaSource: "desktop", chromeMediaSourceId } }) for system audio. That legacy constraint is unsupported/non-functional on Linux and normally fails with “No matching audio source” or does not receive monitor audio by default; even if it resolves, the code does not inspect audioOnlyStream.getAudioTracks(), so systemAudioEnabled recordings would still fall silently onto the video-only stream. Disable the audio request as unsupported on Linux, or add real capability detection before enabling systemAudioEnabled there.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/hooks/useScreenRecorder.ts` around lines 1250 - 1318, Update the Linux
system-audio branch in the screen-recording flow around getDisplayMedia and the
legacy getUserMedia call so Linux does not attempt the unsupported
chromeMediaSource desktop-audio fallback. Gate systemAudioEnabled accordingly,
preserving video-only capture on Linux unless real capability detection confirms
supported audio capture; do not rely on the current audioOnlyStream result.
1 participant
Summary
cursor:neverfrom the portal and relying on the editor's editable overlay on Linux.recordingId, and cursor telemetry start time) to the first real captured frame instead of toDate.now()at button-press time. On Linux/PipeWire, the desktop-capture portal can still be negotiating (DMA-BUF modifier renegotiation) for several seconds after the track reports unmuted; anchoring to wall-clock "button pressed" time let the declared recording duration exceed the video's real content span (playback would jump to the end early) and let the cursor overlay drift out of sync with the picture.waitForFirstVideoFrame()now blocks on an actual rendered frame (viarequestVideoFrameCallback, capped at 3s) beforerecorder.start()is called, so everything anchors to the same verified instant.Fixes #99 (comment)
Test plan
tsc --noEmitpasses cleanSummary by CodeRabbit
New Features
Bug Fixes