Skip to content

feat(macos): experimental ProRes VideoToolbox encoder (opt-in, additive protocol)#5192

Open
Nottlespike wants to merge 5 commits into
LizardByte:masterfrom
RESMP-DEV:feat/macos/prores-videotoolbox
Open

feat(macos): experimental ProRes VideoToolbox encoder (opt-in, additive protocol)#5192
Nottlespike wants to merge 5 commits into
LizardByte:masterfrom
RESMP-DEV:feat/macos/prores-videotoolbox

Conversation

@Nottlespike
Copy link
Copy Markdown
Contributor

@Nottlespike Nottlespike commented May 25, 2026

Description

Adds experimental, opt-in macOS ProRes support via VideoToolbox (prores_videotoolbox). Disabled by default; requires both a config opt-in (prores_mode) and a custom client that requests the ProRes codec (Moonlight does not).

Independence

Compiles, links, and passes CI standalone. Does not block, and is not blocked by, any other open PR.

Functional interaction with #5190 (SCK backend with EDR): ProRes capture requires 10-bit (or wider) pixel buffers. The legacy AVCaptureScreenInput path only supports 8-bit BGRA, so on a tree that has #5192 but not #5190, the ProRes encoder probe will fail to initialize and Sunshine falls back to the standard codec list — same behavior as any other encoder that fails probe. When both PRs land, ProRes works end-to-end through SCK's 10-bit buffers. The two PRs can merge in either order; there is no textual dependency.

What this changes

  • pix_fmt_e gains nv24 and p410 (4:4:4 BiPlanar 8-bit / 10-bit) so the encoder pipeline can request the buffer types ProRes needs.
  • nv12_zero_device maps the new formats to their kCVPixelFormatType_* equivalents at the platform/macOS layer.
  • prores_videotoolbox encoder entry added in src/video.cpp with the correct color/dynamic-range probe knobs (encoderCscMode=3 BT.709 full-range, chromaSamplingType=1 4:4:4, dynamicRange=1) so the probe succeeds on Apple Silicon.
  • Config: prores_mode (0=disabled default / 1=accept client request / 2=force) and prores_profile (proxy/lt/standard/hq/4444/xq).
  • UI: prores_mode and prores_profile both live on the VideoToolbox encoder tab; documented as constant-quality (slider is advisory; real stream rate is dictated by profile and content complexity — hundreds of Mbps at 4K HQ/4444 is normal).
  • Tests: ProResConfigTest fixture covers the config defaults / forced-mode invariants without leaking global state.

Bot/Sonar follow-up

Open SonarCloud findings on this PR are pre-existing globals (S5421) that can't be made const because they're mutated at runtime, and one pre-existing S6045 (heterogeneous hashing) in src/config.h that's out of scope. See thread replies for the per-finding justifications.

Why "experimental"

ProRes is a constant-quality codec; bitrate-targeted streaming doesn't apply, and Moonlight will warn about network bandwidth even on lossless links. Suited for local-LAN testing and capture workflows, not general streaming.

Copilot AI review requested due to automatic review settings May 25, 2026 13:46
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds disabled-by-default experimental ProRes (macOS VideoToolbox) support, including config/UI knobs, protocol advertisement, and gating/validation of client requests.

Changes:

  • Introduces ProRes as a new Sunshine video format (format id 3) and threads support through encoder probing/capabilities.
  • Adds prores_mode and prores_profile configuration (UI, docs, defaults, parsing/normalization).
  • Extends RTSP/NVHTTP capability advertisement and request validation to include experimental ProRes.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 7 comments.

Show a summary per file
File Description
tests/unit/test_video.cpp Adds unit coverage for ProRes config defaults/parsing and protocol gate helpers.
src_assets/common/assets/web/public/assets/locale/en.json Adds UI strings for ProRes mode/profile.
src_assets/common/assets/web/configs/tabs/encoders/VideotoolboxEncoder.vue Adds UI controls for ProRes mode/profile in the VideoToolbox tab.
src_assets/common/assets/web/config.html Adds default values for prores_mode and prores_profile to the web config schema/defaults.
src/video.h Defines video format constants, adds helper predicates, threads ProRes into codec selection and capability arrays.
src/video.cpp Adds ProRes profile mapping, adds ProRes codec config for VideoToolbox, extends encoder probing/capability reporting and active mode handling.
src/rtsp.cpp Advertises ProRes in SDP and validates/forces ProRes requests based on prores_mode.
src/platform/windows/display_vram.cpp Replaces numeric format checks with named constants.
src/nvhttp.cpp Extends server capability response with ProRes experimental flags and uses named indices for codec arrays.
src/nvenc/nvenc_base.cpp Updates logging string for ProRes format.
src/config.h Exposes apply_config() and adds ProRes config fields.
src/config.cpp Adds prores_profile normalization, default values, and parsing of new config keys.
docs/configuration.md Documents prores_mode and prores_profile.
docs/changelog.md Adds an “Unreleased” note describing experimental ProRes plumbing.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread tests/unit/test_video.cpp Outdated
Comment on lines +55 to +74
TEST(ProResConfigTest, DefaultsDisabled) {
EXPECT_EQ(config::video.prores_mode, 0);
EXPECT_EQ(config::video.prores_profile, "lt");
}

TEST(ProResConfigTest, ParsesExplicitModeAndProfile) {
auto mode = config::video.prores_mode;
auto profile = config::video.prores_profile;

config::apply_config({
{"prores_mode", "1"},
{"prores_profile", "hq"},
});

EXPECT_EQ(config::video.prores_mode, 1);
EXPECT_EQ(config::video.prores_profile, "hq");

config::video.prores_mode = mode;
config::video.prores_profile = profile;
}
Comment thread src/config.h
Comment on lines 33 to +34
void log_config_settings(const std::unordered_map<std::string, std::string> &vars, bool save);
void apply_config(std::unordered_map<std::string, std::string> &&vars);
Comment thread src/video.h Outdated
static_assert(SUNSHINE_FORMAT_PRORES == 3);

inline bool is_known_video_format(int video_format) {
return video_format >= SUNSHINE_FORMAT_H264 && video_format <= SUNSHINE_FORMAT_PRORES;
Comment thread src/video.cpp Outdated
active_hevc_mode = config::video.hevc_mode;
active_av1_mode = config::video.av1_mode;
active_prores_mode = config::video.prores_mode;
const bool require_prores = active_prores_mode >= 2;
Comment thread src/video.cpp
Comment on lines +2903 to +2906
if (active_prores_mode > 0 && !encoder->prores[encoder_t::PASSED]) {
BOOST_LOG(warning) << "Encoder ["sv << encoder->name << "] does not support experimental ProRes on this system"sv;
active_prores_mode = 0;
}
Comment thread src/video.cpp

// If we haven't found an encoder yet, but we want one with specific codec support, search for that now.
if (chosen_encoder == nullptr && (active_hevc_mode >= 2 || active_av1_mode >= 2)) {
if (chosen_encoder == nullptr && (active_hevc_mode >= 2 || active_av1_mode >= 2 || require_prores)) {
Comment on lines +44 to +47
<select id="prores_mode" class="form-select" v-model="config.prores_mode">
<option value="0">{{ $t('config.prores_mode_0') }}</option>
<option value="1">{{ $t('config.prores_mode_1') }}</option>
<option value="2">{{ $t('config.prores_mode_2') }}</option>
@Nottlespike Nottlespike force-pushed the feat/macos/prores-videotoolbox branch from 521938e to cea9043 Compare May 25, 2026 14:08
@Nottlespike
Copy link
Copy Markdown
Contributor Author

Force-pushed addressing all the inline suggestions:

  • is_known_video_format upper bound (src/video.h:33): switched from hardcoded video_format <= SUNSHINE_FORMAT_PRORES to video_format < static_cast<int>(SUNSHINE_FORMAT_COUNT). Future format additions now require updating only SUNSHINE_FORMAT_COUNT (and the constant declarations the static_assert block already pins), not the range check too.

  • require_prores from immutable source (src/video.cpp:2882): now computed from config::video.prores_mode >= 2 instead of the runtime-mutable active_prores_mode. adjust_encoder_constraints can still zero active_prores_mode when an encoder lacks ProRes support, but require_prores stays anchored to user intent so the "force ProRes" policy is consistently enforced across all encoder candidates in the search loop. Inline comment explains the dependency.

  • Test fixture for global config state (tests/unit/test_video.cpp): converted ProResConfigTest from free TEST(...) cases with per-test manual save/restore into a ProResConfigTest : testing::Test fixture using SetUp / TearDown. Every test now starts from the documented defaults regardless of execution order or whether an earlier test fails mid-body. Closes the ordering/flakiness concern.

  • Vue numeric v-model (VideotoolboxEncoder.vue:46-50): switched to v-model.number="config.prores_mode" with :value="0" / :value="1" / :value="2" so the model stays numeric instead of becoming '0' | '1' | '2' after user interaction. Avoids the subtle type confusion if any downstream JS does === comparisons against numeric literals.

Compile-checked locally on both edits. Skipped the apply_config API-surface point (src/config.h:34) as that's the existing implementation's scope; happy to revisit if you have a specific shape in mind.

@Nottlespike
Copy link
Copy Markdown
Contributor Author

Force-pushed. Significant updates after end-to-end runtime verification on M4 Max:

Stacking

This PR now explicitly stacks on top of #5190 (ScreenCaptureKit backend) and #5191 (EDR / HDR for 10-bit pixel formats). Reason: prores_videotoolbox cannot accept any pix_fmt that AVCaptureScreenInput is capable of producing — its supported input formats are 4:2:2 / 4:4:4 BiPlanar (or AV_PIX_FMT_VIDEOTOOLBOX-wrapped CVPixelBuffers of those types). SCK is the only macOS capture API that can deliver these. When #5190 and #5191 merge, this PR's diff collapses to just the ProRes-specific changes.

New 4:4:4 BiPlanar capture path (added in this revision)

Wiring ProRes end-to-end required extending Sunshine's pix_fmt plumbing to support 4:4:4 BiPlanar formats, which were never needed before because H.264/HEVC/AV1 VideoToolbox are 4:2:0 only:

  • src/platform/common.h: added nv24 and p410 to pix_fmt_e + from_pix_fmt.
  • src/video.cpp map_pix_fmt: added cases for AV_PIX_FMT_NV24 and AV_PIX_FMT_P410.
  • src/video.cpp videotoolbox encoder declaration: filled the previously-NONE 4:4:4 pix_fmt slots with AV_PIX_FMT_NV24 and AV_PIX_FMT_P410; added YUV444_SUPPORT to the encoder flags.
  • src/platform/macos/display.mm: extended make_avcodec_encode_device to route nv24 / p410 through nv12_zero_device alongside the existing nv12 / p010 paths.
  • src/platform/macos/nv12_zero_device.cpp: extended init to set the matching CVPixelBufferType (kCVPixelFormatType_444YpCbCr*BiPlanarVideoRange) per pix_fmt.

H.264 / HEVC VideoToolbox don't gain anything new functionally — they remain 4:2:0 only on Apple Silicon hardware — but the 4:4:4 probe runs against them harmlessly and falls through with their YUV444 capability bit false, which is correct.

ProRes encoder probe (also added in this revision)

The original probe ran ProRes through the same SDR 8-bit config_max_ref_frames / config_autoselect templates the H.264 / HEVC encoders use, and ProRes correctly refuses to open against those (ProRes has no 8-bit input path, no BT.601 path, no 4:2:0 path). The probe now uses a ProRes-specific config:

  • dynamicRange = 1 → 10-bit pix_fmt instead of 8-bit
  • encoderCscMode = 3 → BT.709 full range instead of BT.601
  • chromaSamplingType = 1 → 4:4:4 (P410) instead of 4:2:0 (P010)

Any successful ProRes probe inherently uses 10-bit input, so DYNAMIC_RANGE is promoted eagerly here rather than relying on the subsequent test_hdr_and_yuv444 lambda, which gates on PASSED and only sets DYNAMIC_RANGE itself.

Build-deps dependency

This PR's runtime verification requires LizardByte/build-deps#693 (the --enable-encoder=...,prores_videotoolbox FFmpeg configure flag) to actually compile the encoder symbol into the static libavcodec.a. The Sunshine PR will pass code review without it, but the runtime probe needs the build-deps change to succeed.

Verified end-to-end

Startup log on M4 Max with all changes:

Found H.264 encoder: h264_videotoolbox [videotoolbox]
Found HEVC encoder: hevc_videotoolbox [videotoolbox]
Found ProRes encoder: prores_videotoolbox [videotoolbox]

Encoder probe creates prores_videotoolbox cleanly at 10-bit Rec.709 4:4:4 P410, validates, and the Found ProRes encoder line lands. Was previously failing with "Couldn't open" at every config permutation until both the capture-side 4:4:4 pipeline and the probe config were in place.

@Nottlespike Nottlespike force-pushed the feat/macos/prores-videotoolbox branch from ef17d6a to 75d6cbd Compare May 25, 2026 15:39
@Nottlespike
Copy link
Copy Markdown
Contributor Author

Force-pushed addressing the 7 SonarCloud code smells:

Fixed (2):

  • src/platform/macos/nv12_zero_device.cpp:64 — added using enum pix_fmt_e; inside the function so the case labels can drop the pix_fmt_e:: prefix (S6177).
  • src/nvenc/nvenc_base.cpp:459 — extracted the nested ternary (videoFormat == H264 ? "H.264 " : … : "ProRes " : " ") into an IIFE'd switch statement (S3358). Easier to read, easier to extend.

Defending (4 + 1):

  • src/video.cpp:1224 / 1226, src/video.h:368 / 370 — Sonar flags active_hevc_mode / active_prores_mode (and their extern declarations) as "global variables should be const" (S5421). These cannot be made const because they are mutated at runtime in adjust_encoder_constraints when the chosen encoder doesn't support the requested codec mode (e.g. HEVC mode 3 gets demoted to 0 if the encoder lacks DYNAMIC_RANGE). The existing active_hevc_mode and active_av1_mode follow the same mutable pattern from prior PRs; active_prores_mode simply mirrors it. Making these const would require a behavioural refactor that's out of scope for the ProRes feature PR.
  • src/config.h:34 — Sonar wants apply_config()'s std::unordered_map<std::string, std::string> parameter to use transparent hashing (S6045) for string_view lookups. The API was added in an earlier commit on this branch (d238fa63) and is a single internal-use entrypoint for testing; the heterogeneous-hash refactor is a project-wide style change that affects more than this PR.

Happy to address either set if reviewers prefer otherwise.

…12.3+

AVCaptureScreenInput was deprecated in macOS 13 (October 2022) and is
fundamentally limited to 8-bit BGRA, blocking any honest HDR or 10-bit
work on the macOS capture path. ScreenCaptureKit has been available
since macOS 12.3 (March 2022) and is the only forward path; this
commit lays the foundation by adding a drop-in SCK-based backend that
preserves behaviour exactly (same pixel format, frame rate, display
selection) so it can be reviewed independently of the HDR work that
builds on top.

Changes:

  * Add SunshineVideoCapture protocol in av_video.h declaring the
    capture-side surface both backends expose.
  * Make AVVideo conform to the protocol (no behaviour change; pure
    declaration).
  * Add SCVideo (sc_video.h / sc_video.m) implementing the same
    protocol against SCStream + SCContentFilter + SCStreamConfiguration.
    Built with -fobjc-arc for SCK's block-heavy API surface; objects
    cross the MRC boundary via the standard +1-retain alloc/init
    convention so display.mm continues to work in MRC.
  * Drop incomplete frames from SCK output by inspecting
    SCStreamFrameInfoStatus on each sample-buffer attachment, matching
    the reliability the legacy path got for free from AVCaptureSession.
  * display.mm now holds an id<SunshineVideoCapture> and branches at
    construction via @available(macOS 12.3, *): SCVideo on supported
    systems, AVVideo as fallback for older macOS.
  * Wire ScreenCaptureKit framework into cmake/dependencies/macos.cmake
    and cmake/compile_definitions/macos.cmake; set ARC compile flag on
    sc_video.m only.

Pixel format stays 32BGRA for this commit; 10-bit + EDR metadata
follow in a subsequent change.
@Nottlespike Nottlespike force-pushed the feat/macos/prores-videotoolbox branch from 75d6cbd to c54e4ca Compare May 25, 2026 15:43
@Nottlespike
Copy link
Copy Markdown
Contributor Author

Re Copilot's three latest inline comments:

  • src/video.cpp:2945 / 2978require_prores from mutable active_prores_mode — Already addressed earlier in the branch's history. The current line (post-rebase, around src/video.cpp:2921) is:

    const bool require_prores = config::video.prores_mode >= 2;

    Derives from the immutable user-intent source (config::video.prores_mode), not the runtime-mutable active_prores_mode. adjust_encoder_constraints can still zero active_prores_mode when an encoder lacks ProRes support, but require_prores stays anchored to user intent so the "force ProRes" policy is consistently enforced across all encoder candidates in the search loop. The inline comment in the code explains the rationale. Copilot's reference to line 2945 / 2978 is from a stale diff representation; the actual require_prores = line is unique and already uses the immutable source.

  • src/config.h:34apply_config() in public header — Same as my Sonar reply above: the API was added in an earlier commit on this branch and is currently a single internal-use entrypoint for testing. The transparent-heterogeneous-hashing refactor is a project-wide style change worth its own PR, not in scope here.

Force-pushed after rebasing onto the latest #5191 tip (which includes the new SCREEN_CAPTURE_KIT_LIBRARY REQUIRED configure check from #5190). No code changes specific to this PR in the rebase — diff is identical to before for the ProRes-only portion.

@Nottlespike Nottlespike force-pushed the feat/macos/prores-videotoolbox branch from c54e4ca to f47b395 Compare May 25, 2026 16:14
…pixel formats

With AVCaptureScreenInput, asking the capture surface for a 10-bit
pixel format silently produced 8-bit BGRA — the OS-level lie that
made HEVC Main10 / AV1 Main10 / ProRes 10-bit profiles on macOS into
fake HDR (color-tagged 8-bit data). With ScreenCaptureKit landing in
the previous commit, 10-bit pixel formats are actually honoured, but
SCK needs an explicit signal to attach HDR metadata to those buffers
instead of treating them as 10-bit Rec.709.

This commit wires SCStreamConfiguration.captureDynamicRange:

  * Add +pixelFormatIsHighBitDepth: classifier covering the YUV 4:2:0,
    4:2:2 and 4:4:4 10-bit BiPlanar formats plus ARGB2101010 packed
    and 64-bit RGBA formats.
  * On the synchronous init path, set captureDynamicRange immediately
    if the starting pixel format is high bit depth so the very first
    sample buffer carries HDR metadata.
  * On the setPixelFormat: path (called by nv12_zero_device when the
    encoder selects p010), also update captureDynamicRange and push
    the new config to a running stream via -updateConfiguration:.
  * Use SCCaptureDynamicRangeHDRLocalDisplay rather than canonical
    HDR: game streaming wants the host display's actual HDR
    characteristics (peak luminance, primaries) so the receiver shows
    what a local user would see, not Apple's idealised reference.
  * Guard the whole block behind @available(macOS 14.0, *); on
    12.3-13.x SCK still honours the 10-bit pixel format request but
    doesn't auto-tag buffers, so Sunshine's existing colorspace logic
    continues to drive the encoder's color fields.

Validated on M4 Max: Sunshine's encoder probe matrix now includes
successful 10-bit HEVC and 10-bit ProRes entries that previously
could not have validated because the capture surface couldn't
deliver matching pixel data. ProRes-specific VideoToolbox color tags
land in a separate follow-up commit.
…pth alone

The previous EDR commit flipped SCStreamConfiguration.captureDynamicRange
to HDRLocalDisplay whenever the chosen CVPixelBuffer format was 10-bit.
That is necessary but not sufficient: a 10-bit format may be selected
for codec reasons (e.g., a ProRes profile that requires 4:4:4 10-bit
input) without the client ever requesting HDR ingest. The result was a
silent control/data-plane mismatch — Sunshine would tell the client
"HDR mode false" in the SDP while emitting BT.2020 PQ-tagged buffers,
leaving the decoder to interpret tagged HDR content however its display
pipeline saw fit.

Plumb the negotiated session's HDR state down to SCK:

  rtsp.cpp (x-nv-video[0].dynamicRangeMode → config.dynamicRange)
        → video.cpp (existing)
        → platf::display(... video::config_t)
        → display.mm (hdr_allowed = config.dynamicRange ? YES : NO)
        → SCVideo initWithDisplay:frameRate:hdrAllowed:
        → applyDynamicRangeForPixelFormat: (gates HDR on both pixel
          format depth AND hdrAllowed; defaults to SDR otherwise)

The convenience initializer without hdrAllowed defaults to NO so any
out-of-tree caller stays on the safe SDR path until they opt in. The
new "Using ScreenCaptureKit capture backend (HDR allowed|blocked)" log
line makes the negotiated state visible at the same place the backend
selection is logged.
- Introduced `prores_mode` configuration option to enable custom clients to request ProRes video streams.
- Added `prores_profile` configuration option to set the FFmpeg ProRes profile.
- Updated changelog to reflect the addition of ProRes encoder plumbing.
- Enhanced configuration documentation with details on ProRes options.
- Implemented necessary changes in the codebase to support ProRes encoding, including updates to video format handling and encoder capabilities.
- Added tests to validate ProRes configuration and protocol behavior.
…olbox tab

The experimental ProRes mode selector landed in the generic Advanced
tab while prores_profile (its inseparable companion — the mode toggle
only makes sense if you also pick a profile) lived on the
VideoToolbox Encoder tab. Splitting the two knobs across tabs hurts
discoverability: users configuring VideoToolbox don't realise ProRes
exists, and users on the Advanced tab can't see the profile that
controls bit depth and chroma subsampling.

Move prores_mode into VideoToolboxEncoder.vue directly above
prores_profile so both controls sit together in their codec-relevant
section. The macOS-only platform guard previously around prores_mode
in Advanced.vue is implicit on the VideoToolbox tab (which is itself
macOS-only), so it can be dropped.

No backend or config change; pure UI relocation.
@Nottlespike Nottlespike force-pushed the feat/macos/prores-videotoolbox branch from f47b395 to 39eb20f Compare May 25, 2026 17:20
@Nottlespike
Copy link
Copy Markdown
Contributor Author

Force-pushed. Rebased onto the new combined #5190 tip (9030b66a) which now bundles SCK + EDR + HDR gating into a single PR.

Updated PR body to make the any-order mergeability explicit: this PR compiles and passes CI standalone. The functional interaction with #5190 is that ProRes capture needs 10-bit buffers; on a tree with #5192 but without #5190 the ProRes encoder probe fails and Sunshine falls back to the standard codec list (same path as any other failed-probe encoder), preserving existing behavior. When both land, ProRes works end-to-end through SCK's 10-bit path.

No code changes from the prior tip beyond the rebase context — same ProRes commit content (687b6c9347d4b0de), same UI relocation commit (39eb20f7f47b3953).

@sonarqubecloud
Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
5 New issues
5 New Code Smells (required ≤ 0)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants