Transcoding pipeline graphs

[Shadowghost]

Summary

Report what a transcode is actually doing - the ordered chain of decode, filter and encode stages, the hardware framework each one runs on, and the frame format flowing between them - as structured data on TranscodingInfo, so the dashboard can draw a pipeline graph instead of a codec string. Along the way, fix the telemetry that feeds the existing transcoding display, which is either wrong or missing today.

Problem

The active-sessions dashboard is the only place a user can see why their server is on fire, and what it tells them is close to useless:

• HardwareAccelerationType is a lie by construction. It reports the server configuration, not what the running job does. A direct-stream remux, a software transcode, an audio-only transcode and a subtitle burn-in all report "QSV" on a QSV-configured server. Users read that as "hardware acceleration is working" and file bug reports when it isn't - or, worse, don't file them when it is silently falling back to software.
• There is no visibility into the filter chain at all. Tone mapping, deinterlacing, subtitle burn-in and scaling are exactly the stages that decide whether a transcode is realtime or not, and none of them are reported. The only way to find out is to read ffmpeg logs on the server.
• hwupload / hwdownload round-trips are invisible. A pipeline that decodes in hardware, downloads to system memory for a software filter, and uploads again to encode is a common and very expensive configuration mistake. Nothing surfaces it.

Proposal

1. A structured pipeline model on TranscodingInfo

TranscodingInfo.Pipeline : TranscodingPipelineInfo
    Stages : IReadOnlyList<TranscodingPipelineStage>

TranscodingPipelineStage
    Type             : TranscodeStageType   // Decode, Scale, Deinterlace, ToneMap, Subtitle,
                                            // Format, Encode, HardwareUpload, HardwareDownload, Copy
    MediaType        : "Video" | "Audio"    // lane the stage belongs to
    Framework        : HardwareFramework    // Software, Qsv, Cuda, Vaapi, D3D11Va, VideoToolbox,
                                            // Amf, OpenCl, Vulkan, Rkmpp, AudioToolbox, V4l2m2m, Dxva2
    Name             : string               // "hevc_qsv", "vpp_qsv", "tonemap_opencl", "h264_qsv"
    Detail           : string               // "1920x1080 → 1280x720", "bt2390", codec long name
    IsHardware       : bool
    EdgeLabel        : string               // frame format leaving this stage, e.g. "nv12 1280x720"
    VideoBitDepth    : int?                 // decode (source) and encode (output) stages only
    VideoRange       : VideoRangeType?      // ditto - comparing the two shows tone mapping happened
    VideoDoViTitle   : string               // "Dolby Vision Profile 8.1 (HDR10)" where the RPU survives

A remux is described as a single Copy stage rather than reported as nothing, so the client never has to
guess whether an empty pipeline means "direct play" or "we don't know".

Rendered, a session looks like this:

Video   [Decode hevc_qsv · QSV · 10-bit HDR10]
          → p010le 3840x2160
        [Scale vpp_qsv · QSV · 3840x2160 → 1920x1080]
          → p010le 1920x1080
        [ToneMap tonemap_opencl · OpenCL · bt2390]     ← hwupload/hwdownload shown if present
          → nv12 1920x1080
        [Encode h264_qsv · QSV · 8-bit SDR]
Audio   [Decode truehd] → [Encode ac3 · 6ch]

2. Derive the pipeline from the ffmpeg command line, enriched by -print_graphs

The command line Jellyfin generates is the most faithful description of the pipeline that is always available: every decoder, filter and encoder is present, and the names encode the framework (_qsv, _vaapi, _opencl, _rkrga). TranscodingPipelineBuilder.Build() tokenizes it and classifies each element. This path has no runtime cost and works on every supported ffmpeg.

ffmpeg 8.0 added -print_graphs, which dumps the negotiated filter graph as JSON - the real filters after ffmpeg's own insertions, with per-filter output dimensions and pad formats. That is strictly better data, but ffmpeg only flushes it on exit, so the long-running transcode cannot produce it. The proposal runs a short-lived graph probe: the same command line rewritten to -t 0 with output redirected to a throwaway directory, which initializes the graph and exits immediately.

Probe policy, because a second ffmpeg is not free:

• only when EncoderVersion >= 8.0 and the pipeline has a video encode stage (a remux or audio-only transcode has no graph worth dumping)
• serialized behind a semaphore, killed after 30 s, cancelled with the parent transcode
• result cached per PlaySessionId, so seeks and segment restarts reuse it
• the dump is only trusted if it describes every chain we expect (a truncated write or schema change falls back to the command-line parse rather than silently dropping a lane)
• any failure is logged at debug and never fails the transcode.

3. Honest hardware-acceleration reporting

HardwareAccelerationType becomes derived from the pipeline: reported only when the job actually has a hardware video stage, none otherwise. This is a behavioural change - servers configured for hardware acceleration will now correctly show none for remuxes, software fallbacks and audio-only transcodes.
That is the point, but it will read as a regression to anyone who has learned to read the old field.

4. Fill in the rest of the transcode telemetry

Additive, nullable fields on TranscodingInfo:

Field	Notes
Speed	realtime multiplier from ffmpeg speed= - the single most useful number for "will this keep up?"
AudioBitrate, VideoBitrate	target rates, separately from the combined Bitrate
BytesTranscoded	measured from the output on disk (summed segments for HLS/DASH), because ffmpeg reports N/A; cached 5 s, kept monotonic so segment cleanup can't make it go backwards
TranscodePositionTicks, TranscodeBufferTicks	how far ahead of the playback head the transcoder is
IsThrottled	whether the throttler currently has ffmpeg paused
SubtitleCodec, SubtitleDeliveryMethod	burn-in vs embed vs external is a major cost driver and is currently invisible
TranscodeProtocol	hls / dash / http

Plus the JobLogger parsing fixes: decimal bitrate units, KiB/MiB/GiB vs kB/MB/GB, padded values, Lsize= on the summary line, N/A, saturating instead of overflowing on Gbits/s).

API surface

All changes are additive and nullable. No existing field changes type or meaning except HardwareAccelerationType, whose meaning is corrected as described above.

New model types in MediaBrowser.Model.Session: TranscodingPipelineInfo, TranscodingPipelineStage, TranscodeStageType, HardwareFramework. These flow through the existing SessionInfoDto on /Sessions and the sessions WebSocket message - no new endpoints.

Clients that ignore Pipeline continue to work unchanged.

Client work

jellyfin-web dashboard: render the stage chain as two lanes (video/audio), colour by Framework, badge IsHardware, show EdgeLabel on connectors, and highlight HardwareUpload/HardwareDownload transitions as the "this is costing you" signal. Speed and IsThrottled belong next to the existing progress readout.

Nothing here is required for the server change to land; the fields are useful in isolation (an admin can already read them via /Sessions).

Risks and open questions

1. The probe is a second ffmpeg process. It is short-lived and serialized, but on a heavily loaded server it briefly takes a hardware session. Should it be behind an encoding option (default on/off)? Should it be skipped when N transcodes are already running?
2. -print_graphs is ffmpeg 8.0+. Everyone below that gets the command-line-derived pipeline, which lacks negotiated formats and EdgeLabel.
3. Parsing ffmpeg arguments is inherently fragile. It is confined to a pure static class with no effect on the transcode, and every failure path degrades to "no pipeline reported", but it will need maintenance as EncodingHelper evolves. 26 unit tests cover the current filter/codec matrix.
4. BytesTranscoded for segmented output walks the transcode directory, which is shared by all concurrent jobs. Even if we cache for 5 s; still O(files in directory) per refresh. Fine at typical scale, worth watching on servers with many concurrent HLS sessions.

Frontend Mock