Video Stream

Guidance for building an app against the -apistream video stream: how to discover it, which format to use, what the parameters mean, and what the client is responsible for. The endpoint list and query parameters are also in the main README.

Note

This feature is in active development and things may change.

Reference Implementation

See https://github.com/spice2x/substream which is a web app that performs streaming & uses WebSocket to send back touch input.

Connecting

Connect to the JSON API first and call capture.get_streams(). Do not guess the stream port or paths: builds may expose different formats, and no stream server listens when -apistream is disabled.

Request:

{
    "id": 1,
    "module": "capture",
    "function": "get_streams",
    "params": []
}

Example response:

{
    "id": 1,
    "errors": [],
    "data": [{
        "port": 1339,
        "formats": [
            { "name": "mjpeg", "path": "/stream.mjpg" },
            { "name": "h264", "path": "/stream.h264" }
        ],
        "screens": [
            { "screen": 0, "width": 1920, "height": 1080, "busy": false },
            { "screen": 1, "width": 1280, "height": 720, "busy": true }
        ]
    }]
}

data is empty when no stream server is available. Treat an API failure or an empty response as no stream: do not fall back to deriving a port from the API port.

Use the same host as the API connection, the returned port, and the path for the desired format. For example:

    http://host:1339/stream.h264?screen=0&fps=30&q=70

The screens array only contains screens whose dimensions are known. A screen may be absent while the game is starting or loading, then appear on a later call. Refresh get_streams() before retrying and when another viewer connects or leaves. If the user explicitly selected a screen that temporarily disappears, keep that selection and wait for it to return rather than silently switching screens.

busy means a stream currently holds that screen. It can be your own stream if you query again after connecting. It is advisory: another client can claim the screen after the API response but before your HTTP connection arrives. A 503 Service Unavailable is still a normal race to handle with backoff. Include busy screens in selection UIs so a user can choose one and wait for it to become free.

Once discovered, open a plain HTTP GET and read the body until you close it or the server does. The stream port itself has no authentication or additional handshake.

An H.264 stream always begins at a keyframe, so there is nothing to seek and no startup state to recover. Every MJPEG frame is independent.

screen, fps and q are fixed for the life of a connection. Changing any of them means opening a new one.

A screen carries one stream at a time. A second connection to a screen that is already streaming is refused with 503 Service Unavailable, and nothing is ever preempted, so an existing viewer cannot be interrupted by someone else connecting.

fps is a ceiling rather than a promise. Nothing competes with you for frames on your own screen, but you receive the lesser of your requested rate and whatever the game is actually redrawing, so a request for 30 on a screen that only updates 21 times a second delivers 21. Derive timing from arrival, not from the value you asked for.

The server ends the connection if the game changes resolution. Treat a closed connection as routine and reconnect rather than surfacing an error.

Choosing a format

	stream.mjpg	stream.h264
Container	multipart/x-mixed-replace	none
Bandwidth	about 2x to 20x higher	low
Plays in a normal player	yes	yes
Works in a browser	yes, in an <img> tag	yes, through WebCodecs or MediaSource
Client work	none	drives a decoder or wraps the stream for MediaSource
Latency	near instant	decoder-dependent

stream.h264 is the default choice for an app. H.264 costs roughly twenty times less bandwidth than MJPEG and phones decode it in hardware, which matters a great deal for battery life. It is a bare elementary stream, so the app feeds the bytes to MediaCodec or VideoToolbox itself and nothing sits between the socket and the decoder.

stream.mjpg needs no container support at all and every frame stands alone, so it is the fallback when nothing else works. It is expensive on bandwidth and on client CPU.

WebCodecs can consume the H.264 access units directly; however, many modern browsers will prevent you from doing this in http unless the client is also localhost. Surprisingly, this works fine with iOS Safari.

Where WebCodecs is unavailable, MediaSource can still play H.264 after the client wraps it in fragmented MP4. The MP4 track header needs the correct display dimensions before the first sample is appended; use the width and height reported for the selected screen rather than hardcoding or guessing them. The substream reference client implements both paths.

Quality

q means different things per format, despite the shared name and default:

• stream.mjpg passes it straight through as the standard 1-100 JPEG quality.
• stream.h264 maps it onto the H.264 rate factor as CRF = 40 - q / 4.

So for H.264:

q	CRF
1	39.75	badly blocky
40	30	visibly lossy
70	22.5	default, close to x264's own default of 23
100	15	high quality

CRF is constant quality, not constant bitrate. q sets how much loss the encoder tolerates; the resulting bitrate floats with how much the screen is moving.

Latency

The server puts a frame on the socket about 5 ms after capturing it, so nearly all of the delay a user sees comes from the client.

The usual cause is a player that buffers ahead and then runs its own clock; one that decides during startup that it is half a second behind will stay half a second behind indefinitely. Whatever you use, find the option that renders on decode instead of scheduling against a presentation clock. In mpv:

mpv --demuxer-lavf-format=h264 --no-correct-pts --container-fps-override=30 --untimed --profile=low-latency http://host:1339/stream.h264

The equivalents are releaseOutputBuffer(index, true) on MediaCodec and kCMSampleAttachmentKey_DisplayImmediately on VideoToolbox. An annex-b stream carries no timestamps of its own, so nothing downstream has a clock to fall behind on in the first place.

Testing

stream.h264 has no container and no timestamps. Some players cope on their own - MPC decodes it as served - while others need the format and frame rate supplied. Either way this is the cheapest way to check the stream is sane:

mpv --demuxer-lavf-format=h264 --no-correct-pts --container-fps-override=30 --untimed --profile=low-latency http://host:1339/stream.h264
ffplay -f h264 -fflags nobuffer -flags low_delay http://host:1339/stream.h264

Set --container-fps-override to the rate you are actually receiving, which is the fps parameter only while the game keeps up with it. Without any value ffmpeg assumes 25. Getting it wrong mostly affects mpv's reported position and its buffering estimates, since --untimed renders each frame as it decodes rather than scheduling against the timeline.

Client notes

Both platforms take the same bytes from stream.h264; nothing is negotiated and the server does not care what the client is.

Android

MediaCodec accepts annex-b directly, with the parameter sets either in band or as csd-0 and csd-1. Call releaseOutputBuffer(index, true) to render on decode rather than scheduling against a clock.

iOS

VideoToolbox has no annex-b entry point. The client has to split the stream on start codes, build a CMVideoFormatDescription from the SPS and PPS with CMVideoFormatDescriptionCreateFromH264ParameterSets, and replace each four byte start code with a four byte big endian length before wrapping the NALs in a CMBlockBuffer. Set kCMSampleAttachmentKey_DisplayImmediately on each sample buffer and leave the display layer's controlTimebase unset; without it you have rebuilt the presentation clock problem.

AVSampleBufferDisplayLayer is much less work than a full VTDecompressionSession and is the better starting point for a test harness. The same API exists on macOS, so the decode path can be proven there before dealing with a device.

A client cannot reach a plain HTTP address on the local network without an NSAppTransportSecurity exception, and iOS 14 and later also require NSLocalNetworkUsageDescription before it will connect to a LAN address at all. Without the latter the connection fails in a way that looks like the server is unreachable.

Flutter

Decode on the platform side and render into a Texture. A platform channel hands the socket data to MediaCodec or VideoToolbox, and the decoded frames go back through a SurfaceTexture on Android or a CVPixelBuffer on iOS. Nothing then sits between the socket and the display, and both platforms keep their render-on-decode behaviour.

Limitations

• One stream per screen. Anything beyond either limit gets 503 Service Unavailable and is closed immediately, so back off rather than retrying in a tight loop.
• A client that vanishes without closing its socket keeps holding its screen until the send times out, which takes a few seconds. A reconnect arriving sooner than that is refused, so retry with backoff rather than assuming the screen is gone for good.
• The screen list is dynamic. A registered screen is omitted until its size has been measured, so an empty or incomplete list during startup is not an error. Re-query with backoff.
• Each stream runs its own encoder on the machine hosting the game, and every captured frame costs that machine a present cycle. Streaming two screens at once roughly halves the rate each one can reach.
• A client that stops reading is dropped after five seconds. The server never buffers a backlog for a slow reader; it skips to the newest frame instead, so frames are lost rather than delayed.
• View only. Touch and other input still go through the JSON API, so a companion app needs both.
• No authentication on the stream port. Anyone who can reach it can watch the screen.
• WinXP builds have no video stream. Neither encoder is compiled in, so nothing listens on the stream port even with -apistream, capture.get_streams() returns no data, and the JSON API's JPEG screen capture is unavailable for the same reason.
• Streaming has performance impact on the game. Some games will be more sensitive than others, depending on the game engine. In general, limit streaming to 30 FPS.