GetOutputList crashes OBS when an output is being torn down (upstream)

[Agash]

Tracking on our side. The fix belongs in obs-websocket; the analysis below is written so it can be handed over as-is.

What happens

GetOutputList reads each output's encoder without holding a reference. If an output is being destroyed concurrently, the encoder is freed between the null check and the read, and OBS crashes.

Reproduced by starting the replay buffer, saving a clip, stopping it, and calling GetOutputList shortly after. OBS 32.2.2, obs-websocket 5.7.

Fault address: 7FFACB1D5C67 (obs.dll)
Thread 6B8C: Thread (pooled) (Crashed)

obs.dll!obs_encoder_get_width+0x87
obs-websocket.dll!`Utils::Obs::ArrayHelper::GetOutputList'::`2'::<lambda_1>::<lambda_invoker_cdecl>+0x5d7
obs.dll!obs_enum_outputs+0x3a
obs-websocket.dll!RequestHandler::GetOutputList+0x4b
obs-websocket.dll!RequestHandler::ProcessRequest+0x196
obs-websocket.dll!WebSocketServer::ProcessMessage+0x115c

Root cause

src/utils/Obs_ArrayHelper.cpp, GetOutputList:

outputJson["outputWidth"] = obs_output_get_width(output);
outputJson["outputHeight"] = obs_output_get_height(output);

obs_output_get_width reaches obs_output_get_width2, which does:

if (flag_encoded(output)) {
    if (output->video_encoders[idx])
        return obs_encoder_get_width(output->video_encoders[idx]);

The null check and the dereference are not atomic with respect to teardown, and obs_enum_outputs gives the callback no reference to the encoder. The websocket thread reads it while the frontend thread frees it.

Also seen as a hang

Earlier in the same session, after SetOutputSettings on a live output, GetOutputList stopped responding for the rest of the OBS session, timing out in code paths unrelated to the call that preceded it. Same function, same field, so probably the same race landing on a lock rather than a free.

Impact for us

Any consumer calling GetOutputList around output lifecycle changes can take OBS down. Nothing a client can do about it: the request is a plain read.

What we do meanwhile

ObsWebSocket.Example run-transport-tests does not start or stop the replay buffer, so GetLastReplayBufferReplay is reported as untested rather than prepared for. SetOutputSettings is not sent either.

[Agash]

Second defect in the same function, found by the request sweep while fixing our side.

GetOutputList reports outputWidth/outputHeight straight from obs_output_get_width/_height with no clamp:

// Obs_ArrayHelper.cpp
outputJson["outputWidth"] = obs_output_get_width(output);
outputJson["outputHeight"] = obs_output_get_height(output);

Both return uint32_t, and for an output that has never started the value is not meaningful. A live OBS 32.2.2 sent:

{ "outputKind": "virtualcam_output", "outputActive": false, "outputWidth": 0, "outputHeight": 2586032160 }

Note the asymmetry: width 0, height garbage. For a non-encoded output the read is scaled_height != 0 ? scaled_height : video_output_get_height(output->video), so scaled_height is holding a stale or uninitialised value while scaled_width is zero.

Client impact is out of proportion to the field: JSON has one numeric type, so a client that types the field as a signed 32-bit int fails to deserialize the whole response, not just that member. It is intermittent and only affects users who happen to have a virtual camera or a capture plugin installed.

Fixed on our side by widening to 64 bits and documenting that a value over 4096 is not a size. Reporting it here because the guard belongs upstream: an idle output could reasonably report 0.