Latency

Latency

Measured on a StarTech PEXHDCAP60L: 1080p60 sustained, around 1 ms between a frame arriving from the card and being drawn. That figure is the age of the frame at the moment the draw call is issued, read from the statistics overlay, which computes it from the QPC timestamp taken when the sample was handed over.

Four decisions account for it. Each is a place where the obvious implementation would have added a queue.

1. A renderer filter that does not render

src/capture/frame_sink.h, src/capture/frame_sink.cpp

The stock DirectShow renderers — VMR9, EVR — schedule samples against the graph clock and keep a queue in front of them. That is correct for playback and exactly wrong here. FrameSink is a minimal IBaseFilter with one input pin (SinkPin, implementing IPin and IMemInputPin) that does the opposite.

SinkPin::Receive forwards to FrameSink::OnSample, which:

1. Checks the sample for an attached media type. A mid-stream format change arrives that way; when the width, height or subtype differ from the current format, all three slots are reallocated and the ready and read indices are reset to −1 so nothing half-sized is ever handed out.
2. Picks a write slot under the lock — PickWriteSlotLocked() returns the one index of three that is neither the slot waiting to be picked up (readyIdx_) nor the slot the reader is holding (readIdx_).
3. Copies outside the lock. The chosen slot cannot be touched by anyone else, and only the upstream streaming thread calls Receive, so there is no second writer. The memcpy — the expensive part — therefore never blocks the render thread.
4. Retakes the lock only to publish: if readyIdx_ was still set, that frame never reached the screen and dropped_ is incremented. Then readyIdx_ becomes the new slot, sequence_ and received_ advance, and lastArrivalQpc_ is stamped.
5. Signals frameEvent_, an auto-reset event, so the render loop can sleep on a handle instead of polling.

Receive returns immediately in every case. Frames arriving faster than they can be shown are overwritten, not buffered, so delay cannot accumulate — the dropped counter in the statistics overlay is the visible consequence.

AcquireFrame is the reader half. It moves readyIdx_ into readIdx_, clears readyIdx_, and returns true when the frame is one the reader has not seen. The returned FrameView always describes the frame currently held, even when that is the previous one — so a render pass triggered by something other than a new frame still has something valid to draw.

ReceiveCanBlock returns S_FALSE, and GetMiscFlags reports AM_FILTER_MISC_FLAGS_IS_RENDERER, which is what stops the graph from inserting anything helpful upstream.

Why three slots

Two would deadlock the copy-outside-the-lock trick: with the reader holding one and one waiting to be picked up, there is no third to write into. Three is the minimum that lets the writer always have a free slot without ever waiting for the reader, and there is no benefit to four — a fourth slot is a queue, which is the thing being avoided.

2. No graph clock

FrameSink::SetSyncSource accepts the clock and stores it, but the graph is run without one: src/capture/video_capture.cpp calls IMediaFilter::SetSyncSource(nullptr) before running. With no reference clock, no filter holds a sample back until its presentation time, and the capture filter delivers as soon as the driver hands it something.

The cost is that DirectShow's own timestamps become advisory. Nothing in CapView uses them for the display path; the recorder derives its timeline from the audio instead (see Recording).

3. Flip-model swap chain, one frame of queue

src/render/d3d_context.cpp

• DXGI_SWAP_EFFECT_FLIP_DISCARD
• IDXGIDevice1::SetMaximumFrameLatency(1)
• DXGI_SWAP_CHAIN_FLAG_ALLOW_TEARING where the adapter supports it, present with DXGI_PRESENT_ALLOW_TEARING and a sync interval of 0
• VSync off by default

Two details are worth stating because both were found the hard way:

A sync interval of 0 without the tearing flag still blocks. On a flip-model swap chain, DXGI treats "no vsync" as "no vsync if you have told me you can tear". Without ALLOW_TEARING on both the swap chain and the present call, the present waits for the next vertical blank anyway. d3d_context.cpp queries IDXGIFactory5::CheckFeatureSupport(DXGI_FEATURE_PRESENT_ALLOW_TEARING, …) and records the answer in tearingSupported().

SetMaximumFrameLatency applies to the device, not to a swap chain. With one frame of queue and two swap chains — which happens when the settings live in their own window — each present waits on the other's frame to retire. That is covered on The settings window; the fix is D3DContext::SetFrameLatency(UINT), which raises the queue to three while the second window exists and drops it back to one when it closes.

4. Format conversion on the GPU

src/render/shaders.h, pass 1 (kConvertPS)

YUY2, UYVY, YVYU, NV12, planar 4:2:0, RGB24 and RGB32 are unpacked in a pixel shader. Nothing is converted on the CPU between Receive and the draw — the bytes go from the capture buffer into a D3D11_USAGE_DYNAMIC texture with Map/Unmap and are decoded on the way out.

Everything in that pass uses texel fetches (Load) rather than samplers, so no hardware filtering can smear anything before it is meant to. Sampling with a filter enabled would interpolate across the chroma subsampling boundary, which is a colour error, not a softness.

The render loop

App::Run in src/app.cpp waits on the sink's frame event with a timeout, and draws when one of three things is true:

const bool wokeOnPicture = lastWait_ == WAIT_OBJECT_0 && lastWaitHadEvent_;
const bool fieldDue      = secondFieldPending_ && nowQpc >= secondFieldQpc_;
if (wokeOnPicture || fieldDue || sinceRenderMs >= 200.0) {
  lastRenderQpc_ = nowQpc;
  RenderFrame();
}

• wokeOnPicture — a new frame arrived. This is the normal case and the only one that matters for latency.

• fieldDue — a bob-style deinterlacer is showing the second field of the frame it already has, and that field's turn has come. Scheduled from FrameSink::lastArrivalQpc(), not from when the first field was drawn: scheduling from the draw is how a second field ends up due after the next frame has already arrived.

  This condition used to read lastWait_ == WAIT_TIMEOUT && secondFieldPending_, and that was a bug with a very specific footprint. MsgWaitForMultipleObjectsEx returns WAIT_TIMEOUT only when nothing else woke the loop. Open the settings window and there is a steady stream of messages, so the wait returns "input available" every time and the timeout branch never runs. The next arriving picture then takes the haveNewFrame branch, which resets fieldIndex_ — so every second field was silently dropped for as long as the dialog was open, and only on interlaced sources.

  It is the same shape of mistake as the WM_TIMER that could not compete with the drag loop's message flood (see The settings window): a schedule must not be conditional on the message queue being quiet. Asking the clock instead is both correct and cheaper.

• sinceRenderMs >= 200 — a floor, so level meters and toasts keep moving when there is no signal at all.

Everything else the message loop wakes up for — mouse moves, timer messages, the settings window's own traffic — deliberately does not trigger a redraw. Before that was true, the pipeline measured 235 draws a second on a source delivering 25, because a second window on screen produces a steady stream of messages and every one of them was being treated as a reason to redraw.

These same three conditions are what the settings window's drag loop applies as well. While a window is being dragged, Windows keeps the thread to itself and this loop does not run at all, so the preview is driven from a message the dialog receives instead — but the decision of whether to draw is identical, down to consuming the same auto-reset capture event with a zero wait. Two different ways in, one rule. See The settings window for the two wrong versions that came before it, both of which were a number.

Where the remaining milliseconds are

The 1 ms figure is frame age at draw time. It does not include, and CapView cannot measure:

• the card's own capture and DMA latency
• the display's processing
• DWM composition, when not in fullscreen

Fullscreen removes the last of those. The overlay's frame-age reading is the honest part of the number; the rest of the chain is the card's and the monitor's.