nvkms: fix duplicated half-image on 2Heads1OR displays (5K 120Hz VR headsets)

[rvdende]

Problem

any display requiring 2Heads1OR shows the left half of the framebuffer duplicated across the output; reproduced on a Vive Pro 2 at 4896x2448@120

---

Fix:

It took quite a bit of digging and logging, but figured out how to get 120hz 5k working under linux. The problem was that the left eye video was copied to the right eye.

Here is some technical info from claude (which actually did the fix for me).

nvkms: honour mergeHeadSection in the modeset viewport origin

Under 2Heads1OR the api head's frame is split horizontally across two hardware heads, and each head must scan out its own section of the surface. That section is selected by a viewport origin offset of viewPort.in.width * mergeHeadSection.

Two of the three paths that set the viewport origin apply the offset: nvApiHeadSetViewportPointIn() when panning (nvkms-flip.c), and nvUpdateFlipEvoHwState() on flip (nvkms-hw-flip.c) -- but the latter only inside "if (pParams->viewPortIn.specified)", and nvidia-drm never specifies viewPortIn, so on the DRM/KMS path it does not run.

nvSetViewPortsEvo() hardcoded the origin to 0 for every head. With nothing else setting it, both merge heads scanned out from x=0 under 2Heads1OR, so the leftmost section was repeated across the output and the right-hand half of the framebuffer was never displayed.

Observed with an HTC Vive Pro 2 at 4896x2448@120 over DisplayPort: the pixel clock (1543470 kHz) exceeds a single head's maximum, so the mode is split across two heads (raster 4996x2574 -> 2498x2574, viewport in width 4896 -> 2448 each). The headset showed the left half of the framebuffer on both eyes. With this change head 0 gets origin x=0 and head 1 x=2448, and the image is correct.

mergeHeadSection is 0 whenever 2Heads1OR is not in use, so this is a no-op for single-head configurations.

[Binary-Eater]

Thanks @rvdende. This fix makes sense to me for handling viewports with 2head1or. Will merge the change internally and mention your contrib in the internal change commit message.

[rvdende]

@Binary-Eater thanks appreciate it

[rvdende]

@Binary-Eater I've run into a stability issue

Separate issue found while testing this under sustained load

While exercising this 2Heads1OR path on a Vive Pro 2 at 4896x2448@120 (DSC Mode = DUAL, pclk 1543470000) driving an OpenXR compositor, I hit a hard hang that takes the whole session down. It is not the doubling bug this PR fixes, but it lives on the same merged-head path, so raising it here.

Symptom. Both merged heads' base window channels stop retiring methods — NVC67E_WINDOW_CHANNEL_DMA (0x0000c67e) instances :0 and :2, i.e. head * 2 for head 0 and head 1. Their pushbuffer rings fill and nvEvoMakeRoom() spins forever:

nvidia-modeset: ERROR: GPU:0: Error while waiting for GPU progress: 0x0000c67e:0 2:0:56:48
nvidia-modeset: ERROR: GPU:0: Error while waiting for GPU progress: 0x0000c67e:0 2:0:56:48
...repeats every 5s indefinitely

Decoding count:fifo_free_count:getOffset:putOffset: the ring is full and the hardware get pointer has stopped advancing — methods were submitted and the display engine is not consuming them.

Why it kills the machine. nvEvoMakeRoom() has no timeout. On expiry it logs and resets startTime, as the existing comment acknowledges:

/*
 * If we have been waiting too long, print an error message.  There
 * isn't much we can do as currently structured, so just reset
 * startTime.
 */

The caller is typically holding the nvkms lock, so every subsequent modeset blocks behind it in D state and the desktop is unusable until a hard reboot. This is the same failure shape as #1192 (unbounded GPFIFO/pushbuffer waits in nvidia-push), one layer up in the EVO display channel.

Fix for the hang. Branch: fix-evo-makeroom-unbounded-wait (commit 3271055, based on this PR's branch).

After two consecutive 5s timeouts the channel is marked dead and further methods are dropped rather than written into a ring nobody is draining. nvEvoMakeRoom() returns void and cannot report failure — nvDmaSetStartEvoMethod() is a static inline with 255 call sites and nvkms has no device-lost concept — so the flag lives on NVDmaBufferEvoRec and is checked in the three places that touch the ring: nvDmaSetStartEvoMethod() (headers), nvDmaSetEvoMethodData() (data words, which callers push directly, so guarding only the header still corrupts the ring), and nvDmaKickoffEvo().

Verified on the reproduction above — both channels now retire and stop, and the desktop survives:

nvidia-modeset: ERROR: GPU:0: Error while waiting for GPU progress: 0x0000c67e:0
nvidia-modeset: ERROR: GPU:0: Error while waiting for GPU progress: 0x0000c67e:0
nvidia-modeset: ERROR: GPU:0: Channel 0x0000c67e:0 has made no progress in 10
                seconds; marking it dead and dropping further methods.

The compositor and client both stayed alive and the session was recoverable by restarting the runtime instead of rebooting. Ten seconds rather than something longer is measured, not arbitrary: a first attempt at 30s was dead code, because the session becomes unusable enough to force a power-cycle after ~15s.

This bounds the hang; it does not fix the underlying stall. Any guidance on why the merged-head window channels stop retiring under load would be welcome — the display engine appears to simply stop consuming, and I have not identified what it is blocked on.

Environment: RTX 4090, 610.43.03 open modules, kernel 7.0.0-28-generic, Hyprland/Wayland, Monado direct-mode DRM lease.

[rvdende]

It seems in simple scenes the 5k 120hz works fine for a long time, but if you add additional load then the VR view freezes and a little while later like 60 seconds the entire desktop session freezes. So there is definitely some other code that needs fixes aswell. Im digging through and trying to piece things bit by bit but to be honest I have very little idea of where to go from here.

[rvdende]

The 2Heads1OR overfetch path looks half-implemented — is the valid-region register meant to be exposed?

While testing this patch on a Vive Pro 2 at 4896x2448@120 (DSC Mode = DUAL, pclk 1543470000, heads 0+1 merged), I went looking for anything else that assumes a head's viewport origin is 0. One thing stands out, and I'd like to check my reading of it before doing anything with it.

nvGetMaxPixelsFetchedPerLine() sizes the fetch bound as though overfetch were enabled:

/*
 * Ampere, which adds "overfetch" to have tiled displays / 2-head-1-OR use cases without
 * visual artefacts at head boundaries:
 * (((SetViewportSizeIn.Width + 14) * SetMaxInputScaleFactor.Horizontal + 1023) >> 10) + 8
 */
return (((inWidth + 14) * maxHDownscaleFactor + 1023) >> 10) + 8;

src/nvidia-modeset/include/nvkms-evo3.h:301-308

But OVERFETCH_ENABLED is never programmed. The fields exist:

clc67d.h:485  NVC67D_WINDOW_SET_WINDOW_USAGE_BOUNDS_OVERFETCH_ENABLED   30:30
clc67d.h:761  NVC67D_HEAD_SET_HEAD_USAGE_BOUNDS_OVERFETCH_ENABLED       16:16
clc87d.h:484  NVC87D_WINDOW_SET_WINDOW_USAGE_BOUNDS_OVERFETCH_ENABLED   30:30
clc87d.h:760  NVC87D_HEAD_SET_HEAD_USAGE_BOUNDS_OVERFETCH_ENABLED       16:16

and the usage bounds are written at those same method offsets, but with older-class field definitions, so the bit goes out as 0:

nvkms-evo3.c:5920   DRF_NUM(C37D, _WINDOW_SET_WINDOW_USAGE_BOUNDS, _MAX_PIXELS_FETCHED_PER_LINE, ...)
nvkms-evo3.c:5952   NVC57D_HEAD_SET_HEAD_USAGE_BOUNDS(head)
nvkms-evo3.c:2553   DRF_NUM(C57D, _WINDOW_SET_WINDOW_USAGE_BOUNDS, _MAX_PIXELS_FETCHED_PER_LINE, ...)
nvkms-evo4.c:831    DRF_NUM(C97D, _WINDOW_SET_WINDOW_USAGE_BOUNDS, _MAX_PIXELS_FETCHED_PER_LINE, ...)

Grepping the whole tree, OVERFETCH appears in the driver only in comments — never in a method write.

And the registers the TODOs point at aren't in the open headers:

nvkms-evo3.c:5825   /* XXXnvdisplay set ViewportValidPointIn to configure overfetch */
nvkms-evo3.c:5874   /* XXXnvdisplay set ViewportValidSizeIn to configure overfetch */
nvkms-evo4.c:736    /* XXXnvdisplay set ViewportValidPointIn to configure overfetch */
nvkms-evo4.c:785    /* XXXnvdisplay set ViewportValidSizeIn to configure overfetch */

ViewportValidPointIn / ViewportValidSizeIn do not appear anywhere under src/common/ — I searched every display class header under all the spellings I could think of. So from the open driver the enable bit is reachable but the region that bounds it is not.

Why this is asymmetric, and why 2Heads1OR is where it shows. The offset only exists in HEAD_SET_VIEWPORT_POINT_IN; NVHwModeViewPortEvo.in carries width/height and no origin. Cross-referencing NVDispHeadStateEvoRec::mergeHeadSection with clangd gives five readers, all of which apply the offset correctly:

nvkms-cursor.c:307    x - (hwViewportInWidth * mergeHeadSection)
nvkms-flip.c:821      x + (hwViewportInWidth * mergeHeadSection)
nvkms-hw-flip.c:1251  viewPortPointIn.x + (viewPort.in.width * mergeHeadSection)
nvkms-evo.c:960       passed to SetRasterParams()
nvkms-evo.c (modeset) the origin this PR sets

and no readers at all in nvkms-evo3.c or nvkms-evo4.c — the HAL layer that programs usage bounds, IMP parameters and fetch limits never sees it. Every hardware-facing calculation there is written as though the head starts at x=0.

Concretely, for this mode (each merged head has viewPort.in.width = 2448 of a 4896-wide surface):

maxPixelsFetchedPerLine = ((2448 + 14) * 1024 + 1023) >> 10) + 8 = 2470

head 0, origin 0      declares fetch of 0    .. 2469   inside the surface
head 1, origin 2448   declares fetch of 2448 .. 4917   22 px past the end of every line

Also worth noting IMP is told nothing about this: AssignPerWindowImpParams() derives everything from pViewPort->in, so the two merged heads present as identical inputs. Instrumenting nvValidateImpOneDisp() on this hardware confirms it — both heads report viewPortIn=2448x2448 viewPortOut=2448x2448@0,0, byte-identical, while the scanout origins differ (0 and 2448). On dGPU that only affects the possible/not-possible decision, since reallocBandwidth is hardcoded to NONE in nvkms-modeset.c and RM-IMP never returns bandwidth figures.

The questions I actually have:

1. Is MAX_PIXELS_FETCHED_PER_LINE a hint used only for isohub arbitration, or does the hardware prefetch up to it? If the latter, the second merged head is declaring a fetch that runs past the end of each line, and that would be specific to mergeHeadSection > 0.
2. Are ViewportValidPointIn / ViewportValidSizeIn intended to be published? Without them the open driver can't complete the overfetch setup those TODOs describe.
3. Is setting OVERFETCH_ENABLED without programming the valid region safe, or does the valid region have to be configured first? Enabling it alone looks like it authorises fetching beyond the viewport with nothing bounding it, which seems worse than leaving it off — hence asking rather than proposing a patch.

I'm not claiming this causes a specific failure; I haven't proven that, and the compositor hang I hit on this path (reported separately) has no GPU fault associated with it. This is narrower: the overfetch allowance is accounted for in the fetch sizing but the feature is never switched on, and the piece that would make switching it on safe isn't available in the open driver. Since this PR is what puts a head at a non-zero viewport origin in the first place, it seemed worth raising here.

Environment: RTX 4090 (AD102, C67D core channel), 610.43.03 open modules, kernel 7.0.0-28-generic, Hyprland/Wayland, Monado direct-mode DRM lease.