Workaround for single-bit VRAM errors — quarantine the bad cell instead of chasing the chip

[JoshBolding]

If memtest_vulkan shows repeated single-bit errors at the same address suffix, that's one defective cell in one VRAM IC — see #43, #65 and #89. The usual conclusion is that it's tolerable for gaming but shouldn't be trusted for ML or rendering, because the failing chip can't be identified and consumer cards have no page retirement.

I hit this on a 3090 FE and ended up sidestepping the identification problem entirely:

https://github.com/JoshBolding/gpu-vram-quarantine

It allocates the whole card in small chunks, runs a walking-bit sweep to find which chunks hold bad cells, frees the clean ones and holds the rest for the life of the process. VRAM is a shared physical resource, so once those pages are held nothing else can be given them — no cooperation needed from whatever you actually want to run. It locates the faults on every run rather than hardcoding an address, because the offset a cell appears at moves with allocator placement.

On my card it holds 8 MiB and leaves 98.9% of a healthy card's usable VRAM, against 84.9% if you just cap allocation below the fault. Verified afterwards with 342 TiB at 64 °C — the temperature where it otherwise fails constantly — with zero errors.

Two things I learned that might help anyone testing: the fault produced nothing for the first ~143 seconds of every run, so short or cold tests pass a broken card. And coverage matters — mine sat above 20 GiB, so anything smaller never touched it.

Linux and CUDA only, validated on one card, and it's a workaround rather than a repair. Caveats are in the README. Interested to hear from anyone who tries it on different hardware.

[galkinvv]

Very interesting.

Some notes: as far as I know cuda on windows has the ability to move allocated data in the physical memory, in partucally for temporal swapping VRAM->System RAM if several apps allocates many VRAM. This seems not yet be implemented in Linux CUDA, but it may become a problem in the future. However, given CUDA may have some low-level-API this may be solveable. This is a "Pro" for using cuda.

The "Con" for using cuda is "deploy nightmare". Recompiling depending on used GPU would require a cuda sdk installed, not only cuda runtime. Maybe this may be made better by compiling against 12.9 sdk (Maxwell-blackwell) with all arches in a single binary. For such app performance is not so important, so builtining coppute_XX arches containg PTX code maybe worth it too for wider compatibility.

the fault produced nothing for the first ~143 seconds of every run, so short or cold tests pass a broken card.

The "practical physics" tells us that GPU thermal stabilization after putting it under a load came after 2-5 minutes depending on heatsink organization. Since many kinds of errors are tempearture dependent - the 5-minutes default duration of memtet_vualkn is the not the random number, but a time ensuring thermal stabilization of any GPU.

[JoshBolding]

The relocation point is the one that worries me, and I hadn't considered it. If the driver can move a cudaMalloc allocation's backing pages, the holder keeps its virtual range but stops owning the physical cell — and would still report the quarantine as active while protecting nothing. Silent failure of exactly the kind the tool exists to prevent.

Linux not doing it today is luck rather than design on my part. The right answer is presumably the VMM API — cuMemCreate gives an explicit physical allocation handle rather than a relocatable mapping. I'll look at moving the holder onto that.

In the meantime I'm adding a watchdog that periodically re-verifies the fault still reproduces inside the chunk being held. If the allocation ever moved, the held chunk would stop failing and the real cell would reappear elsewhere in a scan — detectable, where today it wouldn't be. That also covers a case Olari-A raised on the other thread from a different direction.

The deploy point is fair and easy to fix — I'll build a fat binary against 12.9 with all arches plus PTX so it doesn't need an SDK or an -arch guess. Performance is irrelevant for this, so there's no reason not to.

And thanks for the thermal stabilization figure. I'd observed the ~143 s onset empirically without knowing it lined up with a known 2–5 minute window, or that the 5-minute default was chosen for that reason. That's a much better justification for my search timeouts than "it seemed to take a while."

[JoshBolding]

Both points are addressed.

The fat binary is done — default make now emits SASS for Maxwell through Hopper plus PTX, so there's no -arch guess and no SDK version matching. Ten seconds to build, about 1.1 MB.

The relocation risk is documented in the README as the most serious open issue, and I've added a watchdog as partial mitigation: the holder periodically re-sweeps the memory it's holding, so if the allocation were ever moved the fault would stop reproducing there rather than the whole thing failing silently. The proper fix is still to move the holder onto cuMemCreate for an explicit physical handle, which I haven't done yet.

Your thermal stabilization point turned out to be load-bearing rather than incidental. My first watchdog warned on elapsed time and produced a false warning within two minutes of a healthy quarantine, because a small held region on an idle card simply never fails. It now only counts time spent above a temperature threshold, which is your 2–5 minute observation turned into a control. Temperature comes from NVML via dlopen rather than a link-time dependency, specifically so it doesn't reintroduce the deploy problem you raised.