Found on nvcf-dgxc-k8s-aws-usw2-dev2 while cleaning up after auth testing. Independent of #555, though auth is what exposed it.
Behaviour
cascadeDeleteCheckpoint deletes the catalog row even when the L1 tier delete fails. The API returns 204 and the caller believes the checkpoint is gone, while the on-disk dump survives with nothing left pointing at it.
Observed:
DELETE /api/v1/checkpoints/247eb2e0...__20260809-194514 -> 204
catalog: 0 rows
disk: 33G /var/lib/nvsnap/checkpoints (unchanged)
Why the orphan is unrecoverable
The catalog row is the only record tying a checkpoint id to its L1 location. Once dropped:
The only remaining recourse is removing the directory on the node by hand, which is what I had to do.
Impact
Every failed cascade permanently leaks a full checkpoint. These are large: a single vllm-small dump on this cluster is 33G, of which 32G is pages-11.img. A handful of failures fills a node disk, and the only signal is a warning in the agent log.
The failure mode is not hypothetical. It fired for every delete while the agent ran with --auth-mode=required, because nvsnap-server does not send the agent token (details in #555).
Suggested fix
Do not delete the catalog row unless the L1 delete succeeded. On partial failure, keep the row and surface the state, so the checkpoint stays enumerable and a retry can finish the job. Returning 207 or 500 with a machine-readable per-tier result would let callers distinguish "gone" from "partially gone"; 204 currently claims more than the server knows.
A reconcile sweep that reclaims dumps with no catalog row would be defence in depth, and would pair naturally with #206.
Related
Found on nvcf-dgxc-k8s-aws-usw2-dev2 while cleaning up after auth testing. Independent of #555, though auth is what exposed it.
Behaviour
cascadeDeleteCheckpointdeletes the catalog row even when the L1 tier delete fails. The API returns 204 and the caller believes the checkpoint is gone, while the on-disk dump survives with nothing left pointing at it.Observed:
Why the orphan is unrecoverable
The catalog row is the only record tying a checkpoint id to its L1 location. Once dropped:
checkpoint.sh cleanupcannot find it, since it enumerates through/api/v1/checkpoints.DELETEby id returns 404; there is no row to cascade from.The only remaining recourse is removing the directory on the node by hand, which is what I had to do.
Impact
Every failed cascade permanently leaks a full checkpoint. These are large: a single vllm-small dump on this cluster is 33G, of which 32G is
pages-11.img. A handful of failures fills a node disk, and the only signal is a warning in the agent log.The failure mode is not hypothetical. It fired for every delete while the agent ran with
--auth-mode=required, because nvsnap-server does not send the agent token (details in #555).Suggested fix
Do not delete the catalog row unless the L1 delete succeeded. On partial failure, keep the row and surface the state, so the checkpoint stays enumerable and a retry can finish the job. Returning 207 or 500 with a machine-readable per-tier result would let callers distinguish "gone" from "partially gone"; 204 currently claims more than the server knows.
A reconcile sweep that reclaims dumps with no catalog row would be defence in depth, and would pair naturally with #206.
Related