ASSERTION i>= 0 FAILED in put_ldev

[TomyLobo]

Summary

During a LINSTOR disk migration using linstor resource toggle-disk --migrate-from, DRBD emits a sustained flood of put_ldev assertions:

drbd pm-<randomhex>/0 drbd<id>: ASSERTION i >= 0 FAILED in put_ldev

The assertion storm occurs while the migration transfer is in progress and drops sharply when the transfer completes.

Environment

DRBD module: 9.3.2
LINSTOR controller: 1.33.3-1
LINSTOR satellite: 1.33.3-1
drbd-utils: 9.34.3-1
Kernel: 7.0.6-2-pve
Distribution: Proxmox Virtual Environment 9.2.0, based on Debian 13 trixie, recently updated from 8.4, based on Debian 12 bookworm
Transport: tcp, I think

Trigger

The issue occurs during:

linstor resource toggle-disk \
  --storage-pool "$target_storage_pool" \
  --migrate-from "$migration_source" \
  "$diskless_target_node" \
  "$resource"

The goal is to move the secondary replica to another node that is initially diskless.

Observed behavior

During the migration/transfer, the kernel log is flooded with assertions like:

[Mon Jun 29 16:33:07 2026] drbd pm-<randomhex>/0 drbd<id>: ASSERTION i >= 0 FAILED in put_ldev

The observed rate is approximately:

10,000-20,000 assertion messages per second, sustained

The spam dropped sharply when the transfer concluded, from thousands of messages per second to dozens per second.

This made kernel logging effectively unusable during the event. On the affected node, journalctl retained only about 7 minutes of history because the assertion flood displaced older kernel logs.

Expected behavior

A normal LINSTOR disk migration should not trigger repeated put_ldev assertions.

If this assertion can be hit repeatedly in a hot path, it should at least be rate-limited so that it does not flood the kernel log and obscure diagnostic context.

Impact

- 10k-20k kernel log messages per second
- rapid eviction of useful journal history, making it impossible to diagnose other issues should they occur

Correlation with migration

The issue appears tightly correlated with the active transfer window:

- starts during `linstor resource toggle-disk --storage-pool "$target_storage_pool" --migrate-from "$migration_source" "$diskless_target_node" "$resource"`
- continues at very high rate while transfer is active
- drops sharply when the transfer completes

Changelog check

No obvious matching fix was found in the changelog of the upcoming RC (9.3.3-rc.1). The listed fixes include several teardown/resync/connection-related fixes, but none explicitly mention put_ldev, local-device refcounting, or unbalanced local-device references, so I'm not sure if upgrading to the RC would fix those issues.

[dvance]

Just want to make a quick mention that we observed this in another Proxmox environment with 9.3.3. After downgrading to the latest 9.2.19 we no longer saw these errors in the logs. It does seem to be something specific within the 9.3 branch.

[CTaeumel]

I dug into this and found the root cause. make_resync_request() takes a single get_ldev() reference and releases it exactly once in out_put_ldev, including on the -EAGAIN path. But make_one_resync_request() never takes its own reference — and its allocation-failure branch releases one anyway:

peer_req = drbd_alloc_peer_req(peer_device, GFP_TRY & ~__GFP_RECLAIM,
size, REQ_OP_WRITE);
if (!peer_req) {
drbd_err(device, "Could not allocate resync request\n");
put_ldev(device); /* <-- stray release, caller already owns this ref */
return -EAGAIN;
}
So one failed allocation results in one get_ldev() but two put_ldev() calls, underflowing device->local_cnt. From then on every subsequent get_ldev()/put_ldev() pair decrements to a negative value, tripping D_ASSERT(device, i >= 0) in put_ldev() on every I/O completion — hence the flood. The allocation failure itself is expected/transient (it's a deliberately non-blocking GFP_TRY allocation that can run dry under many parallel resyncs); the bug is purely in the error path's refcounting. It was introduced by 223009b ("drbd: synchronize resync with interval tree and dagtags") and is present in every release since 9.2.0, including current master.

I opened #139 with a one-line fix: removing the stray put_ldev() so the reference is released exactly once, by the caller, matching how read_for_csum()'s defer: path already handles this correctly.

Full disclosure: I'm not a C developer, so I can't fully vouch for the patch myself — I used AI assistance (Fable) to help work through the reasoning, and it looks sound to me, but I'd appreciate someone with DRBD internals experience reviewing/testing #139 before it's merged.

[CTaeumel]

Wo submitted a patch: https://lore.kernel.org/drbd-dev/02903afc-cda1-4d97-8155-4e03c3cc0e51@data-experts.de/T/#u

[fsennj]

Another trigger on the same version: online verify (not migration).

Environment:

• DRBD kernel module 9.3.2 (GIT-hash 0605140bf521251f4f362b5bad2eeaf33dba1e9f), drbd-utils 9.34.3
• Proxmox VE 9.2.4, kernel 7.0.2-6-pve
• LINSTOR, 6 diskful nodes + 1 diskless tiebreaker
• verify-alg sha512, nightly drbdadm verify from cron

Identical message:

drbd <res>/0 drbd1099: ASSERTION i >= 0 FAILED in put_ldev

Measured rate: 90,012 lines in a 5-second sample (~18,000/s). 3000 of 3000 sampled kernel lines were this single message; the journal reached 3.4 GB.

Resource state while storming - no degradation:

<res> role:Primary
  disk:UpToDate open:yes
  peerA role:Secondary  replication:VerifyT  peer-disk:UpToDate  done:39.37
  peerB role:Secondary  peer-disk:Diskless   peer-client:yes

The flood correlates with VerifyT, consistent with your observation that it tracks active data transfer. As in your case, the resource has a diskless peer.

A second-order effect, in support of the rate-limiting request: at ~18k lines/s, journalctl -k --since "1 hour ago" no longer completes - killed at a 20 s timeout with 2,854,992 lines already emitted, versus 16 ms / 59 lines on an unaffected node in the same cluster. Any monitoring that shells out to journalctl under a timeout loses the whole host, so the node silently disappears from inventory instead of reporting a fault. journalctl -k -n N stays fast.

Also worth noting: the flood was local only. rsyslog did not forward it, so the node's remote log stream looked entirely normal (~1 line/s) while the local journal took 18,000/s - meaning log-based alerting cannot see this at all.

Is the fix in #139 expected to cover the verify path too, or is verify a separate path? Happy to test a build.