Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

release-21.2: liveness: improve disk probes during node liveness updates #81514

Merged
merged 3 commits into from Jun 8, 2022
Merged

release-21.2: liveness: improve disk probes during node liveness updates #81514

merged 3 commits into from Jun 8, 2022

Commits on Jun 8, 2022

  1. liveness: improve disk probes during node liveness updates 

    When `NodeLiveness` updates the liveness record (e.g. during
heartbeats), it first does a noop sync write to all disks. This ensures
that a node with a stalled disk will fail to maintain liveness and lose
its leases.

However, this sync write could block indefinitely, and would not respect
the caller's context, which could cause the caller to stall rather than
time out. This in turn could lead to stalls higher up in the stack,
in particular with lease acquisitions that do a synchronous heartbeat.

This patch does the sync write in a separate goroutine in order to
respect the caller's context. The write operation itself will not
(can not) respect the context, and may thus leak a goroutine. However,
concurrent sync writes will coalesce onto an in-flight write.

Additionally, this runs the sync writes in parallel across all disks,
since we can now trivially do so. This may be advantageous on nodes with
many stores, to avoid spurious heartbeat failures under load.

Release note (bug fix): Disk write probes during node liveness
heartbeats will no longer get stuck on stalled disks, instead returning
an error once the operation times out. Additionally, disk probes now run
in parallel on nodes with multiple stores.
    @erikgrinaker
    erikgrinaker committed Jun 8, 2022
    Configuration menu
    Copy the full SHA
    9883db4 View commit details
    Browse the repository at this point in the history

  2. liveness: move stopper to NodeLivenessOptions 

    Release note: None
    @erikgrinaker
    erikgrinaker committed Jun 8, 2022
    Configuration menu
    Copy the full SHA
    c6ce5c8 View commit details
    Browse the repository at this point in the history

  3. liveness: run sync disk write in a stopper task 

    This patch runs the sync disk write during node heartbeats in a stopper
task. The write is done in a goroutine, so that we can respect the
caller's context cancellation (even though the write itself won't).
However, this could race with engine shutdown when stopping the node,
violating the Pebble contract and triggering the race detector. Running
it as a stopper task will cause the node to wait for the disk write to
complete before closing the engine.

Of course, if the disk stalls then node shutdown will now never
complete. This is very unfortunate, since stopping the node is often
the only mitigation to recover stuck ranges with stalled disks. This is
mitigated by Pebble panic'ing the node on stalled disks, and Kubernetes
and other orchestration tools killing the process after some time.

Release note: None
    @erikgrinaker
    erikgrinaker committed Jun 8, 2022
    Configuration menu
    Copy the full SHA
    c8dea85 View commit details
    Browse the repository at this point in the history