Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

redirectpolicy: prevent startup deadlock #27115

Merged
merged 1 commit into from
Jul 31, 2023

Commits on Jul 31, 2023

  1. redirectpolicy: prevent startup deadlock

    This commit prevents a deadlock in startup when local redirect policies
    are in use.
    
    Involved in the deadlock is the node-local pod watcher and the
    redirectpolicy manager. The pod watcher (synchronously) calls into the
    manager, in oder to notify it of a pod event. The manager then tries to
    acquire a reference to the pod store. Due to recent refactoring,
    however, this leads to a wait cycle:
    
    1. Acquiring the pod store reference blocks on initial synchronisation
       of the store, by waiting for the `podStoreSet` channel to be closed.
    2. This channel is closed by the watcher once it receives a `Sync` event
       from the local pod resource. Before the resource emits a Sync event,
       however, it will 'replay' the consumer up to the current snapshot by
       emitting Upsert events.
    3. In the Upsert call, the redirectpolicy manager is called (blocking on
       1.), and we have a classical deadlock.
    
    To fix this, instead of going through the k8s watcher to acquire a
    reference to the local pod store, inject the resource directly into the
    redirect policy manager. This will also block on the store being
    synchronised, but not by way of the `podStoreSet` channel (closed by the
    k8s watcher): it instead relies on the Resource[T]-internal tracking.
    
    Fixes: 3db8b14 (k8s: Use LocalPodResource in pod watcher)
    
    Signed-off-by: David Bimmler <david.bimmler@isovalent.com>
    bimmlerd committed Jul 31, 2023
    Configuration menu
    Copy the full SHA
    31f56d5 View commit details
    Browse the repository at this point in the history