This repository has been archived by the owner on Jul 25, 2022. It is now read-only.
Action plugin execution blocks daemon plugins #2436
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
A long running action plugin will block the polling of all daemon plugins until it completes.
This is because the
Sessions.message
method takes a read lock here:integrated-manager-for-lustre/iml-agent/src/http_comms/session.rs
Line 154 in 5e71ee9
Which is then held for the entirety of the plugin invocation as the plugin is resident inside that
RWLock
.At the same time, A write lock is attempted on the plugins here:
integrated-manager-for-lustre/iml-agent/src/poller.rs
Line 118 in 691b4e0
When the loop tries the write lock on the same plugin that is running the message, it will block until the read lock is released.
Depending on what the plugin is doing, this could potentially be many hours.
To solve this, we should break the dependency on a
RWLock
in general at theSession
level. This can be done by making sure allDaemonPlugin
areClone
. Most plugins have all their state inArc<Mutex<...>>
so this is not too big of a change.In addition, when calling
Sessions.message
we should clone the plugin before invokingon_message
so there will be no reliance on the read lock for the duration of the call.Fixes #2435.
Signed-off-by: Joe Grund jgrund@whamcloud.io
This change is![Reviewable](https://camo.githubusercontent.com/1541c4039185914e83657d3683ec25920c672c6c5c7ab4240ee7bff601adec0b/68747470733a2f2f72657669657761626c652e696f2f7265766965775f627574746f6e2e737667)