Action plugin execution blocks daemon plugins

[jgrund]

A long running action plugin will block the polling of all daemon plugins until it completes.

This is because the Sessions.message method takes a read lock here:

integrated-manager-for-lustre/iml-agent/src/http_comms/session.rs

Line 154 in 5e71ee9

if let State::Active(active) = state.read().await.deref() {

Which is then held for the entirety of the plugin invocation as the plugin is resident inside that RWLock.

At the same time, A write lock is attempted on the plugins here:

integrated-manager-for-lustre/iml-agent/src/poller.rs

Line 118 in 691b4e0

Arc::clone(&state).write().await.deref_mut(),

When the loop tries the write lock on the same plugin that is running the message, it will block until the read lock is released.

Depending on what the plugin is doing, this could potentially be many hours.

To solve this, we should break the dependency on a RWLock in general at the Session level. This can be done by making sure all DaemonPlugin are Clone. Most plugins have all their state in Arc<Mutex<...>> so this is not too big of a change.

In addition, when calling Sessions.message we should clone the plugin before invoking on_message so there will be no reliance on the read lock for the duration of the call.

Fixes #2435.

Signed-off-by: Joe Grund jgrund@whamcloud.io

---

This change is