Modifying the plugin settings while jobs are queued or running

[szy54]

Hello, is the "Gotcha"(mentioned at the bottom of FAQ about modifying plugin settings considered a bug or desired behaviour? Are there any plans for implementing it the way that it's possible to modify the plugin settings on the fly without jenkins restart?

Is your feature request related to a problem? Please describe.
A possible problem emerging from this limitation is described here

Describe the solution you'd like
As a jenkins admin I am able to change and save plugin configuration while builds are running or queued and be sure that the plugin will scale the fleet right.

Describe alternatives you've considered
Actually there's no alternative in my opinion. There are many use cases that you need to change the plugin config here and now without risking that you will need to restart jenkins in the middle of the day.
Scheduling a maintenance window for such config change is a PITA.

Additional context
not applies

[haugenj]

It's not desired per say, but it seems to be a limitation with how Jenkins core loads the plugin configuration and constructs clouds - it essentially restarts our plugin but leaves Jenkins core running. I don't know of a way around that, but in theory jobs should be resubmitted when the new clouds are ready and everything would sort itself out.

The gotcha is there because we've noticed that modifying the config while jobs are running/queued is a common theme when people find scaling issues, and we've found issues related to this in the past (e.g. #172, #236). However, the scaling issues sometimes appear even when the configuration isn't touched. The challenge is that we don't have reliable ways of reproducing these errors, which makes it hard to narrow down where the real problem is.

[mwos-sl]

but in theory jobs should be resubmitted when the new clouds are ready and everything would sort itself out

If by "re-submitting" you mean restarting, I think it isn't desired either. To my mind it's more about rediscovering already running nodes, re-attaching to them and allowing to continue the work they are in the middle if it makes sense.
At least this is what from user perspective is done by a competetive https://plugins.jenkins.io/ec2/ plugin. This one allows reloading configuration on the fly, and builds just continue their execution, no agents suspended. We've been using this one so far and no problems with reloading config, no matter it touched the nodes config or some other one.

The problem with the competetive one (ec2-plugin) is it's not as good with handling spot instances as this one (ec2-fleet-plugin), that's why we'd like to move to ec2-fleet plugin.

[haugenj]

Sorry, I misspoke there, if the cloud configuration is changed and the underlying fleet/asg is the same we reconnect to the instances and existing executors and the jobs that are running continue (just tested this to make sure). If the fleet/asg of the cloud is changed though, we resubmit the job since the executor is lost to a node that we no longer manage and is no longer part of Jenkins.

Changing the config at any point in time should be fine, it just seems to be a common factor when we get scaling problems so we have the gotcha there until we're able to actually reproduce the issues and figure out whether modifying the config is actually buggy