Nodes persistence cleanup, APIs to control loading

[jglick]

When running CloudBees CI in high availability mode we need to control which agents are loaded from disk: blocking some from being loaded at startup, loading them later, reloading them from disk, unloading them from memory while they remain on disk. The four new methods here allow that level of fine-grained control. By default the behavior remains unchanged, of course: the list of agents in memory matches the contents of the nodes directory.

Testing done

Exercised by various functional tests in CloudBees CI.

Proposed changelog entries

• Add experimental APIs to control which agents are loaded and when.

Proposed upgrade guidelines

• N/A

Before the changes are marked as ready-for-merge:

Maintainer checklist

Edit tasklist title

Beta Give feedback Tasklist Maintainer checklist, more options

• Rename
• Copy markdown
• Delete tasklist

Delete tasklist

Delete tasklist block?

Are you sure? All relationships in this tasklist will be removed.

Cancel Delete

1. There are at least two (2) approvals for the pull request and no outstanding requests for change.

   There are at least two (2) approvals for the pull request and no outstanding requests for change.
   Options

   • Convert to issue
   • Toggle completion
   • Rename
   • Remove
2. Conversations in the pull request are over, or it is explicit that a reviewer is not blocking the change.

   Conversations in the pull request are over, or it is explicit that a reviewer is not blocking the change.
   Options

   • Convert to issue
   • Toggle completion
   • Rename
   • Remove
3. Changelog entries in the pull request title and/or **Proposed changelog entries** are accurate, human-readable, and in the imperative mood.

   Changelog entries in the pull request title and/or Proposed changelog entries are accurate, human-readable, and in the imperative mood.
   Options

   • Convert to issue
   • Toggle completion
   • Rename
   • Remove
4. Proper changelog labels are set so that the changelog can be generated automatically.

   Proper changelog labels are set so that the changelog can be generated automatically.
   Options

   • Convert to issue
   • Toggle completion
   • Rename
   • Remove
5. If the change needs additional upgrade steps from users, the `upgrade-guide-needed` label is set and there is a **Proposed upgrade guidelines** section in the pull request title (see [example](https://github.com/jenkinsci/jenkins/pull/4387)).

   If the change needs additional upgrade steps from users, the upgrade-guide-needed label is set and there is a Proposed upgrade guidelines section in the pull request title (see example).
   Options

   • Convert to issue
   • Toggle completion
   • Rename
   • Remove
6. If it would make sense to backport the change to LTS, a Jira issue must exist, be a _Bug_ or _Improvement_, and be labeled as `lts-candidate` to be considered (see [query](https://issues.jenkins.io/issues/?filter=12146)).

   If it would make sense to backport the change to LTS, a Jira issue must exist, be a Bug or Improvement, and be labeled as lts-candidate to be considered (see query).
   Options

   • Convert to issue
   • Toggle completion
   • Rename
   • Remove

Add item to Maintainer checklist

[jglick]

Taking the opportunity to simplify the logic here a bit: #5450 (comment)

[timja]

/label ready-for-merge

---

This PR is now ready for merge, after ~24 hours, we will merge it if there's no negative feedback.

Thanks!

[mattiassluis]

I couldn't find an issue number to comment to hence responding here:
Could it be that a side effect of this change is that renaming a node through Groovy (calling setNodeName) causes duplicate nodes?
We have a groovy init script that randomises node names and after updating to 2.449 it suddenly causes nodes to duplicate after running the script (we are aware that we are using API internals that probably shouldn't be used).

[jglick]

@mattiassluis that seems plausible. Renaming a node remains possible through the GUI so I think you were just using the wrong method all along. Try creating a new Slave object and calling Jenkins.removeNode + Jenkins.addNode, or Jenkins.setNodes. (To reconfigure one agent without renaming, create a new Slave and call updateNode. I am not sure if this is safe to use for a rename as well.)