Investigate mark Jenkins node offline due to machine issue

[llxia]

As we run more and more test builds nightly and weekly with different jvm and versions and parallelization, the machine issue (i.e., space issue, git issue, network issue, etc) on one machine could tank the whole nightly run on that platform. I would like to investigate to see if we can mark Jenkins node offline automatically due to machine issues.

example of a bad run:

It looks like there is a Jenkins plugin for it https://plugins.jenkins.io/offlineonfailure-plugin/

[sxa]

You have to be very careful with anything like that - you run the risk of one type of test (e.g. all system tests) taking all the test boxes off line if there is a specific system test thing that's making the jobs fail catastrophically.

[llxia]

Yes, we will be very careful/strict about the rules. The plan is not to take nodes offline when there is a failure. I would like to like to find a way to take nodes offline for the specific failures that we are certain are the machine issues, such as no disk space, cannot run git, etc. The above plugin may or may not suits our needs, but it is worth investigating.

[llxia]

The above plugin is quite old. We can just use Jenkins API to achieve setting node temporarily offline due to specific errors.