[JENKINS-49707] If a KubernetesComputer disconnects, remove the KubernetesSlave

[jglick]

Downstream of jenkinsci/workflow-durable-task-step-plugin#104. If the pod has disconnected, I am assuming it will not be able to reconnect, and the agent state is lost—true? If so, there is no point in letting any running sh step continue: it should fail now.

[carlossg]

LGTM

[jglick]

Seems RestartPipelineTest is broken by these changes. Investigating.

[jglick]

Hmm, live tests fail because I did not consider this RBAC scenario:

WARNING	o.c.j.p.k.pod.retention.Reaper#activate: failed to set up watcher on kubernetes
io.fabric8.kubernetes.client.KubernetesClientException: pods is forbidden: User "system:serviceaccount:…" cannot watch pods at the cluster scope

I suppose the answer is to watch only within the KubernetesCloud.namespace (also do a startup-time list pods in this namespace), and do not attempt to clean up agents associated with pods using a custom namespace, which I assume is rare anyway.

[jglick]

With @schottsfired’s help I got 61307bf tested in a semirealistic environment.

[schottsfired]

Brilliant, @jglick! Looks good on my end.

[jglick]

Perhaps related to #201.

[jglick]

Tested using AWS Quickstart for CloudBees Core on EKS with a Pipeline targeted to Spot instances. Simulated a termination in various ways. kubectl delete pod:

Cannot contact …: hudson.remoting.ChannelClosedException: Channel "unknown": Remote call on JNLP4-connect connection from … failed. The channel is closing down or has closed down

sudo halt -ff:

java.net.SocketTimeoutException: sent ping but didn't receive pong within 1000ms (after 182 successful ping/pongs)
	at okhttp3.internal.ws.RealWebSocket.writePingFrame(RealWebSocket.java:546)
	at okhttp3.internal.ws.RealWebSocket$PingRunnable.run(RealWebSocket.java:530)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Cannot contact …: java.lang.InterruptedException

…and in either case the agent does get removed and the build aborted. 🎉