GCE workers sometimes die

[wilzbach]

Cannot contact runner-01-gce: java.io.IOException: remote file operation failed: /var/lib/jenkins/dlang_projects@5/BlackEdder/ggplotd at hudson.remoting.Channel@2ad430c7:JNLP4-connect connection from 30.0.195.35.bc.googleusercontent.com/35.195.0.30:58308: hudson.remoting.ChannelClosedException: Remote call on JNLP4-connect connection from 30.0.195.35.bc.googleusercontent.com/35.195.0.30:58308 failed. The channel is closing down or has closed down

script returned exit code -1

https://ci.dlang.io/blue/organizations/jenkins/dlang-org%2Fdmd/detail/PR-7661/3/pipeline

[wilzbach]

Another example:

Cannot contact runner-01-gce: java.io.IOException: remote file operation failed: /var/lib/jenkins/dlang_projects@3/rejectedsoftware/vibe.d at hudson.remoting.Channel@2ad430c7:JNLP4-connect connection from 30.0.195.35.bc.googleusercontent.com/35.195.0.30:58308: hudson.remoting.ChannelClosedException: Remote call on JNLP4-connect connection from 30.0.195.35.bc.googleusercontent.com/35.195.0.30:58308 failed. The channel is closing down or has closed down

script returned exit code -1

https://ci.dlang.io/blue/organizations/jenkins/dlang-org%2Fdmd/detail/PR-7666/4/pipeline

[wilzbach]

More examples:

• https://ci.dlang.io/blue/organizations/jenkins/dlang-org%2Fdruntime/detail/PR-2015/2/pipeline
• https://ci.dlang.io/blue/organizations/jenkins/dlang-org%2Fdruntime/detail/PR-2013/2/pipeline
• https://ci.dlang.io/blue/organizations/jenkins/dlang-org%2Fdmd/detail/PR-7692/1/pipeline
• https://ci.dlang.io/blue/organizations/jenkins/dlang-org%2Fphobos/detail/PR-6008/18/pipeline
• https://ci.dlang.io/blue/organizations/jenkins/dlang-org%2Fdruntime/detail/PR-2018/4/pipeline
• https://ci.dlang.io/blue/organizations/jenkins/dlang-org%2Fdruntime/detail/PR-2015/3/pipeline
• https://ci.dlang.io/blue/organizations/jenkins/dlang-org%2Fdruntime/detail/PR-2013/3/pipeline
• https://ci.dlang.io/blue/organizations/jenkins/dlang-org%2Fdmd/detail/PR-7710/2/pipeline
• https://ci.dlang.io/blue/organizations/jenkins/dlang-org%2Fdmd/detail/PR-7702/2/pipeline

[MartinNowak]

Yip, this is by design running on pre-emptible instances.
I've tried to catch such exceptions and reschedule the job.
#109

Looks like we end up with nested exceptions here IOException wrapping ChannelException :/.
https://ci.dlang.io/blue/organizations/jenkins/dlang-org%2Fdruntime/detail/PR-2013/3/pipeline/#step-298-log-656
https://ci.dlang.io/blue/organizations/jenkins/dlang-org%2Fdruntime/detail/PR-2013/3/pipeline/#step-298-log-656
https://ci.dlang.io/blue/organizations/jenkins/dlang-org%2Fdruntime/detail/PR-2015/3/pipeline/#step-291-log-4017
https://ci.dlang.io/blue/organizations/jenkins/dlang-org%2Fdruntime/detail/PR-2018/4/pipeline/#step-448-log-3

IMO a good argument to ditch nested exceptions.
https://stackoverflow.com/a/2955504/2371032
No don't use e.getCause().getCause() use e.getRootCause() ;).

[MartinNowak]

Mmh, nested exceptions in groovy look different.

def foo() {
  throw new Exception("An Exception")
}

def bar() {
  try {
    foo()
  } catch (Exception e) {
    throw new IOException("Even more exceptions", e);
  }
}

bar()

java.lang.Exception: An Exception
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        ...
	at Script1.bar(Script1.groovy:7)
Caused: java.io.IOException: Even more exceptions
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        ...

So maybe they're are printing an exception catched in a Channel thread. In that case searching for ClosedChannelException in the message could work. You really oughta hate this Jenkins mess.

[MartinNowak]

Suppressed exceptions look like this

def foo() {
  throw new Exception("An Exception")
}

def bar() {
  java.lang.Throwable t
  try {
    foo()
  } catch (Exception e) {
    t = e
  } finally {
    def e = new IOException("Even more exceptions")
    e.addSuppressed(t)
    throw e
  }
}

bar()

Also:   java.lang.Exception: An Exception
		at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
		at Script1.bar(Script1.groovy:8)
java.io.IOException: Even more exceptions
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

[MartinNowak]

Cannot contact runner-01-gce: java.io.IOException: remote file operation failed: /var/lib/jenkins/dlang_projects@2/rejectedsoftware/vibe.d at hudson.remoting.Channel@7ad644a6:JNLP4-connect connection from 66.139.205.35.bc.googleusercontent.com/35.205.139.66:53536: hudson.remoting.ChannelClosedException: channel is already closed