Increase GCE retries to 3

[wilzbach]

It's still sometimes failing, e.g. https://ci.dlang.io/blue/organizations/jenkins/dlang-org%2Fphobos/detail/PR-6029/8/pipeline

[dlang-bot]

Thanks for your pull request, @wilzbach!

[wilzbach]

Another PR hitting this: https://ci.dlang.io/blue/organizations/jenkins/dlang-org%2Fdmd/detail/PR-7735/4/pipeline

[wilzbach]

Interestingly the restart doesn't seem to work on PRs: https://ci.dlang.io/blue/organizations/jenkins/dlang-org%2Fdmd/detail/PR-7735/12/pipeline

[MartinNowak]

Not a good approach, if we cannot even run a single batch job on a just started instance, than I'd rather stop using them.
As for https://ci.dlang.io/job/dlang-org/job/dmd/job/PR-7735/12/consoleFull, retry seems to work fine

[Pipeline] [rejectedsoftware/vibe.d] echo
[rejectedsoftware/vibe.d] java.io.IOException: remote file operation failed: /var/lib/jenkins/dlang_projects at hudson.remoting.Channel@4162cae6:JNLP4-connect connection from 117.95.205.35.bc.googleusercontent.com/35.205.95.117:54386: hudson.remoting.ChannelClosedException: Remote call on JNLP4-connect connection from 117.95.205.35.bc.googleusercontent.com/35.205.95.117:54386 failed. The channel is closing down or has closed down
[Pipeline] [rejectedsoftware/vibe.d] node
[rejectedsoftware/vibe.d] Running on runner-01.ci.lxd in /var/lib/jenkins/workspace/dlang-org_dmd_PR-7735-B3CYZW5SU3CSBQTA6VTWKIBAWFA6ZHACGPH4XNNIWYFPUJ6KMLUA

It's just that we hit the overall timeout of 1hr :o. That alone could be a reason to abandon GCE.

[rejectedsoftware/vibe.d] http/vibe/http/server.d(2026,56): Deprecation: variable vibe.http.server.HTTPServerOption.distribute is deprecated - Use runWorkerTaskDist or start threads separately. It will be removed in 0.9.
Body did not finish within grace period; terminating with extreme prejudice
[Pipeline] [rejectedsoftware/vibe.d] }

[MartinNowak]

Let's see how much parallelizing vibe-d's tests helps.

[wilzbach]

BTW I'm seeing these spurious failures that ideally should be part of the retry loop too:

• Jenkins's fails to write files (locking problem?) #135 (filesystem)
• Cannot contact runner-01-gce: java.lang.InterruptedException #136 (InteruptedException)
• Vibe.d: failed to connect to host #126 (failed to connect to host)

[wilzbach]

Jenkins is no longer in use.