Worker communication reliability

[sparshev]

Hello folks, we recently moved to AWS and our network reliability showed not good results: our jenkins JNLP agents are disconnecting here and there during checkouts or uploads failing the builds. So maybe someone can describe for a noob how concourse Workers communicates with Web and how it will behave during the pipeline execution?

For example: there is a git fetch operation in progress on the worker. Right in the middle the web node becomes unavailable. What the worker and pipeline will do? Ideally - they both will continue as is until there is nothing to do and wait, and when the communication between them will recover pipeline will continue.

[taylorsilva]

Hey, I can explain how your example would work. Basics first: the web and worker nodes communicate over SSH. The web node needs to allow ingress over it's ssh listening post, which is 2222 by default on the web nodes. The Workers need egress access to the web node. Workers don't need to allow any ingress access to any of their own ports.

Web nodes are responsible for orchestrating everything. The workers are told what containers to run. Those are the roles they both play.

Now let's walk through your example:

there is a git fetch operation in progress on the worker.

So the web node has already sent commands to the worker to start a container that will use git to download your repo.

Right in the middle the web node becomes unavailable. What the worker and pipeline will do? Ideally - they both will continue as is until there is nothing to do and wait, and when the communication between them will recover pipeline will continue.

The worker will keep doing what it was doing, the git repo will finish downloading.
The web node will see the worker disappear. By default I believe it will wait 5mins for the worker to come back. Some logs may be missed but the final result of the worker should be captured.

If the worker node is gone for a long time, like ten's of minutes, then the web node will eventually forget about whatever was happening on that worker. But if the worker comes back quickly then it'll look like nothing happened from a pipeline/job perspective.

[taylorsilva]

Bad use of the word forget. In that scenario where it appears the worker is gone for good, concourse will fail the job. That's basically all you care about and what will happen. Concourse also keeps track of all the artifacts generated during a build and GC's them once a build is done. If the worker disappears, concourse will assume those temporary artifacts are gone, which is what I meant by "forget". That's very much an implementation detail though so not something that you necessarily need to care about as a pipeline builder.