Automatic retry mechanism if task fail to fetch material

[ariefrahmansyah]

Issue Type

• Feature proposal

Summary

I'm using GitLab as main Git repository and sometimes I have connectivity issue that made git fetching failed. Can GoCD have an automatic retry mechanism to re-trigger the material fetching task?

Any other info

I'm thinking for something like Concourse's attempt: https://concourse-ci.org/attempts-step-modifier.html

[arvindsv]

@ariefrahmansyah If it's an auto polled material, it should retry in around a minute. If not, there's currently no mechanism to retry (a forced trigger or timer, for instance).

[ariefrahmansyah]

@arvindsv sorry, it's not auto poll material, but it's when running task failed to fetch material. the task will be failed.

If it happens, we need to manually retrigger the task.

[stale]

This issue has been automatically marked as stale because it has not had activity in the last 90 days.
If you can still reproduce this error on the master branch using local development environment or on the latest GoCD Release, please reply with all of the information you have about it in order to keep the issue open.
Thank you for all your contributions.

[dominik-ba]

I would really love to have this feature!

[mattgauntseo-sentry]

We run into this issue every now and then, it would be great to have a method to set retry count and / or a wait between retries.

[chadlwilson]

@mattgauntseo-sentry Have you determined root cause of the issue in your case? Is it a git material or some other material type or plugin material? Is it a temporary connectivity problem or target material source rate limit?

[mattgauntseo-sentry]

We use github and it presents itself as a ssh connection issue. It's always intermittent. I don't believe this is a rate limit, but wouldn't rule it out.

I'll share an example log when it happens again.

[chadlwilson]

Hmm. GitHub definitely has some sort of rate limit on https (although mixed info available on that, as they say they dont limit?) and GoCD's polling nature can be a problem for that, but not 100% sure what it looks like for ssh, and whether it's different on enterprise plans.

Regardless useful to know, as there are probably multiple layers this could be addressed (specific material type, more abstract for all materials - that type of thing.

Is this a standard OOTB GoCD git material or some extra plugin (eg for PRs)?

[mattgauntseo-sentry]

This is a standard OOTB git material with an ssh url (git@github.com/...). I believe these are repos in an enterprise plan.

[mattgauntseo-sentry]

Just ran into this during a test:

[go] Start to prepare <pipeline name>/52/deploy_primary/1/deploy on <elastic agent name>-<elastic agent ID> [/go]
[go] Start to update materials.

[go] Start updating <pipeline group> at revision <commit revision> from git@github.com:<org>/<repo>.git
STDERR: Cloning into '/go/pipelines/<pipeline name>/<pipeline group>'...
[GIT] Fetching changes
STDERR: ssh: connect to host github.com port 22: Connection timed out
STDERR: fatal: Could not read from remote repository.
STDERR: 
STDERR: Please make sure you have the correct access rights
STDERR: and the repository exists.
git fetch failed for [git@github.com:<org>/<repo>.git]

[chadlwilson]

Yeah, that looks like a general networking issue where at best anything done at GoCD level will be a bit of a fudge. But perhaps a necessary one in some setups.

If you see these types of errors exclusively on agents rather than the server logs you might want to explore anything different about the agent environment or specific to elastic agents (although the server can typically recover due to its polling nature, assuming you're not using webhooks).

[mattgauntseo-sentry]

I've seen this on the server too with config repos hitting this error, but in those cases the polling gets us to an eventual good state.