fix(registry): retry pull/push/tag 3 times - should help with slow networks and slow registries

[helgi]

Fixes #977

[deis-bot]

@mboersma and @krancour are potential reviewers of this pull request based on my analysis of git blame information. Thanks @helgi!

[codecov-io]

Current coverage is 85.18% (diff: 100%)

No coverage report found for master at 4273f76.

Powered by Codecov. Last update 4273f76...4412ab5

[slack]

SGTM, does this get logged anywhere that an operator would see?

[helgi]

No, I opted not to do that, tho if it fails 3 times then it will raise the errors all the way up

[felixbuenemann]

@helgi Do you think your fix will also help with errors like this:

Successfully built a6b761447898
Pushing to registry
remote: etail":{"message":"Put http://localhost:5555/v1/repositories/cms/: EOF"},"error":"Put http://localhost:5555/v1/repositories/cms/: EOF"}

I'm using an S3-backed on-cluster registry where the bucket is in the same region as the nodes.

[helgi]

It could potentially help, only if the network starts behaving or resuming works, if part of the data got pushed.

Are you seeing that consistently or sporadically?

[felixbuenemann]

Sporadically, I forgot to check all the logs to get a better insight into what went wrong.