ACME http01Challenge randomly not works with k8s after pull new docker image

[jkralik]

Subject of the issue

When acme server challenging acme client (lego) it gets error:

• dial tcp: i/o timeout
• dial tcp 10.106.221.133:80: connect: connection refused

It is caused by ingress when during container startup with updated docker image.

Your environment - kubectl version

Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T11:13:54Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.2", GitCommit:"c97fe5036ef3df2967d086711e6c0c405941e14b", GitTreeState:"clean", BuildDate:"2019-10-15T19:09:08Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"

Expected behaviour

Use retry-able http client (https://github.com/hashicorp/go-retryablehttp) for acme http challenge.

Actual behaviour

Randomly acme client doesn't obtain certificate for updated container.

Additional context

When I replaced

certificates/acme/authority.go

Line 265 in 967e86a

client := http.Client{

by

        import retryablehttp "github.com/hashicorp/go-retryablehttp"
        ...
	client := retryablehttp.NewClient()
	client.HTTPClient = &http.Client{
		Timeout: 20 * time.Second,
	}
	client.RetryWaitMin = 100 * time.Millisecond
	client.RetryWaitMax = 1 * time.Second
	client.RetryMax = 10

It is works as is expected.

[jkralik]

I fount that lego uses package for retry: https://github.com/cenkalti/backoff
e.g:
https://github.com/go-acme/lego/blob/add2bea5778e18eda3696bd4cfe7dc7818bb5d06/acme/api/api.go#L81

[sourishkrout]

@jkralik please track #168 for ACME retries which we're currently completing. That's essentially a more wholistic way to solve this in a standard compliant way.