Failure to put git images to Azure Blob.

[monaka]

Moved from deis/workflow#340.

NAMESPACE   NAME                           READY     STATUS    RESTARTS   AGE       NODE
deis      deis-builder-549937908-ohf0y   0/1       CrashLoopBackOff   5         6m        10.0.2.4

$ kubectl logs   --namespace deis -f deis-builder-549937908-ohf0y                                                                                                                         
2016/09/16 01:32:47 Error creating storage driver (Put https://{censored}.blob.core.windows.net/builder?restype=container: dial tcp: i/o timeout)
m

It looks randomly failure.

[bacongobbler]

An i/o timeout likely means that the connection to azure timed out or the connection was lost. Third party object stores (Azure in particular) are known for latency issues from time to time. Can you try pushing again and confirm that your network connection to Azure Blob Storage is stable?

Also, can you confirm that you're connecting to a datacenter close to your approximate location? Is it correct to assume that your cluster is based near Japan? According to https://azure.microsoft.com/en-us/regions/services/ there appears to be both Japan west and east regions available for Azure Blob Storage.

[monaka]

Both Storage and VM are in same region (Japan east).
The storage grade is Standard-LRS.
And there is no system failure report by Azure.

It is just a guess for now. But it may be caused DNS related issues.

[monaka]

I got verbose logs by kube-dns/dnsmasq.

It was queried not only blob.core.windows.net but also blob.core.windows.net.cluster.local.
Is it expected behavior?

dnsmasq: query[A] {censored}.blob.core.windows.net.cluster.local from 172.16.12.25
dnsmasq: cached {censored}.blob.core.windows.net.cluster.local is NXDOMAIN
dnsmasq: query[AAAA] {censored}.blob.core.windows.net.cluster.local from 172.16.12.25
dnsmasq: cached {censored}.blob.core.windows.net.cluster.local is NXDOMAIN
dnsmasq: query[AAAA] {censored}.blob.core.windows.net from 172.16.12.25
dnsmasq: cached {censored}.blob.core.windows.net is <CNAME>
dnsmasq: cached blob.ty1prdstr03a.store.core.windows.net is NODATA-IPv6
dnsmasq: query[A] {censored}.blob.core.windows.net from 172.16.12.25
dnsmasq: cached {censored}.blob.core.windows.net is <CNAME>
dnsmasq: cached blob.ty1prdstr03a.store.core.windows.net is {IP at Azure}
dnsmasq: query[A] {censored}.blob.core.windows.net.cluster.local from 172.16.12.4
dnsmasq: cached {censored}.blob.core.windows.net.cluster.local is NXDOMAIN
dnsmasq: query[AAAA] {censored}.blob.core.windows.net.cluster.local from 172.16.12.4
dnsmasq: cached {censored}.blob.core.windows.net.cluster.local is NXDOMAIN
dnsmasq: query[AAAA] {censored}.blob.core.windows.net from 172.16.12.4

[bacongobbler]

Yep! That's expected behaviour within kubernetes as the DNS lookup within the containers first tries .cluster.local before going outwards, in case there's a service provided that resolves blob.core.windows.net.cluster.local.

[monaka]

@bacongobbler Thank you for your reply. I see Kube looks sane.
Then suspection points are "DNS throttling by Azure". ... or a bug in Alpine kubernetes/kubernetes#30215 ?.

[monaka]

I got it. In this case, the root cause was my misconfiguration.
kubedns/kubedns container was killed continuously by kubedns/healthz.
I close this issue as it is not by Builder. Thank you for your support.