DNS issues in created containers (oc cluster up, v1.4.0-rc1 on Fedora 24)

[nicolaferraro]

The oc cluster up command on Fedora 24 with Openshift Client version v1.4.0-rc1 produces a wrong DNS configuration.

When starting up, I see the warning (it should be related to external resolution.. what fails is the internal one):

WARNING: Binding DNS on port 8053 instead of 53

The /etc/resolv.conf file of the registry pod looks like:

search default.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.33.64.180
options ndots:5

With this config I'm not able to resolve any internet address.

I've not checked the build pods created when using S2I, but all the builds fail with error: Could not resolve host: github.com; Unknown error, so there's probably a DNS issue also in the S2I build containers.

The /etc/resolv.conf file in the origin pod contains the following config:

# Generated by NetworkManager
search rom.redhat.com redhat.com
nameserver 10.11.5.19
nameserver 10.5.30.160

(that is the config of the host machine).

github.com is resolvable from the origin pod, but cannot be resolved by the registry or by any build pod.

Version

oc v1.4.0-rc1+b4e0954
kubernetes v1.4.0+776c994
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://10.33.64.180:8443
openshift v1.4.0-rc1+b4e0954
kubernetes v1.4.0+776c994

Steps To Reproduce

1. oc cluster up
2. login to the GUI
3. Click on "Add to project"
4. Select "Python"
5. Select version "3.5" of the Python app
6. Enter a name and use the default Github repository
7. Look at the build logs

Current Result

Cloning "https://github.com/openshift/django-ex.git" ...
error: build error: fatal: unable to access 'https://github.com/openshift/django-ex.git/': Could not resolve host: github.com; Unknown error

Expected Result

Successful build.

Additional Information

oadm diagnostic:

Note] Determining if client configuration exists for client/cluster diagnostics
Info:  Successfully read a client config file at '/home/nferraro/.kube/config'
Info:  Using context for cluster-admin access: 'default/10-33-64-180:8443/system:admin'

[Note] Running diagnostic: ConfigContexts[brms2/192-168-99-100:8443/admin]
       Description: Validate client config context is complete and has connectivity

Build with BUILD_LOGLEVEL=5:

I1202 11:56:14.205583       1 builder.go:53] $BUILD env var is {"kind":"Build","apiVersion":"v1","metadata":{"name":"example-2","namespace":"myproject","selfLink":"/oapi/v1/namespaces/myproject/builds/example-2","uid":"514a0f99-b886-11e6-b8f7-507b9dcf1324","resourceVersion":"776","creationTimestamp":"2016-12-02T11:56:11Z","labels":{"app":"example","buildconfig":"example","openshift.io/build-config.name":"example","openshift.io/build.start-policy":"Serial"},"annotations":{"openshift.io/build-config.name":"example","openshift.io/build.number":"2"}},"spec":{"serviceAccount":"builder","source":{"type":"Git","git":{"uri":"https://github.com/openshift/django-ex.git","ref":"master"}},"strategy":{"type":"Source","sourceStrategy":{"from":{"kind":"DockerImage","name":"centos/python-35-centos7@sha256:4696720a1783ebeebc0d078b5dead8d86688f959e943d486d1340b7c7d083dca"},"env":[{"name":"BUILD_LOGLEVEL","value":"5"}]}},"output":{"to":{"kind":"DockerImage","name":"172.30.131.184:5000/myproject/example:latest"},"pushSecret":{"name":"builder-dockercfg-pp74d"}},"resources":{},"postCommit":{},"nodeSelector":null,"triggeredBy":[{"message":"Manually triggered"}]},"status":{"phase":"New","outputDockerImageReference":"172.30.131.184:5000/myproject/example:latest","config":{"kind":"BuildConfig","namespace":"myproject","name":"example"}}}
 
I1202 11:56:14.207381       1 builder.go:61] Build: &api.Build{TypeMeta:unversioned.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:api.ObjectMeta{Name:"example-2", GenerateName:"", Namespace:"myproject", SelfLink:"/oapi/v1/namespaces/myproject/builds/example-2", UID:"514a0f99-b886-11e6-b8f7-507b9dcf1324", ResourceVersion:"776", Generation:0, CreationTimestamp:unversioned.Time{Time:time.Time{sec:63616276571, nsec:0, loc:(*time.Location)(0x78e5c40)}}, DeletionTimestamp:(*unversioned.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"openshift.io/build.start-policy":"Serial", "app":"example", "buildconfig":"example", "openshift.io/build-config.name":"example"}, Annotations:map[string]string{"openshift.io/build.number":"2", "openshift.io/build-config.name":"example"}, OwnerReferences:[]api.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:api.BuildSpec{CommonSpec:api.CommonSpec{ServiceAccount:"builder", Source:api.BuildSource{Binary:(*api.BinaryBuildSource)(nil), Dockerfile:(*string)(nil), Git:(*api.GitBuildSource)(0xc420806ec0), Images:[]api.ImageSource(nil), ContextDir:"", SourceSecret:(*api.LocalObjectReference)(nil), Secrets:[]api.SecretBuildSource(nil)}, Revision:(*api.SourceRevision)(nil), Strategy:api.BuildStrategy{DockerStrategy:(*api.DockerBuildStrategy)(nil), SourceStrategy:(*api.SourceBuildStrategy)(0xc420c43ba0), CustomStrategy:(*api.CustomBuildStrategy)(nil), JenkinsPipelineStrategy:(*api.JenkinsPipelineBuildStrategy)(nil)}, Output:api.BuildOutput{To:(*api.ObjectReference)(0xc420351b20), PushSecret:(*api.LocalObjectReference)(0xc4207c1fa0), ImageLabels:[]api.ImageLabel(nil)}, Resources:api.ResourceRequirements{Limits:api.ResourceList(nil), Requests:api.ResourceList(nil)}, PostCommit:api.BuildPostCommitSpec{Command:[]string(nil), Args:[]string(nil), Script:""}, CompletionDeadlineSeconds:(*int64)(nil), NodeSelector:map[string]string(nil)}, TriggeredBy:[]api.BuildTriggerCause{api.BuildTriggerCause{Message:"Manually triggered", GenericWebHook:(*api.GenericWebHook
Cause)(nil), GitHubWebHook:(*api.GitHubWebHookCause)(nil), ImageChangeBuild:(*api.ImageChangeCause)(nil)}}}, Status:api.BuildStatus{Phase:"New", Cancelled:false, Reason:"", Message:"", StartTimestamp:(*unversioned.Time)(nil), CompletionTimestamp:(*unversioned.Time)(nil), Duration:0, OutputDockerImageReference:"172.30.131.184:5000/myproject/example:latest", Config:(*api.ObjectReference)(0xc420351ce0)}}
I1202 11:56:14.207509       1 builder.go:68] Master version "v1.4.0-rc1+b4e0954", Builder version "v1.4.0-rc1+b4e0954"
I1202 11:56:14.207873       1 builder.go:160] Running build with cgroup limits: api.CGroupLimits{MemoryLimitBytes:92233720368547, CPUShares:2, CPUPeriod:100000, CPUQuota:-1, MemorySwap:92233720368547}
I1202 11:56:14.208641       1 sti.go:204] With force pull false, setting policies to if-not-present
I1202 11:56:14.208662       1 sti.go:211] The value of ALLOWED_UIDS is [1-]
I1202 11:56:14.208688       1 sti.go:219] The value of DROP_CAPS is [KILL,MKNOD,SETGID,SETUID,SYS_CHROOT]
I1202 11:56:14.208769       1 cfg.go:44] Locating docker auth for image centos/python-35-centos7@sha256:4696720a1783ebeebc0d078b5dead8d86688f959e943d486d1340b7c7d083dca and type PULL_DOCKERCFG_PATH
I1202 11:56:14.208834       1 cfg.go:112] Using Docker authentication configuration in '/root/.docker/config.json'
I1202 11:56:14.208897       1 cfg.go:56] Problem accessing /root/.docker/config.json: stat /root/.docker/config.json: no such file or directory
I1202 11:56:14.208907       1 cfg.go:44] Locating docker auth for image 172.30.131.184:5000/myproject/example:latest and type PUSH_DOCKERCFG_PATH
I1202 11:56:14.208927       1 cfg.go:112] Using Docker authentication configuration in '/var/run/secrets/openshift.io/push/.dockercfg'
I1202 11:56:14.209107       1 cfg.go:79] Using serviceaccount user for Docker authentication for image 172.30.131.184:5000/myproject/example:latest
I1202 11:56:14.211182       1 docker.go:515] Using locally available image "centos/python-35-centos7@sha256:4696720a178..."
I1202 11:56:14.212414       1 sti.go:247] Creating a new S2I builder with build config: "Builder Name:\t\t\tPython 3.5\nBuilder Image:\t\t\tcentos/python-35-centos7@sha256:4696720a1783ebeebc0d078b5dead8d86688f959e943d486d1340b7c7d083dca\nBuilder Image Version:\t\tbc1241345c4866ffedc46b3816a276c4d898761a\nBuilder Base Version:\t\ta8deee2\nSource:\t\t\t\tfile:///tmp/s2i-build529532182/upload/src#master\nOutput Image Tag:\t\tmyproject/example-2:a6d06f85\nEnvironment:\t\t\tOPENSHIFT_BUILD_NAME=example-2,OPENSHIFT_BUILD_NAMESPACE=myproject,OPENSHIFT_BUILD_SOURCE=https://github.com/openshift/django-ex.git,OPENSHIFT_BUILD_REFERENCE=master,BUILD_LOGLEVEL=5\nLabels:\t\t\t\t\nIncremental Build:\t\tdisabled\nRemove Old Build:\t\tdisabled\nBuilder Pull Policy:\t\tif-not-present\nPrevious Image Pull Policy:\talways\nQuiet:\t\t\t\tdisabled\nLayered Build:\t\t\tdisabled\nWorkdir:\t\t\t/tmp/s2i-build529532182\nDocker NetworkMode:\t\tcontainer:4b7cdfb06be4d13ea563f358594fd639224c2535a757d518f4dd1afdd1cec174\nDocker Endpoint:\t\tunix:///var/run/docker.sock\n"
I1202 11:56:14.213691       1 docker.go:515] Using locally available image "centos/python-35-centos7@sha256:4696720a178..."
I1202 11:56:14.218296       1 docker.go:515] Using locally available image "centos/python-35-centos7@sha256:4696720a178..."
I1202 11:56:14.218356       1 docker.go:726] Image sha256:df571e928aaf1479166ad8afd03f68381a14184ebc6a0b02ef536b324511e93b contains io.openshift.s2i.scripts-url set to "image:///usr/libexec/s2i"
I1202 11:56:14.218574       1 sti.go:253] Starting S2I build from myproject/example-2 BuildConfig ...
I1202 11:56:14.218594       1 sti.go:184] Preparing to build myproject/example-2:a6d06f85
Cloning "https://github.com/openshift/django-ex.git" ...
I1202 11:56:14.219430       1 source.go:122] git ls-remote --heads https://github.com/openshift/django-ex.git
I1202 11:56:14.219455       1 repository.go:367] Executing GIT_ASKPASS=true PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=example-2-build BUILD={"kind":"Build","apiVersion":"v1","metadata":{"name":"example-2","namespace":"myproject","selfLink":"/oapi/v1/namespaces/myproject/builds/example-2","uid":"514a0f99-b886-11e6-b8f7-507b9dcf1324","resourceVersion":"776","creationTimestamp":"2016-12-02T11:56:11Z","labels":{"app":"example","buildconfig":"example","openshift.io/build-config.name":"example","openshift.io/build.start-policy":"Serial"},"annotations":{"openshift.io/build-config.name":"example","openshift.io/build.number":"2"}},"spec":{"serviceAccount":"builder","source":{"type":"Git","git":{"uri":"https://github.com/openshift/django-ex.git","ref":"master"}},"strategy":{"type":"Source","sourceStrategy":{"from":{"kind":"DockerImage","name":"centos/python-35-centos7@sha256:4696720a1783ebeebc0d078b5dead8d86688f959e943d486d1340b7c7d083dca"},"env":[{"name":"BUILD_LOGLEVEL","value":"5"}]}},"output":{"to":{"kind":"DockerImage","name":"172.30.131.184:5000/myproject/example:latest"},"pushSecret":{"name":"builder-dockercfg-pp74d"}},"resources":{},"postCommit":{},"nodeSelector":null,"triggeredBy":[{"message":"Manually triggered"}]},"status":{"phase":"New","outputDockerImageReference":"172.30.131.184:5000/myproject/example:latest","config":{"kind":"BuildConfig","namespace":"myproject","name":"example"}}}
 SOURCE_REPOSITORY=https://github.com/openshift/django-ex.git SOURCE_URI=https://github.com/openshift/django-ex.git SOURCE_REF=master ORIGIN_VERSION=v1.4.0-rc1+b4e0954 BUILD_LOGLEVEL=5 ALLOWED_UIDS=1- DROP_CAPS=KILL,MKNOD,SETGID,SETUID,SYS_CHROOT PUSH_DOCKERCFG_PATH=/var/run/secrets/openshift.io/push EXAMPLE_SERVICE_HOST=172.30.13.139 EXAMPLE_PORT_8080_TCP=tcp://172.30.13.139:8080 EXAMPLE_PORT_8080_TCP_PORT=8080 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT_53_UDP_PROTO=udp EXAMPLE_PORT_8080_TCP_ADDR=172.30.13.139 KUBERNETES_PORT=tcp://172.30.0.1:443 KUBERNETES_PORT_443_TCP_PROTO=tcp KUBERNETES_PORT_53_TCP_ADDR=172.30.0.1 KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_SERVICE_PORT_DNS=53 EXAMPLE_SERVICE_PORT=8080 EXAMPLE_SERVICE_PORT_8080_TCP=8080 EXAMPLE_PORT=tcp://172.30.13.139:8080 EXAMPLE_PORT_8080_TCP_PROTO=tcp KUBERNETES_SERVICE_HOST=172.30.0.1 KUBERNETES_SERVICE_PORT=443 KUBERNETES_PORT_443_TCP=tcp://172.30.0.1:443 KUBERNETES_PORT_53_UDP_PORT=53 KUBERNETES_PORT_53_TCP=tcp://172.30.0.1:53 KUBERNETES_PORT_53_TCP_PORT=53 KUBERNETES_SERVICE_PORT_DNS_TCP=53 KUBERNETES_PORT_443_TCP_ADDR=172.30.0.1 KUBERNETES_PORT_53_UDP=udp://172.30.0.1:53 KUBERNETES_PORT_53_UDP_ADDR=172.30.0.1 KUBERNETES_PORT_53_TCP_PROTO=tcp HOME=/root OPENSHIFT_CONTAINERIZED=true KUBECONFIG=/var/lib/origin/openshift.local.config/master/admin.kubeconfig git ls-remote --heads https://github.com/openshift/django-ex.git
I1202 11:56:14.239126       1 repository.go:435] Error executing command: exit status 128
I1202 11:56:14.239269       1 source.go:128] fatal: unable to access 'https://github.com/openshift/django-ex.git/': Could not resolve host: github.com; Unknown error
I1202 11:56:14.239307       1 cleanup.go:33] Removing temporary directory /tmp/s2i-build529532182
I1202 11:56:14.239326       1 fs.go:159] Removing directory '/tmp/s2i-build529532182'
F1202 11:56:14.239918       1 helpers.go:115] error: build error: fatal: unable to access 'https://github.com/openshift/django-ex.git/': Could not resolve host: github.com; Unknown error

[csrwng]

@nicolaferraro it's really not a DNS issue. It's an iptables issue. Workaround is to use run iptables -F before running oc cluster up. Dupe of #10139