After configuring the Metal3 networks, Minikube cannot access k8s.gcr.io

[cgruver]

When building Metal3 on Centos 7, the following is displayed upon the second start of Minikube:

VM is unable to access k8s.gcr.io, you may need to configure a proxy or set --image-repository

As a result, the deployment does not proceed beyond:

   - Waiting for task completion (up to 2400 seconds)  - Command: 'check_k8s_entity statefulsets cluster-api-controller-manager   cluster-api-provider-baremetal-controller-manager'

Because of the network issue, you will see:

kubectl get pods --all-namespaces

 metal3        cluster-api-controller-manager-0                      0/1     ImagePullBackOff    0          5m52s
 metal3        cluster-api-provider-baremetal-controller-manager-0   0/2     ErrImagePull        0          5m52s
 metal3        metal3-baremetal-operator-688bb8c4c4-sxmhd            0/6     Init:ErrImagePull   0          5m52s

Accessing Minikube via minikube ssh and looking at network config reveals two default routes. I don't know if this has anything to do with it, but during the initial configuration of Minikube it was able to access the internet while just on the default KVM network.

0.0.0.0         192.168.122.1   0.0.0.0         UG        0 0          0 eth0
0.0.0.0         192.168.111.1   0.0.0.0         UG        0 0          0 eth3
172.17.0.0      0.0.0.0         255.255.0.0     U         0 0          0 docker0
172.22.0.0      0.0.0.0         255.255.255.0   U         0 0          0 eth2
192.168.39.0    0.0.0.0         255.255.255.0   U         0 0          0 eth1
192.168.111.0   0.0.0.0         255.255.255.0   U         0 0          0 eth3
192.168.111.1   0.0.0.0         255.255.255.255 UH        0 0          0 eth3
192.168.122.0   0.0.0.0         255.255.255.0   U         0 0          0 eth0
192.168.122.1   0.0.0.0         255.255.255.255 UH        0 0          0 eth0

[cgruver]

It's interesting to note that DNS resolution works.

$ ping yahoo.com
PING yahoo.com (98.138.219.231): 56 data bytes
^C
--- yahoo.com ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss

Feels like either a routing issue, or a firewall sitting in the way somewhere....

[cgruver]

I noticed that the shell function init_minikube is called at the end of 01_prepare_host.sh before all of the networks are configured. It is then called again in 03_launch_mgmt_cluster.sh.

By commenting out the invocation of init_minikube in 01_prepare_host.sh, it seems to work now.

The 04_verify.sh script completed with no errors:

OK - worker-0 Baremetalhost exist
OK - worker-0 Baremetalhost address correct
OK - worker-0 Baremetalhost mac address correct
OK - worker-0 Baremetalhost status OK
OK - worker-0 Baremetalhost credentials secret exist
OK - worker-0 Baremetalhost password correct
OK - worker-0 Baremetalhost user correct
OK - worker-0 Baremetalhost VM exist
OK - worker-0 Baremetalhost VM interface provisioning exist
OK - worker-0 Baremetalhost VM interface baremetal exist
OK - worker-0 Baremetalhost introspecting completed

OK - Container httpd running. 
OK - Container registry running
OK - Container vbmc running
OK - Container sushy-tools running

Number of failures : 0

[alosadagrande]

Same here, workaround worked as suggested by @cgruver (thanks!). Why is it called init_minikube twice?

Also, the deployment is configured as imagePullPolicy to Always. Since we already downloaded many images in the previous step locally to the minikube instance, seems to me more efficient to change it to ifNotPresent - seems reasonable?

[stbenjam]

Same here, workaround worked as suggested by @cgruver (thanks!). Why is it called init_minikube twice?

It was an optimization for CI, but there might be some bug here we need to investigate why this is causing problems for some people.

Also, the deployment is configured as imagePullPolicy to Always. Since we already downloaded many images in the previous step locally to the minikube instance, seems to me more efficient to change it to ifNotPresent - seems reasonable?

Could you file a separate issue for this? Thanks!

[metal3-io-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues will close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[metal3-io-bot]

Stale issues close after 30d of inactivity. Reopen the issue with /reopen. Mark the issue as fresh with /remove-lifecycle stale.

/close

[metal3-io-bot]

@metal3-io-bot: Closing this issue.

In response to this:

Stale issues close after 30d of inactivity. Reopen the issue with /reopen. Mark the issue as fresh with /remove-lifecycle stale.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.