option to pass custom provision file while (make cluster-up)

[kunalkushwaha]

/kind enhancement

Problem:
while working behind network proxy, make cluster-up fails to setup cluster while downloading CNI config. Also, no docker pull or kubectl apply is successful as no image gets downloaded due to lack of proxy settings in docker-daemon.

.
.
Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of machines by running the following on each node
as root:

  kubeadm join 192.168.66.101:6443 --token abcdef.1234567890123456 --discovery-token-ca-cert-hash sha256:23d5e3383df6878d799c4b326f8910d521894dc9a403dcac9dd6a3800236fac3

+ kubectl --kubeconfig=/etc/kubernetes/admin.conf apply -f https://raw.githubusercontent.com/coreos/flannel/v0.9.1/Documentation/kube-flannel.yml
The connection to the server raw.githubusercontent.com was refused - did you specify the right host or port?
provisioning node node01 failed
Makefile:104: recipe for target 'cluster-up' failed
make: *** [cluster-up] Error 1

Expected Behaviour

• modify provision scripts like node.sh or node<node-no>.sh etc.
• gocli run should be able to consume custom scripts if indicated explicitly

[rmohr]

kubectl --kubeconfig=/etc/kubernetes/admin.conf apply -f https://raw.githubusercontent.com/coreos/flannel/v0.9.1/Documentation/kube-flannel.yml

@kunalkushwaha that's an issue in the base scripts. They should never download on teprovision state anything. They should work offline by never pulling on restart of VMIs.

[kunalkushwaha]

#71 fixes issue partially for using make cluster-up script without change behind proxy. To create VMI's docker daemon should be able to pull images, which requires proxy settings for docker daemon.

This issue may be fixed with special flag for --proxy-settings instead of passing all scripts once more, but I feel #70 gives ability to customize environment, if anything specific required for setup without any changes in scripts in upstream code.

[rmohr]

#71 fixes issue partially for using make cluster-up script without change behind proxy. To create VMI's docker daemon should be able to pull images, which requires proxy settings for docker daemon.

Just keep in mind that we extra have a registry included if you do "make cluster-up". You can also push all images which should be available for tests to that registry. It makes tests more reliable because the images are then very fast downloaded.

This issue may be fixed with special flag for --proxy-settings instead of passing all scripts once more, but I feel #70 gives ability to customize environment, if anything specific required for setup without any changes in scripts in upstream code.

I am not against the hook but it kind of goes against the purpose of kubevirtci. The purpose is that you have a completely pre-canned and prepared cluster. Setting correct proxy settings or something like this for sure make sense but the purpose is exactly to not customize the clusters, since it is there to provide a base for reproducible testruns. Locally and on CI.

[rmohr]

Also we have some tests in kubevirt/kubevirt which reach out to the internet (e.g. run yum). That are buggy tests. They should never have to reach the internet (except the one test which ensures that VMIs can reach the internet in general, which can be skipped easily).

[kunalkushwaha]

The purpose is that you have a completely pre-canned and prepared cluster. Setting correct proxy settings or something like this for sure make sense but the purpose is exactly to not customize the clusters, since it is there to provide a base for reproducible testruns. Locally and on CI.

I agree with you and understand customizing setup goes against kubevertci original goal.
My test VMI images resides in local network registry, but requires proxy settings for accessing that too.

Will be it okay, If I modify PR #70 just to set docker daemon proxy settings?. It will solve issue working behind proxy.

[rmohr]

Will be it okay, If I modify PR #70 just to set docker daemon proxy settings?. It will solve issue working behind proxy.

Works for me.

My test VMI images resides in local network registry, but requires proxy settings for accessing that too.

Let me just say again that you can also push them to the included registry, which will result in more reliable test runs.

[kunalkushwaha]

My test VMI images resides in local network registry, but requires proxy settings for accessing that too.

Let me just say again that you can also push them to the included registry, which will result in more reliable test runs.

The VMI images are pushed by others too. Though it should be doable, but proxy will be helpful. Right now I am not using it for CI tests, so its more of a dev environment setup for finding issues for specific VMIs

Will be it okay, If I modify PR #70 just to set docker daemon proxy settings?. It will solve issue working behind proxy.

Works for me.

Thanks, I will update the PR for just proxy settings.