devfile integration tests should run against Kubernetes cluster

[kadel]

Everything related to devfiles should work with Kubernetes as well as OpenShift clusters.

Integration tests need to be executed against Kubernetes as well as OpenShift cluster

/area devfile
/kind test
/area testing
/priority high

[johnmcollier]

@kadel I've been thinking about the same. As both a stopgap and the most free/easiest solution, I was thinking about bringing up Minikube in Travis, and running the devfile tests against it.

[kadel]

@kadel I've been thinking about the same. As both a stopgap and the most free/easiest solution, I was thinking about bringing up Minikube in Travis, and running the devfile tests against it.

+1
minikube start --vm-driver=none should work well on Travis.

[kadel]

The tricky part will be nicely structuring our tests as we want to keep testing both Kubernetes and OpenShift.

We have 2 groups of tests:

• tests that need to be executed with both Kubernetes and OpenShift (everything around devfile, except when Route is involved)
• tests that can be executed only with OpenShift cluster (legacy s2i components, and devfile component when Route is created)

[prietyc123]

We have raised a concern in test-platform whether we can get a kubernetes cluster using openshift CI/prow. However I have gone through some of the documents to setup the kubernetes cluster on travis CI.
There are certain configuration w.r.t Kubernetes which needs to be taken care like writing some bash script to start minikube for example : https://github.com/bitnami/kubernetes-travis/tree/minimal-minikube-branch/scripts .
As part of running test script is concerned, tests may need some restructure w.r.t to oc and kubectl commands, it can be handled smartly through env var IMO. @mohammedzee1000 @amitkrout

[girishramnani]

I would suggest to start on travis and not consider having a kubernetes cluster on openshift.

[amitkrout]

@prietyc123 refer https://minikube.sigs.k8s.io/docs/start/linux/ for how to install minikube on travis (Ubuntu VM). While installing make sure that you are using --driver=none . if travis does not support configuration for --driver=none (However there is an open issue kubernetes/minikube#2704) then use ``--driver=docker``` in your travis configuration script.

[prietyc123]

I would suggest to start on travis and not consider having a kubernetes cluster on openshift.

Yes, As OpenShift CI does not support such functionality (Got confirmation form test platform team), so as per the above discussion the only way left behind is travis CI.

[prietyc123]

• Setup Kubernetes cluster on travis CI and handle odo project create on Kubernetes and Openshift Setup Kubernetes cluster on travis CI and handle odo project create on Kubernetes and Openshift cluster #2809
• Running devfile catalog integration test on kubernetes cluster using travis CI Setup Kubernetes cluster on travis CI and handle odo project create on Kubernetes and Openshift cluster #2809
• Running devfile create integration test on kubernetes cluster using travis CI devfile create Interation test on kubernetes cluster using travis CI #2884
• Running devfile delete integration test on kubernetes cluster using travis CI devfile delete Integration test on kubernetes cluster #2913
• Running devfile push integration test on kubernetes cluster using travis CI devfile push Integration test on kubernetes cluster #3041
• Running devfile watch integration test on kubernetes cluster using travis CI devfile watch Interation test on kubernetes cluster using travis CI #2916

[prietyc123]

#2913 is blocked and issue has been covered in #2928

Issue has been shortly explained in #2913 (comment)

[prietyc123]

#2884 is also blocked. Refer comment : #2884 (comment)

[prietyc123]

pr #3041 and #2913 is blocked due to #2877 (comment)

[prietyc123]

Blocked due to issue #3055

[prietyc123]

Running devfile delete integration test on kubernetes cluster using travis CI #2913 and Running devfile push integration test on kubernetes cluster using travis CI #3041 is in review. Need to update some requested change for the proposed pr.

Running devfile create integration test on kubernetes cluster using travis CI #2884 is ready to get reviewed but on hold due to #3124. Once PR gets merged I will open it for review.

[prietyc123]

As per the comment I am going to open a separate pr to address the comment.

[prietyc123]

#2913 is blocked and issue has been covered in #2928

Fixes the issue #2928 with pr #3109 and unblocked #2913

[prietyc123]

#2884 is also blocked. Refer comment : #2884 (comment)

#2884 was failing due to #2888 and gets unblocked with pr #3124

[prietyc123]

This issue is in queue for a long time. And I have already proposed the prs related to this issue. It took time because I was getting lots of unknown issue while executing it and tried to resolve those one by one. While execution I tried to resolve racing condition with the namespace by copying the kubeconfig file to local temporary config file. Tried the same with openshift cluster and successfully executing those on travis CI.

But here comes the twist... When I am trying to copy the kubeconfig to the temp config on prow I am facing some other challenges Like image streams are not found in that temporary config. I am not sure how much unknown challenges will come up. And at the same time I don't want team to get blocked on running their tests on kubernetes cluster, as it is equally Important. More importantly all the tests are running on kubernetes cluster so its not the failure from test side. Its particularly failing on certain platform like prow.

IMO a possible solution is Lets handle the kubeconfig thing on travis CI for now and I will open a new pr to handle the current issue with kubeconfig on prow. I am suspecting it could be an infra issue. Anyway I will be working on that issue in this sprint itself. And in this way I guess going forward every individual will be able to run their pr on minikube.

[girishramnani]

• Setup Kubernetes cluster on travis CI and handle odo project create on Kubernetes and Openshift Setup Kubernetes cluster on travis CI and handle odo project create on Kubernetes and Openshift cluster #2809
• Running devfile catalog integration test on kubernetes cluster using travis CI Setup Kubernetes cluster on travis CI and handle odo project create on Kubernetes and Openshift cluster #2809
• Running devfile create integration test on kubernetes cluster using travis CI devfile create Interation test on kubernetes cluster using travis CI #2884
• Running devfile delete integration test on kubernetes cluster using travis CI devfile delete Integration test on kubernetes cluster #2913
• Running devfile push integration test on kubernetes cluster using travis CI devfile push Integration test on kubernetes cluster #3041
• Running devfile watch integration test on kubernetes cluster using travis CI devfile watch Interation test on kubernetes cluster using travis CI #2916

here