Move from the kubernetes cluster to oc cluster up cluster

[surajssd]

Right now e2e tests on semaphore CI run on a Kubernetes cluster running in a VM, we should move to Kubernetes cluster which is spawned using oc cluster up, so that adding tests for openshift objects become easier.

[cdrage]

So first test right now seems to fail (I believe because we cannot create multiple namespaces using kubectl with the OpenShift cluster...)

--- FAIL: Test_Integration (0.03s)
    --- FAIL: Test_Integration/Testing_configMap (0.02s)
        e2e_test.go:398: namespace "configmap" created
        e2e_test.go:404: error running kedge: error running "command: create -n configmap -f /home/wikus/dropbox/dev/go/src/github.com/kedgeproject/kedge/docs/examples/configmap/db.yaml -f /home/wikus/dropbox/dev/go/src/github.com/kedgeproject/kedge/docs/examples/configmap/web.yaml"
                 fork/exec /home/wikus/dropbox/dev/go/src/github.com/kedgeproject/kedge/kedge: permission denied
        e2e_test.go:237: successfully deleted namespace: "configmap"
    --- FAIL: Test_Integration/Testing_secret (0.04s)
        e2e_test.go:398: namespace "secrets" created
        e2e_test.go:404: error running kedge: error running "command: create -n secrets -f /home/wikus/dropbox/dev/go/src/github.com/kedgeproject/kedge/docs/examples/secrets/db.yaml -f /home/wikus/dropbox/dev/go/src/github.com/kedgeproject/kedge/docs/examples/secrets/web.yaml"
                 fork/exec /home/wikus/dropbox/dev/go/src/github.com/kedgeproject/kedge/kedge: permission denied
        e2e_test.go:237: successfully deleted namespace: "secrets"
    --- FAIL: Test_Integration/Test_portMappings (0.04s)
        e2e_test.go:398: namespace "portmappings" created
        e2e_test.go:404: error running kedge: error running "command: create -n portmappings -f /home/wikus/dropbox/dev/go/src/github.com/kedgeproject/kedge/docs/examples/portMappings/wordpress.yaml -f /home/wikus/dropbox/dev/go/src/github.com/kedgeproject/kedge/docs/examples/portMappings/mariadb.yaml"
                 fork/exec /home/wikus/dropbox/dev/go/src/github.com/kedgeproject/kedge/kedge: permission denied
        e2e_test.go:237: successfully deleted namespace: "portmappings"
    --- FAIL: Test_Integration/Test_jobs (0.04s)
        e2e_test.go:398: namespace "jobs" created
        e2e_test.go:404: error running kedge: error running "command: create -n jobs -f /home/wikus/dropbox/dev/go/src/github.com/kedgeproject/kedge/docs/examples/jobs/job.yaml"
                 fork/exec /home/wikus/dropbox/dev/go/src/github.com/kedgeproject/kedge/kedge: permission denied
        e2e_test.go:237: successfully deleted namespace: "jobs"
    --- FAIL: Test_Integration/Testing_health (0.16s)
        e2e_test.go:398: namespace "health" created
        e2e_test.go:404: error running kedge: error running "command: create -n health -f /home/wikus/dropbox/dev/go/src/github.com/kedgeproject/kedge/docs/examples/health/db.yaml -f /home/wikus/dropbox/dev/go/src/github.com/kedgeproject/kedge/docs/examples/health/web.yaml"
                 fork/exec /home/wikus/dropbox/dev/go/src/github.com/kedgeproject/kedge/kedge: permission denied
        e2e_test.go:237: successfully deleted namespace: "health"
    --- FAIL: Test_Integration/Normal_Wordpress_test (1.14s)
        e2e_test.go:398: namespace "wordpress" created
        e2e_test.go:404: error running kedge: error running "command: create -n wordpress -f /home/wikus/dropbox/dev/go/src/github.com/kedgeproject/kedge/examples/wordpress/wordpress.yaml -f /home/wikus/dropbox/dev/go/src/github.com/kedgeproject/kedge/examples/wordpress/mariadb.yaml"
                 fork/exec /home/wikus/dropbox/dev/go/src/github.com/kedgeproject/kedge/kedge: permission denied
        e2e_test.go:237: successfully deleted namespace: "wordpress"
    --- FAIL: Test_Integration/Testing_healthChecks (1.34s)
        e2e_test.go:398: namespace "healthchecks" created
        e2e_test.go:404: error running kedge: error running "command: create -n healthchecks -f /home/wikus/dropbox/dev/go/src/github.com/kedgeproject/kedge/docs/examples/healthchecks/db.yaml -f /home/wikus/dropbox/dev/go/src/github.com/kedgeproject/kedge/docs/examples/healthchecks/web.yaml"
                 fork/exec /home/wikus/dropbox/dev/go/src/github.com/kedgeproject/kedge/kedge: permission denied
        e2e_test.go:237: successfully deleted namespace: "healthchecks"
    --- FAIL: Test_Integration/Testing_includeResources (1.53s)
        e2e_test.go:398: namespace "include-resources" created
        e2e_test.go:404: error running kedge: error running "command: create -n include-resources -f /home/wikus/dropbox/dev/go/src/github.com/kedgeproject/kedge/docs/examples/includeResources/app.yaml"
                 fork/exec /home/wikus/dropbox/dev/go/src/github.com/kedgeproject/kedge/kedge: permission denied
        e2e_test.go:237: successfully deleted namespace: "include-resources"
    --- FAIL: Test_Integration/Testing_single_file (1.60s)
        e2e_test.go:398: namespace "singlefile" created
        e2e_test.go:404: error running kedge: error running "command: create -n singlefile -f /home/wikus/dropbox/dev/go/src/github.com/kedgeproject/kedge/docs/examples/single_file/wordpress.yml"
                 fork/exec /home/wikus/dropbox/dev/go/src/github.com/kedgeproject/kedge/kedge: permission denied
        e2e_test.go:237: successfully deleted namespace: "singlefile"
    --- FAIL: Test_Integration/Testing_customVol (1.00s)
        e2e_test.go:398: namespace "customvol" created
        e2e_test.go:404: error running kedge: error running "command: create -n customvol -f /home/wikus/dropbox/dev/go/src/github.com/kedgeproject/kedge/docs/examples/customVol/db.yaml -f /home/wikus/dropbox/dev/go/src/github.com/kedgeproject/kedge/docs/examples/customVol/web.yaml"
                 fork/exec /home/wikus/dropbox/dev/go/src/github.com/kedgeproject/kedge/kedge: permission denied
        e2e_test.go:237: successfully deleted namespace: "customvol"
FAIL
exit status 1
FAIL    github.com/kedgeproject/kedge/tests/e2e 2.253s
Makefile:82: recipe for target 'test-e2e' failed
make: *** [test-e2e] Error 1

[surajssd]

@cdrage login to the cluster as system:admin, a normal user cannot create multiple ns in openshift by default.

[kadel]

@cdrage login to the cluster as system:admin, a normal user cannot create multiple ns in openshift by default.

don't do that ;-) it is like running commands as root.

Normal user can create multiple namespaces/project, using oc new-project

[cdrage]

@surajssd @kadel

Already tried using oc login, doesn't do anything / still can't create namespaces (via kubectl). We'd most likely have to modify e2e_test.go

[kadel]

Already tried using oc login, doesn't do anything / still can't create namespaces (via kubectl).

it will not work with kubectl, oc does some magic so it is able to create namesaces without admin privileges. you have to use oc new-project command

[surajssd]

@kadel the test suite uses client-go so it creates namespaces and there is no going away from it, because all things happen after that and are heavily dependent on namespace.

[kadel]

@surajssd
hmm :-( than the only option might be to switch to system:admin as you said.

But please make sure that you switch back to regular user after namespace is created.
I think that some permission setting will be also required for newly created namespace.

Please don't run tests as system:admin

[cdrage]

After some discussion we are going to split-up the tests to run in two separate instances.

• 1: Using minikube
• 2: Using minishift or oc cluster up

Both using the same set of examples (with the exception of OpenShift-specific ones using DeploymentConfig).

[surajnarwade]

Since we have descent test framework on semaphore CI for both kubernetes and openshift and it takes less time too. are we done with this issue ? can we close this ? @surajssd @cdrage @kadel

[cdrage]

Yeah, this is good now 👍 Kubernetes is being ran under minikube with --vm-driver=none (using Docker) and OpenShift is using minishift with --vm-driver=virtualbox.

Tests are also passing in less than 20 minutes which is better than what we had before (over 45 minutes!)