Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add possibility to deploy OpenShift in vagrant machine #631

Merged
merged 11 commits into from Jan 31, 2018

Conversation

@cynepco3hahue
Copy link
Member

cynepco3hahue commented Jan 4, 2018

Deployment works fine, but VM failed to start with the error:
5m 5m 24 virt-handler, node0 Warning SyncFailed virError(Code=1, Domain=10, Message='internal error: process exited while connecting to monitor: libvirt: error : cannot execute binary /usr/local/bin/qemu-system-x86_64: Permission denied')

Signed-off-by: Lukianov Artyom alukiano@redhat.com

@cynepco3hahue

This comment has been minimized.

Copy link
Member Author

cynepco3hahue commented Jan 4, 2018

@berrange Do you have any thought why can it fail? Under our Dockerfile we copy it

COPY qemu-kube /usr/local/bin/qemu-system-x86_64
RUN chmod a+x /usr/local/bin/qemu-system-x86_64
ENV PATH /usr/local/bin:/bin:/sbin

I believe it connects to file owner or to selinux permissions.

@rmohr

This comment has been minimized.

Copy link
Member

rmohr commented Jan 5, 2018

@cynepco3hahue nice. I recently worked on #611 to formalise bringing in new dev/testing environment providers. Could you have a look there and maybe already rebase on top of it? I think having the same Vagrantfile for the deployment is nice, but I would opt for having a completely separate provider for the rest (scripts, ...). Maybe "vagrant-openshift"? Have a look at https://github.com/rmohr/kubevirt/blob/4f615c18a2906e87323cfbfdc567d0cc56e3935f/docs/env-providers.md, to see on how you can implement the provider.

@cynepco3hahue

This comment has been minimized.

Copy link
Member Author

cynepco3hahue commented Jan 7, 2018

@rmohr Thanks, will check it.

@cynepco3hahue cynepco3hahue force-pushed the cynepco3hahue:vagrant_deploy_openshift branch 2 times, most recently from 60acc86 to d9bc79c Jan 8, 2018
@cynepco3hahue

This comment has been minimized.

Copy link
Member Author

cynepco3hahue commented Jan 8, 2018

After SELinux problems with context, I receive new warning

Events:
  FirstSeen     LastSeen        Count   From                    SubObjectPath   Type            Reason          Message
  ---------     --------        -----   ----                    -------------   --------        ------          -------
  5m            5m              1       virt-handler, master                    Normal          Created         VM defined.
  4m            22s             7       virt-handler, master                    Warning         SyncFailed      virError(Code=38, Domain=10, Message='monitor socket did not show up: No such file or directory')
  22s           12s             6       virt-handler, master                    Warning         SyncFailed      dial unix /var/run/kubevirt/sockets/kube-system_testvm_sock: connect: no such file or directory
  4s            4s              1       virt-handler, master                    Normal          Deleted         VM undefined
  4s            4s              1       virt-handler, master                    Warning         Stopped         The VM crashed.

It can be SELinux problems again, will continue to investigate it.
I think we need to start work with SELinux enabled, it will save us a lot of nerves in the future 😢

@cynepco3hahue cynepco3hahue force-pushed the cynepco3hahue:vagrant_deploy_openshift branch from ef74993 to ff1af46 Jan 9, 2018
@cynepco3hahue

This comment has been minimized.

Copy link
Member Author

cynepco3hahue commented Jan 9, 2018

@rmohr I opened a separate issue for selinux problems #639 and set the permissive mode for OpenShift environment. I ran tests locally and most tests succeeded to pass.
But replicaset tests fail consistently.

@cynepco3hahue

This comment has been minimized.

Copy link
Member Author

cynepco3hahue commented Jan 10, 2018

After the fix #642 I still have three consistent failures:

• Failure [63.390 seconds]
VirtualMachineReplicaSet
/home/alukiano/go/src/kubevirt.io/kubevirt/tests/replicaset_test.go:209
  A valid VirtualMachineReplicaSet given
  /home/alukiano/go/src/kubevirt.io/kubevirt/tests/replicaset_test.go:208
    should remove replicas once the VM is marked for deletion [It]
    /home/alukiano/go/src/kubevirt.io/kubevirt/tests/replicaset_test.go:123

    Timed out after 60.000s.
    Expected
        <int>: 2
    to be zero-valued

    /home/alukiano/go/src/kubevirt.io/kubevirt/tests/replicaset_test.go:122
------------------------------
• Failure [1.993 seconds]
VirtualMachineReplicaSet
/home/alukiano/go/src/kubevirt.io/kubevirt/tests/replicaset_test.go:209
  A valid VirtualMachineReplicaSet given
  /home/alukiano/go/src/kubevirt.io/kubevirt/tests/replicaset_test.go:208
    should remove owner references on the VM if it is orphan deleted [It]
    /home/alukiano/go/src/kubevirt.io/kubevirt/tests/replicaset_test.go:156

    Expected
        <[]v1.OwnerReference | len:1, cap:4>: [
            {
                APIVersion: "kubevirt.io/v1alpha1",
                Kind: "VirtualMachineReplicaSet",
                Name: "replicasetxdh7f",
                UID: "a5809fd2-f614-11e7-b7b9-525400f2ebe5",
                Controller: true,
                BlockOwnerDeletion: true,
            },
        ]
    to be empty

    /home/alukiano/go/src/kubevirt.io/kubevirt/tests/replicaset_test.go:153


•... Timeout [120.422 seconds]
RegistryDisk
/home/alukiano/go/src/kubevirt.io/kubevirt/tests/registry_disk_test.go:130
  Ephemeral RegistryDisk
  /home/alukiano/go/src/kubevirt.io/kubevirt/tests/registry_disk_test.go:129
    should launch multiple VMs using ephemeral registry disks [It]
    /home/alukiano/go/src/kubevirt.io/kubevirt/tests/registry_disk_test.go:117

    Timed out

    /home/alukiano/go/src/kubevirt.io/kubevirt/tests/registry_disk_test.go:117

I think all replica set failures happen because of bug in GC of earlier versions of k8s

$ oc version
oc v3.7.0+7ed6862
kubernetes v1.7.6+a08f5eeb62
features: Basic-Auth GSSAPI Kerberos SPNEGO

The problem that CentOS OpenShift repository includes only packages for the 3.7 version.
@fabiand Do we want to align our code only with the latest OpenShift build? (In this case, we will need to have some separate repo where I can put latest OpenShift packages or to build this packages on vagrant machine)

@fabiand

This comment has been minimized.

Copy link
Member

fabiand commented Jan 10, 2018

@cynepco3hahue I wouldn't go as far as building the packages ourselfs, but we should see that we can consume more recent builds in order to pick up bugfixes quickly.

Would you build openshift origin from master in order to fix this?

@cynepco3hahue

This comment has been minimized.

Copy link
Member Author

cynepco3hahue commented Jan 11, 2018

@fabiand Currently I trying to integrate build of openshift master packages to vagrant process, but only clone of openshift repository takes a lot of time, so I prefer to use some openshift CI repos if openshift project has them.

@rmohr

This comment has been minimized.

Copy link
Member

rmohr commented Jan 11, 2018

@cynepco3hahue it would also work for me, if you would first fetch the k8s version from the cluster, and skip the two tests if it is a kubernetes 1.7 version. Make sure to add a proper description as the skip reason.

As an example how to do that, we skip two tests on our jenkins instance like this:

It( "test which should be skipped on jenkins", func() {
    if _, exists := os.LookupEnv("JENKINS_HOME"); exists {
        Skip("Skip log query tests for JENKINS ci test environment")
    }
})
@cynepco3hahue

This comment has been minimized.

Copy link
Member Author

cynepco3hahue commented Jan 11, 2018

@rmohr Like the last measures we can do it, but if we want to work aligned to the last version of k8s we will need to use last versions of OpenShift.

@rmohr

This comment has been minimized.

Copy link
Member

rmohr commented Jan 11, 2018

@rmohr Like the last measures we can do it, but if we want to work aligned to the last version of k8s we will need to use last versions of OpenShift.

I would still disable them for now, so that we can proceed with the PR. Will give it an in-depth review tomorrow morning.

@fabiand

This comment has been minimized.

Copy link
Member

fabiand commented Jan 11, 2018

Yes, let's just skip the tests for now.

The openshift problem can be handled separately.

@cynepco3hahue

This comment has been minimized.

Copy link
Member Author

cynepco3hahue commented Jan 14, 2018

@rmohr @fabiand I see yours point will skip tests and continue to work on it under another PR.

@cynepco3hahue cynepco3hahue force-pushed the cynepco3hahue:vagrant_deploy_openshift branch from ff1af46 to 1cb79ca Jan 14, 2018
@cynepco3hahue

This comment has been minimized.

Copy link
Member Author

cynepco3hahue commented Jan 14, 2018

I skipped two replicaset tests, so now all pass.

Ran 35 of 43 Specs in 489.282 seconds
SUCCESS! -- 35 Passed | 0 Failed | 0 Pending | 8 Skipped PASS

the message under the skip

Skip the test because of problem in the earlier version of kubernetes 1.7.6, possible issue https://github.com/kubernetes/kubernetes/issues/50793
@cynepco3hahue

This comment has been minimized.

Copy link
Member Author

cynepco3hahue commented Jan 14, 2018

CI fails on some dependency issue, I think glide cc will solve the problem in the CI environment.

@cynepco3hahue cynepco3hahue changed the title [WIP] Add possibility to deploy OpenShift in vagrant machine Add possibility to deploy OpenShift in vagrant machine Jan 15, 2018
@cynepco3hahue cynepco3hahue force-pushed the cynepco3hahue:vagrant_deploy_openshift branch from 65c20e8 to 82e2a34 Jan 16, 2018
@cynepco3hahue

This comment has been minimized.

Copy link
Member Author

cynepco3hahue commented Jan 16, 2018

@fabiand @rmohr Please review 😄

Copy link
Member

rmohr left a comment

Done with my first review round. Could you also make sure that you have rebased on latest master?


// SkipBecauseOfVersion will skip the test if current k8s version is less than expected one
func SkipBecauseOfVersion(version string, reason string) {
if v, exists := os.LookupEnv("KUBE_VERSION"); exists {

This comment has been minimized.

Copy link
@rmohr

rmohr Jan 16, 2018

Member

Would you mind directly querying the k8s version REST endpoint inside go?

This comment has been minimized.

Copy link
@cynepco3hahue

cynepco3hahue Jan 16, 2018

Author Member

@rmohr Do you have an example, I tried to find how can I do it and without result 😞

This comment has been minimized.

Copy link
@rmohr

rmohr Jan 16, 2018

Member

Sure:

kubecli.CoreV1().RESTClient().Get().AbsPath("/version").Do().Raw()

This comment has been minimized.

Copy link
@rmohr

rmohr Jan 16, 2018

Member

btw running a kubectl command with -v 9 attached is a nice trick to see what it is querying.

[nodes]
master openshift_node_labels="{'region': 'infra','zone': 'default'}" openshift_schedulable=true openshift_ip=$master_ip
$inv_nodes

This comment has been minimized.

Copy link
@rmohr

rmohr Jan 16, 2018

Member

Does this require a specific start order, so that provisioning succeeds on the nodes?

This comment has been minimized.

Copy link
@cynepco3hahue

cynepco3hahue Jan 16, 2018

Author Member

Do you mean under [nodes] section?

@cynepco3hahue cynepco3hahue force-pushed the cynepco3hahue:vagrant_deploy_openshift branch from 2dee1e3 to 873bb56 Jan 17, 2018
@cynepco3hahue

This comment has been minimized.

Copy link
Member Author

cynepco3hahue commented Jan 17, 2018

@rmohr Can you please review again?
By the way, I checked containerized deployment and it worked pretty well for me except some rpm check that failed(openshift/openshift-ansible#6756). So I hope soon we will have a possibility to deploy latest OpenShift on our vagrant machines without additional dancing 😄

@cynepco3hahue

This comment has been minimized.

Copy link
Member Author

cynepco3hahue commented Jan 24, 2018

Good news for containerized deployment, the issue is fixed and deployment passed without any troubles I will update PR to use it 😄

@cynepco3hahue cynepco3hahue force-pushed the cynepco3hahue:vagrant_deploy_openshift branch from 873bb56 to d582f7f Jan 24, 2018
cynepco3hahue added 10 commits Jan 4, 2018
Signed-off-by: Lukianov Artyom <alukiano@redhat.com>
Signed-off-by: Lukianov Artyom <alukiano@redhat.com>
Signed-off-by: Lukianov Artyom <alukiano@redhat.com>
Signed-off-by: Lukianov Artyom <alukiano@redhat.com>
Signed-off-by: Lukianov Artyom <alukiano@redhat.com>
Signed-off-by: Lukianov Artyom <alukiano@redhat.com>
Signed-off-by: Lukianov Artyom <alukiano@redhat.com>
Signed-off-by: Lukianov Artyom <alukiano@redhat.com>
Signed-off-by: Lukianov Artyom <alukiano@redhat.com>
Signed-off-by: Lukianov Artyom <alukiano@redhat.com>
@cynepco3hahue cynepco3hahue force-pushed the cynepco3hahue:vagrant_deploy_openshift branch 2 times, most recently from d7d11d9 to 45bea73 Jan 24, 2018
@cynepco3hahue

This comment has been minimized.

Copy link
Member Author

cynepco3hahue commented Jan 24, 2018

@fabiand @rmohr Now we can use latest OpenShift versions for our testing, I checked replicaset test and they passed without any troubles. Can you review?

Signed-off-by: Lukianov Artyom <alukiano@redhat.com>
@cynepco3hahue cynepco3hahue force-pushed the cynepco3hahue:vagrant_deploy_openshift branch from 45bea73 to 4e9f6a3 Jan 24, 2018
@cynepco3hahue

This comment has been minimized.

Copy link
Member Author

cynepco3hahue commented Jan 24, 2018

Ran 38 of 44 Specs in 592.643 seconds
SUCCESS! -- 38 Passed | 0 Failed | 0 Pending | 6 Skipped PASS
@fabiand

This comment has been minimized.

Copy link
Member

fabiand commented Jan 25, 2018

This is awesome!

@cynepco3hahue

This comment has been minimized.

Copy link
Member Author

cynepco3hahue commented Jan 28, 2018

ci test please

1 similar comment
@cynepco3hahue

This comment has been minimized.

Copy link
Member Author

cynepco3hahue commented Jan 29, 2018

ci test please

@rmohr

This comment has been minimized.

Copy link
Member

rmohr commented Jan 31, 2018

ok to test

@rmohr

This comment has been minimized.

Copy link
Member

rmohr commented Jan 31, 2018

retest this please

@rmohr
rmohr approved these changes Jan 31, 2018
Copy link
Member

rmohr left a comment

Looks good enough to bring it initially in. I will make sure we start testing against it once we have the libvirt and networking changes in.

@rmohr rmohr merged commit 54c952a into kubevirt:master Jan 31, 2018
4 of 6 checks passed
4 of 6 checks passed
check-patch.el7.x86_64 Test script failed
Details
standard-ci Running tests
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
coverage/coveralls Coverage remained the same at 66.588%
Details
kubevirt-functional-tests/jenkins/pr/vagrant-dev All is well still in progress...
Details
kubevirt-functional-tests/jenkins/pr/vagrant-release All is well still in progress...
Details
@cynepco3hahue cynepco3hahue deleted the cynepco3hahue:vagrant_deploy_openshift branch Feb 5, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.