Lots of ci/prow lanes are failing due to gcloud problem

[jlojosnegros]

Hi all,

I am working on a PR and have seen some ci/lanes failing due to a problem with gcloud.
I have been looking around and I have notice is also failing on many other PRs.

I would like to know if this is an already known issue and if so, if there is a known solution I can apply or if, on the other hand some investigation is needed.

Thanks in advance

Activated service account credentials for: [do-not-delete-ci-provisioner@XXXXXXXXXXXXXXXXXXXXXX.iam.gserviceaccount.com]
Updated property [core/project].
You do not currently have this command group installed.  Using it 
requires the installation of components: [alpha]
ERROR: (gcloud) You cannot perform this action because you do not have permission to modify the Google Cloud SDK installation directory [/usr/local/gcloud/google-cloud-sdk].
Re-run the command with sudo: sudo /usr/local/gcloud/google-cloud-sdk/bin/gcloud ...

[jlojosnegros]

I have also notice that kata-containers ci lane is failing due to an error in the golang version.

Repo /tmp/jenkins/workspace/kata-containers-2-crio-PR/go/src/github.com/kata-containers/kata-containers already in git's safe.directory
/tmp/jenkins/workspace/kata-containers-2-crio-PR/go/src/github.com/kata-containers/kata-containers/src/runtime /tmp/jenkins/workspace/kata-containers-2-crio-PR/go/src/github.com/kata-containers/tests
golang.mk:60: *** "ERROR: golang minor version too old: got 1.18.1, need atleast 1.19".  Stop.

[littlejawa]

Yes. We are experiencing issues on the kata CI side, and this is a side effect. We are working on it, but other issues need to be fixed before we can make this pass.

[littlejawa]

I have made a quick fix to unblock PRs, it should be testing correctly now.
A cleaner patch is on its way on the kata side, but that should be transparent for the cri-o devs.
Sorry for the delay.

[haircommander]

interestingly, that's actually not the failure. for some reason, prow finds that abnormal and posts it, but the failure is farther down:

Container test exited with code 1, reason Error
---
ble.builtin.seboolean to ansible.posix.seboolean
Using module file /home/deadbeef/.local/lib/python3.9/site-packages/ansible_collections/ansible/posix/plugins/modules/seboolean.py
<localhost> PUT /home/deadbeef/.ansible/tmp/ansible-local-57899fa_6bdea/tmp6v3hsvwu TO /home/deadbeef/.ansible/tmp/ansible-tmp-1669733325.0824344-65783-279178949064296/AnsiballZ_seboolean.py
<localhost> EXEC /bin/sh -c 'chmod u+x /home/deadbeef/.ansible/tmp/ansible-tmp-1669733325.0824344-65783-279178949064296/ /home/deadbeef/.ansible/tmp/ansible-tmp-1669733325.0824344-65783-279178949064296/AnsiballZ_seboolean.py && sleep 0'
<localhost> EXEC /bin/sh -c 'sudo -H -S -n  -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-kmhhazrdresplefzcydljawiqjlkcwza ; GOPATH=/usr/go /usr/bin/python3 /home/deadbeef/.ansible/tmp/ansible-tmp-1669733325.0824344-65783-279178949064296/AnsiballZ_seboolean.py'"'"' && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /home/deadbeef/.ansible/tmp/ansible-tmp-1669733325.0824344-65783-279178949064296/ > /dev/null 2>&1 && sleep 0'

TASK [Containers using cgroups and/or systemd after 7.5 are permitted] *********
task path: /home/deadbeef/cri-o/contrib/test/ci/system.yml:65
fatal: [localhost]: FAILED! => {
    "changed": false,
    "invocation": {
        "module_args": {
            "ignore_selinux_state": false,
            "name": "container_manage_cgroup",
            "persistent": true,
            "state": true
        }
    }
}

MSG:

SELinux boolean container_manage_cgroup does not exist.

something is funky in the container-selinux package on rhel 9. @sohankunkerkar is looking into it

[haircommander]

I am pretty sure this particular error is fixed now, where are you seeing it?

[jlojosnegros]

Yeah sorry. It fails but with different errors, my bad.

Still here #6388

• kata-containers
• ci-fedora-integration
• ci-rhel-integration

are failing recurrently

kata-containers seems to be failing due to a timeout in here...

Signature ok
subject=CN = Kata Containers Webhook
Getting CA Private Key
secret/pod-annotate-webhook-certs created
mutatingwebhookconfiguration.admissionregistration.k8s.io/pod-annotate-webhook created
deployment.apps/pod-annotate-webhook created
service/pod-annotate-webhook created
[webhook-check.sh:70] INFO: Going to check the kata-webhook installation
ERROR: The pod-annotate-webhook deployment is unavailable after 60s waiting

ci-fedora-integration and ci-rhel-integration are failing while trying to run localintegration

TASK [run integration tests] ***************************************************
task path: /home/deadbeef/cri-o/contrib/test/ci/test.yml:61
fatal: [localhost]: FAILED! => {
    "ansible_job_id": "764668902847.128257",
    "changed": true,
    "cmd": "make localintegration 2>&1 > /tmp/artifacts/testout.txt",
    "delta": "0:07:18.327718",
    "end": "2023-01-25 09:24:58.315529",
    "finished": 1,
    "invocation": {
        "module_args": {
            "_raw_params": "make localintegration 2>&1 > /tmp/artifacts/testout.txt",
            "_uses_shell": true,
            "argv": null,
            "chdir": "/usr/go/src/github.com/cri-o/cri-o",
            "creates": null,
            "executable": "/bin/bash",
            "removes": null,
            "stdin": null,
            "stdin_add_newline": true,
            "strip_empty_ends": true
        }
    },
    "rc": 2,
    "results_file": "/tmp/artifacts/async/764668902847.128257",
    "start": "2023-01-25 09:17:39.987811",
    "started": 1
}

but I am not sure what the problem is but it seems unlikely to me that it can be related with the change in the PR.

I've been looking at the latests PR merged and it seems this ci-lanes have been overridden but I do not know if it was due to the same failure or not.

P.S.:
here are the latest run of each of the lanes

• http://jenkins.katacontainers.io/job/kata-containers-2-crio-PR/4704/
• https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/cri-o_cri-o/6388/pull-ci-cri-o-cri-o-main-ci-fedora-integration/1618169037454315520
• https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/cri-o_cri-o/6388/pull-ci-cri-o-cri-o-main-ci-rhel-integration/1618169052998406144

[haircommander]

thanks for mentioning! we have issue open for them: #6523 #6522 #6481
All we're aware of and in various stages of fixing

[jlojosnegros]

Do you think #6388 could be merged overriding the ci-lanes or should we wait?

[haircommander]

yeah definitely :)