Optimze detecting if calico exists

[hangscer8]

What type of PR is this?

Uncomment only one /kind <> line, hit enter to put that in a new line, and remove leading whitespaces from that line:

/kind api-change
/kind bug
/kind cleanup
/kind design
/kind documentation
/kind failing-test
/kind feature
/kind flake

/kind bug

What this PR does / why we need it:

• optimize detect the existence of calico, because sometimes the file calicoctl.sh exists ,but not the related crds, such as clusterinformations.
• Add check for ClusterInformation of calico.

TASK [network_plugin/calico : Get current calico version] **********************
fatal: [node1]: FAILED! => {"changed": false,"stdout": "v3.24.5", "stdout_lines": ["v3.24.5"] , "cmd": "set -o pipefail && /usr/local/bin/calicoctl.sh version | grep 'Client Version:' | awk '{ print $3}'", "delta": "0:00:02.873448", "end": "2023-03-09 17:19:44.681338", "msg": "non-zero return code", "rc": 1, "start": "2023-03-09 17:19:41.807890", "stderr": "Unable to retrieve Cluster Version or Type: resource does not exist: ClusterInformation(default) with error: clusterinformations.crd.projectcalico.org \"default\" not found", "stderr_lines": ["Unable to retrieve Cluster Version or Type: resource does not exist: ClusterInformation(default) with error: clusterinformations.crd.projectcalico.org \"default\" not found"]}

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

[Calico] Optimize the detection of calico existence 

[k8s-ci-robot]

Hi @hangscer8. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[floryut]

@hangscer8 Please check #9861

This part has changed a lot recently ;)

[cyclinder]

check version fails because calico's crd(clusterinformations.crd.projectcalico.org) is not yet installed, which causes calicoctl version to fail, in this case, the exit code should not be 1, we should ignore this error indeed.

But is there a better way to resolve it? There are some thoughts:

• ignore this error
• add a new task to check if stdout matches the version of the regular expression:

grep  -Eo 'v[0-9]+.[0-9]+.[0-9]' <<< stdout

/cc @zhan9san Any thoughts?

[cyclinder]

/ok-to-test

[zhan9san]

@cyclinder

Thanks for making me involved.

I don't think ignoring error is a good option. Instead I prefer to catch the expected error.

[zhan9san]

IMO, first time we introduce the calico check.yml is to work around some compatibility issues when we perform the calico upgrade.

Can we check whether calico's crd(clusterinformations.crd.projectcalico.org) is installed like below?

kubespray/roles/network_plugin/calico/tasks/check.yml

Lines 46 to 51 in fff4005

	- name: Check if calico exists
	stat:
	path: "{{ bin_dir }}/calicoctl.sh"
	register: calico_exists
	run_once: True
	delegate_to: "{{ groups['kube_control_plane'][0] }}"

[cyclinder]

I'm curious why the crds doesn't exist when calico is upgraded. check_version is only happen in calico upgrade?

[zhan9san]

I'm curious why the crds doesn't exist when calico is upgraded.

Yeah, it's weird that crds doesn't exist when calico is upgraded.

check_version is only happen in calico upgrade?

It always run in cluster.yml -> role: kubernetes/preinstall -> Run calico checks

Except the fresh install, as if it is a fresh install, calicoctl.sh would not exist, and Check that current calico version is enough for upgrade block would not run as its condition(when: calico_exists.stat.exists) is false.

kubespray/cluster.yml

Line 34 in fff4005

- { role: kubernetes/preinstall, tags: preinstall }

kubespray/roles/kubernetes/preinstall/tasks/main.yml

Lines 126 to 132 in fff4005

	- name: Run calico checks
	include_role:
	name: network_plugin/calico
	tasks_from: check
	when:
	- kube_network_plugin == 'calico'
	- not ignore_assert_errors

I'll try to reproduce this issue.

[hangscer8]

I'm curious why the crds doesn't exist when calico is upgraded.

Yeah, it's weird that crds doesn't exist when calico is upgraded.

check_version is only happen in calico upgrade?

It always run in cluster.yml -> role: kubernetes/preinstall -> Run calico checks

Except the fresh install, as if it is a fresh install, calicoctl.sh would not exist, and Check that current calico version is enough for upgrade block would not run as its condition(when: calico_exists.stat.exists) is false.

kubespray/cluster.yml

Line 34 in fff4005

- { role: kubernetes/preinstall, tags: preinstall }

kubespray/roles/kubernetes/preinstall/tasks/main.yml

Lines 126 to 132 in fff4005

	- name: Run calico checks
	include_role:
	name: network_plugin/calico
	tasks_from: check
	when:
	- kube_network_plugin == 'calico'
	- not ignore_assert_errors

I'll try to reproduce this issue.

It is possible that i sometimes kill the e2e ci job which runs cluster.yml playbook on the target ci node to install k8s cluster when the job is running, it causes the half installation of calico.

[zhan9san]

It is possible that i sometimes kill the e2e ci job which runs cluster.yml playbook on the target ci node to install k8s cluster when the job is running, it causes the half installation of calico.

So the issue becomes how to detect the calico is installed correctly.

How about checking both calicoctl.sh and crd?

[hangscer8]

It is possible that i sometimes kill the e2e ci job which runs cluster.yml playbook on the target ci node to install k8s cluster when the job is running, it causes the half installation of calico.

So the issue becomes how to detect the calico is installed correctly.

How about checking both calicoctl.sh and crd?

I think it is a good idea. I will fix that.

[cyclinder]

Seems lgtm

/cc @floryut @zhan9san, Can you take a final look?

[zhan9san]

Thanks @hangscer8

LGTM

[cyclinder]

Thanks!

/lgtm

[floryut]

@hangscer8 Thank you
And thanks guys for the discussion and review
/kind Network

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: floryut, hangscer8

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [floryut]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment