Improving Antrea Matrix Compatibility CI

[XinShuYang]

Describe what you are trying to solve
We are aiming to simplify the Antrea matrix compatibility CI jobs by leveraging Kind and CAPA for testing instead of the existing CAPV setup. This initiative involves recovering outdated Jenkins CI jobs and introducing kind and CAPA to Antrea matrix compatibility tests.

Describe the solution you have in mind

• Recovering centOS and Photon test cases for both antrea-upgrade-matrix-compatibility-test and antrea-matrix-compatibility-test jobs based on CAPV before CAPA PR Add CI for testing antrea compatibility with 4 K8s versions on CAPA #5476 is merged.
• Removing the Ubuntu test case for both antrea-upgrade-matrix-compatibility-test and antrea-matrix-compatibility-test jobs from the Jenkins CI environment. These jobs have not been executed for a long time and can already be replaced by corresponding Kind jobs.
• Introducing Kind to Ubuntu test cases in Antrea matrix compatibility jobs #6179 This job will automatically trigger periodic runs to ensure the ongoing compatibility of Antrea across different Kubernetes versions and configurations.
• Enabling CAPA to replace CAPV for different OS such as CentOS and Photon. #6180

Describe how your solution impacts user flows

1. By transitioning to Kind-based testing and removing outdated Jenkins jobs, we aim to simplify CI maintenance efforts for the maintainers. This will lead to a more efficient and streamlined CI process.
2. New Kind-based periodic job will automate the testing process, eliminating the need for users to manually trigger corresponding Kind jobs.

Additional context
Any feedback or recommendations are appreciated, as these jobs were created a long time ago, and I may have missed some details. cc @tnqn @antoninbas @luolanzone

[rajnkamr]

Currently Antrea matrix test wrt k8s versions with CAPA is still not merged #5476 , in case we want to move to kind based matrix testing for k8 versions, we need to streamline current CPAV matrix jobs as well !

[XinShuYang]

Currently Antrea matrix test wrt k8s versions with CAPA is still not merged #5476 , in case we want to move to kind based matrix testing for k8 versions, we need to streamline current CPAV matrix jobs as well !

The current CAPV matrix job tests are based on Kubernetes v1.17.5 and v1.18.2, so I believe they have not been used for a long time. Before merging the CAPA job PR, I will first attempt to restore the necessary CAPV jobs for matrix tasks. Given that some tests can be covered by Kind testing, we can consider removing them and replacing others with CAPA afterward.

[antoninbas]

Overall sounds good to me.

New Kind-based periodic job will automate the testing process, eliminating the need for users to manually trigger corresponding Kind jobs during release phases.

I am not sure I agree with this statement, or maybe I am misunderstanding. We should still validate releases by running a "full" test suite, and periodic CI jobs will not change that. However, I imagine that well-monitored periodic CI jobs will reduce the amount of test failures that have to be investigated and resolved "at the last minute" during the release phase. However, historically, we haven't been good at keeping track of failures in periodic CI jobs and addressing the issues as they arise, as far as I know. Some mechanism that automatically opens a Github issue when a periodic CI job starts failing (e.g., with 2 consecutive failures) could help, but I am not sure how easy it is to set up.

[antoninbas]

I will first attempt to restore the necessary CAPV jobs for matrix tasks

What's the value in trying to restore / recover these old jobs before introducing the new (Kind / CAPA based) ones?

[XinShuYang]

I will first attempt to restore the necessary CAPV jobs for matrix tasks

What's the value in trying to restore / recover these old jobs before introducing the new (Kind / CAPA based) ones?

There are matrix jobs validating Antrea based on CentOS and Photon OS. Do we have other running jobs that can replace them? If so, we indeed don't need to restore the CAPV matrix jobs before the CAPA upgrade.

[XinShuYang]

Overall sounds good to me.

New Kind-based periodic job will automate the testing process, eliminating the need for users to manually trigger corresponding Kind jobs during release phases.

I am not sure I agree with this statement, or maybe I am misunderstanding. We should still validate releases by running a "full" test suite, and periodic CI jobs will not change that. However, I imagine that well-monitored periodic CI jobs will reduce the amount of test failures that have to be investigated and resolved "at the last minute" during the release phase. However, historically, we haven't been good at keeping track of failures in periodic CI jobs and addressing the issues as they arise, as far as I know. Some mechanism that automatically opens a Github issue when a periodic CI job starts failing (e.g., with 2 consecutive failures) could help, but I am not sure how easy it is to set up.

Yes, what I mean is periodic jobs can report failures in time. Regarding job failure notifications, the current Jenkins email notification is disabled due to the migration. I will attempt to enable it first.

[rajnkamr]

I will first attempt to restore the necessary CAPV jobs for matrix tasks

What's the value in trying to restore / recover these old jobs before introducing the new (Kind / CAPA based) ones?

There are matrix jobs validating Antrea based on CentOS and Photon OS. Do we have other running jobs that can replace them? If so, we indeed don't need to restore the CAPV matrix jobs before the CAPA upgrade.

Compatibility Matrix test of antrea with k8 versions for ubuntu is taken care ( CAPA PR #5476 - non kind) , For centos and photon Antrea matrix compatibility tests, kind can be leveraged. Periodic job approach seems to save on resources which should be usually available for regular jobs.
Email notification for period job failures must be enabled for tracking of periodic jobs. May be we can add checking period job result in release checklist.

[antoninbas]

@XinShuYang you have these 2 list items:

• Recovering centOS and Photon test cases for both antrea-upgrade-matrix-compatibility-test and antrea-matrix-compatibility-test jobs based on CAPV.
• Enabling CAPA jobs for different OS such as CentOS or Photon.

I guess I do not understand how they relate to each other. If you want to enable CAPA jobs for these OS's, why do you need the first list item?

[rajnkamr]

@XinShuYang
I have same question as @antoninbas , do we still want to use CAPV, we wanted to run most of the capv matrix jobs on CAPA.
It might be better to split 3rd and 4th items to different issues

"3.Introducing Kind to Ubuntu test cases in Antrea matrix compatibility jobs. This job will automatically trigger periodic runs to ensure the ongoing compatibility of Antrea across different Kubernetes versions and configurations.
4. Enabling CAPA jobs for different OS such as CentOS or Photon."

[XinShuYang]

@XinShuYang you have these 2 list items:

* Recovering centOS and Photon test cases for both antrea-upgrade-matrix-compatibility-test and antrea-matrix-compatibility-test jobs based on CAPV.

* Enabling CAPA jobs for different OS such as CentOS or Photon.

I guess I do not understand how they relate to each other. If you want to enable CAPA jobs for these OS's, why do you need the first list item?

@antoninbas The final goal is to use CAPA instead of CAPV in the matrix test. However, since the PR for CAPA #5476 has not been merged yet (and I think it may take longer for review and merge), I prefer to first restore the existing CAPV jobs.

[XinShuYang]

@XinShuYang I have same question as @antoninbas , do we still want to use CAPV, we wanted to run most of the capv matrix jobs on CAPA. It might be better to split 3rd and 4th items to different issues

"3.Introducing Kind to Ubuntu test cases in Antrea matrix compatibility jobs. This job will automatically trigger periodic runs to ensure the ongoing compatibility of Antrea across different Kubernetes versions and configurations. 4. Enabling CAPA jobs for different OS such as CentOS or Photon."

@rajnkamr Yes, I created two issues to track the progress, thanks for the suggestion.