e2e-test: make e2e-test codes can be reused as a lib #1642

liudalibj · 2023-12-27T06:07:59Z

make test/provisioner codes can be reused as a lib
make test/e2e codes can be reused as a lib

liudalibj · 2023-12-27T08:15:08Z

run e2e-test with the changed codes, test passed on providers:

libvirt
ibmcloud

provider aws and azure doesn't be tested as no related env.

stevenhorsman

I left a couple of minor comments and questions, but overall I think this is good and should help us be more flexible with 'third-party' cloud providers in future. I think we need to ensure that it is reviewed/tested on AWS/Azure before merging.

stevenhorsman · 2024-01-02T10:35:14Z

test/provisioner/provision_ibmcloud_self_mgr.go

nit: I think we could move this file instead of deleting and re-adding test/provisioner/ibmcloud/provision_self_mgr.go ?

do this in commit 00b5b34184f57070ec95d9bf0805c26d0680a209
but in the files of this pr still show as deleting and re-adding.
We can check the history with command:

git log --follow test/provisioner/ibmcloud/provision_self_mgr.go commit 4e8623cd2d3fc9110a5955b123f85a4e51f4857f Author: Da Li Liu <liudali@cn.ibm.com> Date: Wed Jan 3 11:13:59 2024 +0800 e2e-test: make test provisioner codes can be reused by other package - make the test provisioner codes be public Fixes: #1641 Signed-off-by: Da Li Liu <liudali@cn.ibm.com> commit 00b5b34184f57070ec95d9bf0805c26d0680a209 Author: Da Li Liu <liudali@cn.ibm.com> Date: Wed Jan 3 11:04:45 2024 +0800 e2e-test: split provisioner source codes to different folder - create provider folders for different provisioner Fixes: #1641 Signed-off-by: Da Li Liu <liudali@cn.ibm.com> commit 6a0e39305b2ea65ba62cb22d65a66cd48d90c2f1 Author: Wainer dos Santos Moschetta <wainersm@redhat.com> Date: Wed Mar 1 17:05:51 2023 -0300 ...

test/e2e/aws_common.go

test/e2e/common_suite.go

test/provisioner/libvirt/provision_common.go

stevenhorsman · 2024-01-02T18:19:45Z

I'm seeing the overlay fail now:

Wait for the cc-operator-controller-manager deployment be available
INFO[0015] Customize the overlay yaml file
INFO[0018] Install the cloud-api-adaptor
Wait for the cc-operator-daemon-install DaemonSet be available
FATA[0320] timed out waiting for the condition

I'll have a play with it manually, but thought it was mentioning as it's an area that has had some refactoring

stevenhorsman · 2024-01-02T19:20:29Z

I'm seeing the overlay fail now:
Wait for the cc-operator-controller-manager deployment be available
INFO[0015] Customize the overlay yaml file
INFO[0018] Install the cloud-api-adaptor
Wait for the cc-operator-daemon-install DaemonSet be available
FATA[0320] timed out waiting for the condition
I'll have a play with it manually, but thought it was mentioning as it's an area that has had some refactoring

Ok, there is something else going on here that I don't fully understand. It looks like we don't get a caa pod as the ds has desired number of 0?:

# kubectl get ds -A
NAMESPACE                        NAME                           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                            AGE
confidential-containers-system   cc-operator-daemon-uninstall   0         0         0       0            0           katacontainers.io/kata-runtime=cleanup   93m
confidential-containers-system   cloud-api-adaptor-daemonset    0         0         0       0            0           node.kubernetes.io/worker=               93m
kube-flannel                     kube-flannel-ds                2         2         2       2            2           <none>                                   7h56m
kube-system                      kube-proxy                     2         2         2       2            2           kubernetes.io/os=linux                   7h56m

Maybe the latest operator isn't compatible with the 0.8 peer pods config?

stevenhorsman · 2024-01-02T19:50:16Z

Ok, there is something else going on here that I don't fully understand. It looks like we don't get a caa pod as the ds has desired number of 0?:

# kubectl get ds -A
NAMESPACE                        NAME                           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                            AGE
confidential-containers-system   cc-operator-daemon-uninstall   0         0         0       0            0           katacontainers.io/kata-runtime=cleanup   93m
confidential-containers-system   cloud-api-adaptor-daemonset    0         0         0       0            0           node.kubernetes.io/worker=               93m
kube-flannel                     kube-flannel-ds                2         2         2       2            2           <none>                                   7h56m
kube-system                      kube-proxy                     2         2         2       2            2           kubernetes.io/os=linux                   7h56m

Maybe the latest operator isn't compatible with the 0.8 peer pods config?

I'm getting the same result on the main branch, so I guess it's either a problem with my environment, or something widerspread. I'll try it on libvirt tomorrow to see if that helps

huoqifeng

The changes looks good to me except the git history mentioned in #1642 (comment), and thanks @stevenhorsman trying this change. I'll tick my approval, please merge it after get these 2 comments addressed, thank you! @liudalibj @stevenhorsman

huoqifeng

LGTM

- create provider folders for different provisioner Fixes: confidential-containers#1641 Signed-off-by: Da Li Liu <liudali@cn.ibm.com>

- make the test provisioner codes be public Fixes: confidential-containers#1641 Signed-off-by: Da Li Liu <liudali@cn.ibm.com>

- use 'git mv' try to keep commit history for refactor test files Fixes: confidential-containers#1641 Signed-off-by: Da Li Liu <liudali@cn.ibm.com>

- make the test codes be public - remove some dup codes fixes confidential-containers#1641 Signed-off-by: Da Li Liu <liudali@cn.ibm.com>

liudalibj · 2024-01-03T09:56:56Z

Ok, there is something else going on here that I don't fully understand. It looks like we don't get a caa pod as the ds has desired number of 0?:
# kubectl get ds -A
NAMESPACE                        NAME                           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                            AGE
confidential-containers-system   cc-operator-daemon-uninstall   0         0         0       0            0           katacontainers.io/kata-runtime=cleanup   93m
confidential-containers-system   cloud-api-adaptor-daemonset    0         0         0       0            0           node.kubernetes.io/worker=               93m
kube-flannel                     kube-flannel-ds                2         2         2       2            2           <none>                                   7h56m
kube-system                      kube-proxy                     2         2         2       2            2           kubernetes.io/os=linux                   7h56m
Maybe the latest operator isn't compatible with the 0.8 peer pods config?
I'm getting the same result on the main branch, so I guess it's either a problem with my environment, or something widerspread. I'll try it on libvirt tomorrow to see if that helps

@stevenhorsman thanks for try this big pr in your local. Here is passed internal job for this PR with ibmcloud provider, https://sys-zaas-k8s-jenkins.swg-devops.com/job/cloud-api-adaptor-e2e-tests-opensource/1121/console you can do a reference.

stevenhorsman · 2024-01-03T10:06:16Z

@stevenhorsman thanks for try this big pr in your local. Here is passed internal job for this PR with ibmcloud provider, https://sys-zaas-k8s-jenkins.swg-devops.com/job/cloud-api-adaptor-e2e-tests-opensource/1121/console you can do a reference.

Thanks DaLi - for those who can't see the test runs behind the IBM firewall I can confirm that I can see that the e2e tests all passed:

09:42:47  PASS
09:42:47  ok  	github.com/confidential-containers/cloud-api-adaptor/test/e2e	1157.975s
09:42:47  e2e test finished.

so I won't worry too much about the failures I was seeing and right it off as a problem with my dev environment. I'll try the libvirt locally whilst waiting for the Ci libvirt tests to finish running, but if they are green then that shouldn't block either.

stevenhorsman

The code looks good to me and I can see that the CI e2e tests are working, so I think once we've had an Azure/AWS reviewer this is good to merge. Thanks DaLi!

mkulke

Checked out the branch and ran the e2e test for Azure successfully 👍

bpradipt · 2024-01-08T08:26:42Z

e2e provisioning for AWS itself doesn't work. In the first scenario when using CIDR "20.0.0.0/16" it failed to create the private network because it uses the default CIDR and not the input CIDR as the basis
Second, when I used the 10.0.0.0/24 as the CIDR it failed with the following error:

time="2024-01-08T13:47:32+05:30" level=info msg="CNI add-on installed. Waiting to be activated (timeout=5m0s)..."
F0108 13:48:33.729214 1350686 env.go:369] Setup failure: envconfig: client failed: Get "https://B6B78244D9897CA8283E46154094DF66.gr7.us-east-2.eks.amazonaws.com/api?timeout=32s": getting credentials: decoding stdout: couldn't get version/kind; json parse error: json: cannot unmarshal string into Go value of type struct { APIVersion string "json:\"apiVersion,omitempty\""; Kind string "json:\"kind,omitempty\"" }

None of these errors are due to this PR though. But unfortunately I can't test without figuring out what's wrong with the AWS e2e provisioning. This is going to take time. So considering this situation, please don't hold the PR.

cc @wainersm for awareness

bpradipt

/lgtm

liudalibj · 2024-01-08T08:40:07Z

e2e provisioning for AWS itself doesn't work. In the first scenario when using CIDR "20.0.0.0/16" it failed to create the private network because it uses the default CIDR and not the input CIDR as the basis Second, when I used the 10.0.0.0/24 as the CIDR it failed with the following error:
time="2024-01-08T13:47:32+05:30" level=info msg="CNI add-on installed. Waiting to be activated (timeout=5m0s)..."
F0108 13:48:33.729214 1350686 env.go:369] Setup failure: envconfig: client failed: Get "https://B6B78244D9897CA8283E46154094DF66.gr7.us-east-2.eks.amazonaws.com/api?timeout=32s": getting credentials: decoding stdout: couldn't get version/kind; json parse error: json: cannot unmarshal string into Go value of type struct { APIVersion string "json:\"apiVersion,omitempty\""; Kind string "json:\"kind,omitempty\"" }
None of these errors are due to this PR though. But unfortunately I can't test without figuring out what's wrong with the AWS e2e provisioning. This is going to take time. So considering this situation, please don't hold the PR.

cc @wainersm for awareness

Thanks for the update @bpradipt so that I will merge this pr now.
Please feel free leave comments when you try the new e2e test cases after the AWS e2e provisioning issue is fixed.

liudalibj requested review from wainersm, stevenhorsman, mkulke and genjuro214 December 27, 2023 06:08

liudalibj added the test_e2e_libvirt Run Libvirt e2e tests label Dec 27, 2023

liudalibj requested review from bpradipt and huoqifeng December 27, 2023 06:36

liudalibj force-pushed the ext-tests branch from fa31837 to 1fb1aa9 Compare December 28, 2023 06:30

liudalibj marked this pull request as draft December 28, 2023 14:57

liudalibj force-pushed the ext-tests branch 5 times, most recently from 9221487 to cf2e237 Compare January 1, 2024 16:09

liudalibj marked this pull request as ready for review January 1, 2024 16:10

liudalibj added the e2e-test label Jan 1, 2024

liudalibj changed the title ~~e2e-test: make e2e-test codes can be reused by other package~~ e2e-test: make e2e-test codes can be reused as a lib Jan 1, 2024

liudalibj force-pushed the ext-tests branch 2 times, most recently from 8fffc6a to d4a45e0 Compare January 1, 2024 17:04

stevenhorsman approved these changes Jan 2, 2024

View reviewed changes

stevenhorsman requested changes Jan 2, 2024

View reviewed changes

test/provisioner/libvirt/provision_common.go Show resolved Hide resolved

liudalibj force-pushed the ext-tests branch 5 times, most recently from fd27771 to 7ba872f Compare January 2, 2024 16:36

huoqifeng reviewed Jan 3, 2024

View reviewed changes

huoqifeng approved these changes Jan 3, 2024

View reviewed changes

liudalibj added 3 commits January 3, 2024 11:04

e2e-test: split provisioner source codes to different folder

00b5b34

- create provider folders for different provisioner Fixes: confidential-containers#1641 Signed-off-by: Da Li Liu <liudali@cn.ibm.com>

e2e-test: make test provisioner codes can be reused by other package

4e8623c

- make the test provisioner codes be public Fixes: confidential-containers#1641 Signed-off-by: Da Li Liu <liudali@cn.ibm.com>

e2e-test: rename test files to common ones

8ee261d

- use 'git mv' try to keep commit history for refactor test files Fixes: confidential-containers#1641 Signed-off-by: Da Li Liu <liudali@cn.ibm.com>

liudalibj force-pushed the ext-tests branch 3 times, most recently from d71cc85 to 724ecb8 Compare January 3, 2024 08:25

e2e-test: make e2e-test codes can be reused by other package

0a81ff8

- make the test codes be public - remove some dup codes fixes confidential-containers#1641 Signed-off-by: Da Li Liu <liudali@cn.ibm.com>

liudalibj force-pushed the ext-tests branch from 724ecb8 to 0a81ff8 Compare January 3, 2024 09:11

stevenhorsman approved these changes Jan 3, 2024

View reviewed changes

mkulke approved these changes Jan 4, 2024

View reviewed changes

bpradipt approved these changes Jan 8, 2024

View reviewed changes

liudalibj merged commit aea3761 into confidential-containers:main Jan 8, 2024
25 checks passed

liudalibj mentioned this pull request Feb 6, 2024

Make cloud providers be loaded as plugins for CAA #1692

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

e2e-test: make e2e-test codes can be reused as a lib #1642

e2e-test: make e2e-test codes can be reused as a lib #1642

liudalibj commented Dec 27, 2023 •

edited

liudalibj commented Dec 27, 2023

stevenhorsman left a comment

stevenhorsman Jan 2, 2024

liudalibj Jan 3, 2024

stevenhorsman commented Jan 2, 2024

stevenhorsman commented Jan 2, 2024 •

edited

stevenhorsman commented Jan 2, 2024

huoqifeng left a comment

huoqifeng left a comment

liudalibj commented Jan 3, 2024

stevenhorsman commented Jan 3, 2024

stevenhorsman left a comment

mkulke left a comment

bpradipt commented Jan 8, 2024

bpradipt left a comment

liudalibj commented Jan 8, 2024

e2e-test: make e2e-test codes can be reused as a lib #1642

e2e-test: make e2e-test codes can be reused as a lib #1642

Conversation

liudalibj commented Dec 27, 2023 • edited

liudalibj commented Dec 27, 2023

stevenhorsman left a comment

Choose a reason for hiding this comment

stevenhorsman Jan 2, 2024

Choose a reason for hiding this comment

liudalibj Jan 3, 2024

Choose a reason for hiding this comment

stevenhorsman commented Jan 2, 2024

stevenhorsman commented Jan 2, 2024 • edited

stevenhorsman commented Jan 2, 2024

huoqifeng left a comment

Choose a reason for hiding this comment

huoqifeng left a comment

Choose a reason for hiding this comment

liudalibj commented Jan 3, 2024

stevenhorsman commented Jan 3, 2024

stevenhorsman left a comment

Choose a reason for hiding this comment

mkulke left a comment

Choose a reason for hiding this comment

bpradipt commented Jan 8, 2024

bpradipt left a comment

Choose a reason for hiding this comment

liudalibj commented Jan 8, 2024

liudalibj commented Dec 27, 2023 •

edited

stevenhorsman commented Jan 2, 2024 •

edited