Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace Fabric8 Kubernetes Mock Server with Kindcontainer #9722

Conversation

scholzj
Copy link
Member

@scholzj scholzj commented Feb 21, 2024

Type of change

  • Enhancement / new feature

Description

Currently, we use the Fabric8 Kubernetes Mock Server in Mock tests. It provides a simple CRUD server that replicates some of the behaviors of Kubernetes API. But is is not perfect:

This PR replaces it with the KindContainer project. It is based on test containers and provides several flavors of Kubernetes. The one interesting for us is the API server mode. Instead of using a full Kubernetes cluster, it starts only the API Server and Etcd. You can use the API server as you would use any Kubernetes cluster. But due to the other components missing, nothing is deployed and nobody reacts to the resources you create. Because it is a real API Server, it supports all features such as validation, server-side-apply, In selectors etc.

The use of the API Server test container is wrapped into a MockKube3 that replaces MockKube2 which was based on the Fabric8 Kubernetes Mock Server. Similarly, it has a set of controllers that can be used to emulate the behavior of real Kubernetes such as making the pods or services ready etc. In addition, we also need to help with resource deletion as the garbage collection relies on finalizers that need to be removed.

All of the tests using the MockKube2 and/or the Fabric8 mock server were rewritten to use MockKube3 and the KindContainer API Server. This required significant changes. Starting the test container is slower compared to the CRUD server (which is basically just an HTTP server). To make the tests reasonably fast, the tests had to be rewritten to start the test container only once per test suite and share it between tests. To achieve this, it uses a different namespace for each test based on the test name and uses it during the tests. As we now use a proper API Server, we also have to pass the complete validation done by the API server:

  • Specify all required fields
  • Use appropriate API calls to update the status subresource
  • Use proper fields in OwnerReferences
  • etc.

To compare the times before and after this change, here is the result of the tests using the Fabric8 Mock Server before this PR (it is a passing run, but some tests failed on the first run):

2024-02-23T10:34:39.6617408Z [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.499 s -- in io.strimzi.operator.common.operator.resource.PodOperatorMockTest
2024-02-23T10:58:51.2778961Z [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.490 s -- in io.strimzi.operator.cluster.operator.assembly.ReconnectingWatcherMockTest
2024-02-23T11:01:16.9958696Z [WARNING] Tests run: 30, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 61.49 s -- in io.strimzi.operator.cluster.operator.assembly.ConnectorMockTest
2024-02-23T11:01:30.6856673Z [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.70 s -- in io.strimzi.operator.cluster.operator.assembly.KafkaAssemblyOperatorWithPoolsKRaftMockTest
2024-02-23T11:01:36.1103378Z [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.417 s -- in io.strimzi.operator.cluster.operator.assembly.KafkaConnectAssemblyOperatorMockTest
2024-02-23T11:04:20.3794335Z [ERROR] Tests run: 5, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 105.6 s <<< FAILURE! -- in io.strimzi.operator.cluster.operator.assembly.KafkaUpgradeDowngradeMockTest
2024-02-23T11:04:49.1015844Z [ERROR] Tests run: 9, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 28.30 s <<< FAILURE! -- in io.strimzi.operator.cluster.operator.assembly.StrimziPodSetControllerMockTest
2024-02-23T11:06:07.2170796Z [INFO] Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 77.94 s -- in io.strimzi.operator.cluster.operator.assembly.KafkaAssemblyOperatorMockTest
2024-02-23T11:06:54.0901566Z [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.402 s -- in io.strimzi.operator.cluster.operator.assembly.KafkaMirrorMaker2AssemblyOperatorMockTest
2024-02-23T11:08:08.3849291Z [INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 74.03 s -- in io.strimzi.operator.cluster.operator.assembly.PartialRollingUpdateMockTest
2024-02-23T11:09:02.1042213Z [ERROR] Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 53.22 s <<< FAILURE! -- in io.strimzi.operator.cluster.operator.assembly.KafkaUpgradeDowngradeWithKRaftMockTest
2024-02-23T11:09:08.1789616Z [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.063 s -- in io.strimzi.operator.cluster.operator.assembly.KafkaAssemblyOperatorCustomCertMockTest
2024-02-23T11:10:41.1954950Z [ERROR] Tests run: 11, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 92.53 s <<< FAILURE! -- in io.strimzi.operator.cluster.operator.assembly.KafkaAssemblyOperatorWithPoolsMockTest
2024-02-23T11:11:37.2537652Z [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 42.18 s -- in io.strimzi.operator.cluster.operator.assembly.JbodStorageMockTest
2024-02-23T11:12:48.9518660Z [INFO] Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 55.80 s -- in io.strimzi.operator.cluster.operator.resource.events.KubernetesRestartEventsMockTest
2024-02-23T11:12:50.5876408Z [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.716 s -- in io.strimzi.operator.cluster.leaderelection.LeaderElectionManagerMockTest
2024-02-23T11:14:16.0994193Z [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 44.11 s -- in io.strimzi.operator.cluster.operator.assembly.KafkaUpgradeDowngradeMockTest
2024-02-23T11:14:34.2225914Z [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.13 s -- in io.strimzi.operator.cluster.operator.assembly.KafkaAssemblyOperatorWithPoolsMockTest
2024-02-23T11:14:41.8126352Z [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.268 s -- in io.strimzi.operator.cluster.operator.assembly.KafkaUpgradeDowngradeWithKRaftMockTest
2024-02-23T11:14:53.1172336Z [ERROR] Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 11.58 s <<< FAILURE! -- in io.strimzi.operator.cluster.operator.assembly.StrimziPodSetControllerMockTest
2024-02-23T11:14:54.3147735Z [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.939 s -- in io.strimzi.operator.cluster.operator.assembly.StrimziPodSetControllerMockTest
2024-02-23T11:22:48.5831747Z [INFO] Tests run: 27, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.60 s -- in io.strimzi.operator.user.operator.KafkaUserOperatorMockTest
2024-02-23T11:23:04.9677759Z [INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.12 s -- in io.strimzi.operator.user.UserControllerMockTest

And after:

2024-02-24T22:19:50.6052270Z [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.65 s -- in io.strimzi.test.mockkube3.MockKube3ControllersMockTest
2024-02-24T22:20:58.3603590Z [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.692 s -- in io.strimzi.operator.common.operator.resource.PodOperatorMockTest
2024-02-24T22:42:35.0170871Z [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.27 s -- in io.strimzi.operator.cluster.operator.assembly.ReconnectingWatcherMockTest
2024-02-24T22:44:53.4438064Z [ERROR] Tests run: 29, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 56.85 s <<< FAILURE! -- in io.strimzi.operator.cluster.operator.assembly.ConnectorMockTest
2024-02-24T22:45:05.1981088Z [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.74 s -- in io.strimzi.operator.cluster.operator.assembly.KafkaAssemblyOperatorWithPoolsKRaftMockTest
2024-02-24T22:45:14.9478985Z [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.746 s -- in io.strimzi.operator.cluster.operator.assembly.KafkaConnectAssemblyOperatorMockTest
2024-02-24T22:46:44.7549177Z [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 34.05 s -- in io.strimzi.operator.cluster.operator.assembly.KafkaUpgradeDowngradeMockTest
2024-02-24T22:46:54.7191582Z [INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.778 s -- in io.strimzi.operator.cluster.operator.assembly.StrimziPodSetControllerMockTest
2024-02-24T22:47:22.7065020Z [INFO] Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.92 s -- in io.strimzi.operator.cluster.operator.assembly.KafkaAssemblyOperatorMockTest
2024-02-24T22:47:49.2892659Z [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.598 s -- in io.strimzi.operator.cluster.operator.assembly.KafkaMirrorMaker2AssemblyOperatorMockTest
2024-02-24T22:48:10.8790766Z [INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 21.45 s -- in io.strimzi.operator.cluster.operator.assembly.PartialRollingUpdateMockTest
2024-02-24T22:48:40.2816250Z [INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.92 s -- in io.strimzi.operator.cluster.operator.assembly.KafkaUpgradeDowngradeWithKRaftMockTest
2024-02-24T22:48:48.9927474Z [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.711 s -- in io.strimzi.operator.cluster.operator.assembly.KafkaAssemblyOperatorCustomCertMockTest
2024-02-24T22:49:21.7827525Z [INFO] Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 32.52 s -- in io.strimzi.operator.cluster.operator.assembly.KafkaAssemblyOperatorWithPoolsMockTest
2024-02-24T22:49:46.2912657Z [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.05 s -- in io.strimzi.operator.cluster.operator.assembly.JbodStorageMockTest
2024-02-24T22:50:23.4713853Z [INFO] Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 24.58 s -- in io.strimzi.operator.cluster.operator.resource.events.KubernetesRestartEventsMockTest
2024-02-24T22:50:27.7580725Z [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.499 s -- in io.strimzi.operator.cluster.leaderelection.LeaderElectionManagerMockTest
2024-02-24T22:51:12.5752354Z [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.417 s -- in io.strimzi.operator.cluster.operator.assembly.ConnectorMockTest
2024-02-24T22:57:23.4353379Z [INFO] Tests run: 27, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.80 s -- in io.strimzi.operator.user.operator.KafkaUserOperatorMockTest
2024-02-24T22:57:42.9465754Z [INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.32 s -- in io.strimzi.operator.user.UserControllerMockTest

The speeds of the tests differ in both runs in both directions. But in general, the time spent on these tests is not significantly higher - especially when including the timeouts in the Fabric8 Mock Server tests that seem to be a bit more common.

This PR should resolve #6740.

Checklist

  • Write tests
  • Make sure all tests pass
  • Reference relevant issue(s) and close them after merging

@scholzj scholzj added this to the 0.40.0 milestone Feb 21, 2024
@scholzj
Copy link
Member Author

scholzj commented Feb 21, 2024

@Vaibhav-Nazare I'm trying in this PR an alternative approach to the Mock Kubernetes cluster that is different from the one used in PR #9306. It is using TestContainer that we already use for Kafka for example so I hope it has a better chance to work on various platforms. Do you think you would be able to try this on ppc64le and see if it works? Thanks a lot.

@Vaibhav-Nazare
Copy link
Contributor

Yes @scholzj will try on ppc64le and update the results here.

@Vaibhav-Nazare
Copy link
Contributor

Tests results on ppc64le:
image

@scholzj
Copy link
Member Author

scholzj commented Feb 22, 2024

@Vaibhav-Nazare Great, thanks a lot.

@scholzj scholzj force-pushed the add-mockkube3-based-on-a-real-Kubernetes-API-Server branch from 39236d2 to 0069ddb Compare February 23, 2024 17:45
Signed-off-by: Jakub Scholz <www@scholzj.com>
@scholzj scholzj force-pushed the add-mockkube3-based-on-a-real-Kubernetes-API-Server branch from e2e897c to 88b4b28 Compare February 24, 2024 21:58
@scholzj scholzj changed the title Add MockKube3 based on real Kubernetes API Server Replace Fabric8 Kubernetes Mock Server with Kindcontainer Feb 24, 2024
@scholzj scholzj marked this pull request as ready for review February 25, 2024 00:08
@scholzj
Copy link
Member Author

scholzj commented Feb 25, 2024

@ppatierno This should now be ready for review. Thanks.

Signed-off-by: Jakub Scholz <www@scholzj.com>
@ppatierno ppatierno merged commit 7e25cb5 into strimzi:main Feb 26, 2024
13 checks passed
@scholzj scholzj deleted the add-mockkube3-based-on-a-real-Kubernetes-API-Server branch February 26, 2024 09:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add support for In Label selector to Fabric8 mock Kubernetes server
3 participants