Implement integration testing across Crossplane ecosystem

[hasheddan]

What problem are you facing?

Currently, all integration testing across the Crossplane ecosystem is performed in a manual ad-hoc manner that results in a significant amount of inefficient effort and uncaught errors leading up to new releases. It is desirable to run automated integration tests on a more frequent basis such that errors are caught faster and releases are delivered more smoothly.

How could Crossplane help solve your problem?

Developing a framework that can be used across Crossplane projects would standardize the implementation and execution of integration tests. The initial implementation should be broad and flexible, deferring much of the logic to the written tests. Over time, commonly used tests and functionality can be incorporated into the framework itself.

Steps

• Create design doc for integration testing framework Integration testing framework design doc #1032
• Create integration testing framework (work can be tracked here)
• Implement initial integration tests in stack-aws Implement integration tests crossplane-contrib/provider-aws#73
• Implement initial integration tests in stack-azure Implement integration tests crossplane-contrib/provider-azure#61
• Implement initial integration tests in stack-gcp Implement integration tests crossplane-contrib/provider-gcp#87
• Implement initial integration tests in stack-rook

[suskin]

Summarizing some discussion on this from Slack with @hasheddan:

As a developer, I'd like to run fast, narrowly scoped "integration" tests which test my controller logic (for example). I'd like to be able to run them as part of my PR test suite, for ones which are fast enough and which don't use external resources or infrastructure.

This would likely be accomplished by allowing us to easily use the local mode of the envtest library, which depends on local apiserver binaries being available. Solving it in this way would mean updating the build images to include those binaries.

This came up in the context of a test in #1214

[negz]

@hasheddan, @rahulgrover99 do you know if this issue should still be "In Progress" in the v1.4 timeframe? Is it being actively worked on?

[hasheddan]

@negz is it not to my knowledge. I feel like it would it make sense to close this one out as much of the context is stale, and more specific issues related to this effort would serve as better items for folks to work on.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.