e2e-ansible-molecule test improvements

[asmacdo]

The ansible-molecule GH action takes on average 35 minutes to run on CI, which is more than double the test time for the other e2e tests and is a bottleneck for the whole project. I recently went through these tests and collected a handful of improvements to make. The goal would be to bring the ansible-molecule test runtime down to about 15 minutes to be on-par with go and helm e2e.

[ ] Asynchronous setup
We create each object and wait for it to finish before we create the next. This change brings the runtime down from 35 to 27 minutes
operator-framework/operator-sdk#5679

[ ] Separate memcached ansible molecule test into a separate action
We run 2 sets of tests (in serial), advanced-molecule-operator and memcached-ansible-molecule. The tests in advanced-molecule-operator are our primary ansible tests. memcached-ansible-molecule also runs some tests that could be consolidated into advanced-molecule-operator with the exception of the cluster setup. memcached-ansible-molecule is designed to be similar to how operator authors might test their operators, and so uses the kind scenario within molecule. This scenario creates a kind cluster. ansible molecule e2e tests therefore create a cluster, deploy operators and test them, destroys the cluster, creates a second cluster, deploys operators and tests them and destroys the cluster. There is no shared state, and separating this component could probably cut the runtime down by half.

[ ] Refactor the test re-generation
The main difficulty is that the test operators are scaffolded from scratch each time in order to keep up with operator-sdk updates, and then all test code is inserted. This would make sense for files that the operator-sdk actively changes, but the majority of the changes are to playbooks, molecule verify tasks, and the watches file. So rather than searching for strings and inserting code, I propose that we simply checkin the playbooks, molecule verify tasks, and the watches file for each of the test operators, and then clobber them. For playbooks and verify tasks, we will be replacing nothing. The watches file can be generated by api creation, but does not include extras like finalizers. This change will bring the testing requirements closer to the requirements of operator-authors, who also have to frequently upgrade operator-sdk version.

[camilamacedo86]

Could we begin to have the molecule tests scaffolded by default under the testdata?
WDYT?

[asmacdo]

@camilamacedo86, do you mean committing them to the repo? They are currently scaffolded there at test runtime, but not committed to the repo. If so, big +1

If I misunderstood could you elaborate?

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[camilamacedo86]

/remove-lifecycle stale
/lifecycle frozen