Add Molecule to Docker role

[aharrisson]

What type of PR is this?

Uncomment only one /kind <> line, hit enter to put that in a new line, and remove leading whitespaces from that line:

/kind api-change
/kind bug

/kind cleanup

/kind design
/kind documentation
/kind failing-test
/kind feature
/kind flake

What this PR does / why we need it:
This adds Molecule configuration for one role:

• container-runtime/docker

Adding Molecule would simplify the testability of the roles individually, which could potentially make the verification of changes less time-consuming.
https://molecule.readthedocs.io/en/stable/

The functionality can easily be extended to cover more roles.

The Molecule configuration uses delegated driver configured for kubevirt and could be run both locally using a nodeport or through CI.
The commit is non-intrusive and only adds Molecule files that does not affect the actual role implementation.

Which issue(s) this PR fixes:

Special notes for your reviewer:
The initial PR contained Molecule for multiple roles and multiple scenarios for each role. I have slimmed it down to just one role and added CI support. The two tip commits are just there for test purposes and removes all CI jobs except the molecule test.

Does this PR introduce a user-facing change?:

NONE

[k8s-ci-robot]

Welcome @aharrisson!

It looks like this is your first PR to kubernetes-sigs/kubespray 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/kubespray has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[aharrisson]

/assign @riverzhang

[riverzhang]

@aharrisson Can you add some documentation? It's hard to understand how it works.

[riverzhang]

/assign @mattymo @woopstar

[mattymo]

I don't see much value with this PR as it is. The way it's written, there are 60 new nearly identical molecule.yml files. That's just outrageous and hard to maintain. If they're identical, they should just be located in a central place, such as tests/molecule. What would be even better is to template them out with the role/OS and run them in a systematic fashion. Storing multiple copies of the same file is going to make any future maintainer of this code quite unhappy.

Second, if you want to add such test capability, can you enable this with the docker molecule provider and actually add these tests to a job in our gitlab CI pipeline? We don't have a convenient route to run virtualbox, so the currently provider config is not very useful.

[mattymo]

/hold

[aharrisson]

Thank you for your comments.

The code (or rather configuration) duplication in this PR is intentional, though of course arguable. Molecule is meant to run tests on roles individually, thus storing them centrally would go against that goal.

Regarding the provider configuration, I fully agree that it should conform to the CI setup. I am unfamiliar with your CI environment and also not entirely sure how well Kubernetes would run on Docker.

My point with this PR was to emphasize that treating the roles as individual components rather than parts of a monolith would be beneficial in many aspects. Also, running Molecule on the roles shows that there are some problems that could need some attention.

[MarkusTeufelberger]

I wrote molecule tests for the bootstrap-os role a while ago... the thing I liked was that it makes it relatively easy to actually test all supported distributions, but it took quite some effort and will take even more effort to get onto kubevirt (which the CI system of kubespray is using) instead of virtualbox controlled by vagrant (which molecule is using by default).

There can be a middle ground between having tests for each role and duplicating files, but in general only very few roles in kubespray are standalone (or even designed to be used standalone) to begin with.

[aharrisson]

Thank you for the comment, @MarkusTeufelberger. We discussed your molecule tests for bootstrap-os on slack a while back.
The fact that very few roles are standalone should not be a reason against Molecule IMO. Rather the other way around.

My reasons for submitting this PR is not to necessarily start to use Molecule (though I think the project would benefit from it). It is rather to solve issues that Molecule finds. I started configuring Molecule for my own sake, partly to get my bearings in Kubespray and understand the flow of the play(s), but also to catch issues that I could submit fixes for. But just trying to run Molecule on a few roles made me realize that there are not only minor lint issues, but also more fundamental issues with dependencies and idempotency.

I'd be happy to discuss (on slack?) how we could incorporate Molecule in the project to fit the CI environment and to benefit the development of the project.

[mattymo]

Note to reviewers: this (and following 4 lines) have to be removed before merging

[aharrisson]

Thank you for highlighting that. I edited the PR description and added information that the two tip commits are for testing purposes only. They should not be merged.

[Miouge1]

I love the idea, of using molecule. I'm happy to get help out to get things moving to get that in 2.13.

If you need, reach out on Kubernetes' Slack #kubespray-dev

Btw, you need to rebase on latest master to resolve merge conflicts.

[Miouge1]

I've been doing some tests with Ubuntu 20.04 (#5835) and @champtar on Centos 8 (#5842), it looks like Ansible 2.8 might become a requirement for those newer OSs and that would also open the door to this PR.

[andersharrisson-ess]

@Miouge1 , Thank you for the update.
I am not sure if everyone is in agreement regarding the Molecule driver to use? This PR runs the tests on kubevirt, which in this implementation requires Ansible 2.8. If we prefer Vagrant, that would not be an issue, but then we would have to reconsider the CI setup.

[Miouge1]

@andersharrisson-ess, indeed I would have preferred to go with the Vagrant driver to make it possible to run locally, but I don't have the bandwidth to make that happen, so I'm OK with the kubevirt option.

Moreover some roles might not need a full blown VM, and could do nested with the docker driver (adduser, bootstrap-os, bastion, ...).

I think we need to drop the "only changes" part since that works only with GitLab merge request from what I understand in the doc.

I am happy to get this moving, I think we need to start experimenting with molecule and iterate.

[Miouge1]

To give more info the Vagrant stuff... I did a PoC to run Vagrant in Docker with auto scaling GitLab runners provisioned through docker-machine with the Packet driver.

The thing is that docker-machine is not deprecated but not actively maintained either (AFAIK), therefore I'm not convinced that it's the right way to go at this stage, and back to square one on Vagrant CI.

[MarkusTeufelberger]

For bootstrap-os at least there are lots of distros that don't offer base images which look like a typical installation of that distro, so VMs is nearly the only way to test them.

[Miouge1]

@MarkusTeufelberger yes that is definitely a problem in os-bootstrap. I've created a PR with a PoC to use Vagrant in auto scaler mode (as described above) in #5845. If docker-machine ever gets deprecated we can maybe do a static server instead.

[Miouge1]

Now that we have #5845 is merged, I think we can move forward with this PR with a few changes:

• Switch to vagrant+libvirt driver
• Drop the kubevirt driver for molecule

Any objections?

[aharrisson]

Thank you, @Miouge1
I am a bit preoccupied atm, but will try to fix this PR accordingly next week.
Do we have any plan to fix the issue with only: changes: in the pipeline? i.e. running CI on the same place as the SCM (github vs gitlab)?

[aharrisson]

Refactored to use Vagrant (again). Aligned with the implementation in #5845

Possibly prepare.yml could be placed in a generic location to be reused by some other roles: However, since Kubespray doesn't use role dependencies correctly, we would not be able to reuse it for all roles. I added the preparation in the role, since that would make it more explicit what steps are needed for that individual role.

[Miouge1]

I have tested this locally, it works nicely. Also the pipeline includes the tests, so I'm OK with this.

/approve
/lgtm

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: aharrisson, Miouge1

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [Miouge1]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Miouge1]

@mattymo can you /hold cancel ?

[mattymo]

/hold cancel

[Miouge1]

/ok-to-test