Drop or replace drivers with ansible content

[ssbarnea]

In order to make it easier to write tests, we will investigate dropping current molecule drivers and replacing them with pure ansible collections.

• Add utility Ansible module/filter to collections like docker, podman,... that could ease provisioning of these resources
• Find a way to expose in collection metadata the molecule support, so people could search for collections that can easily be used with molecule.

Please note that even now, anyone using delegated driver can already do this. The difference is that we will try to a way to facility the migration from someone using a particular driver with minimal effort, so they would not have to rewrite their entire test.

[apatard]

I'm not convinced it'll be a "win" in the end. The driver glue/code is small and usually the issues are the playbooks (the notable exception being vagrant and it's vagrant plugin). Converting them to module/filter is going to add extra failure paths and maintainance imho.

Moreover, when I think how handling collection search paths and things like that seems to lead to troubles in -lint, I don't see how moving everything to a collection is an improvement.

Side-question: I'm not sure to understand how it'll work in details. For instance, how 'molecule login' will work after this change ? Do you have more details ?

[tima]

@apatard The problem is that the driver glue/code is code. However small and trivial it may be to a python programmer, it is constraining to those who are not. Most Ansible content creators are not programmers so this is a "win" in that it will let all content creators more effectively test their content.

I will have to leave your side question to @ssbarnea.

[kksat]

I might not have complete understanding of your suggestion.

In order to make it easier to write tests, we will investigate dropping current molecule drivers and replacing them with pure ansible collections.

You can do that already. Just create another collection and move content from playbook https://github.com/ansible-community/molecule-plugins/blob/main/src/molecule_plugins/docker/playbooks/create.yml to playbook in that collection.

Let's call new collection molecule.docker, playbook will be called create (for create, destroy for destroy, etc).

Remove contents of file https://github.com/ansible-community/molecule-plugins/blob/main/src/molecule_plugins/docker/playbooks/create.yml and add following

- name: Create
  ansible.builtin.import_playbook: molecule.docker.create

This way you decoupled driver and playbooks to manage docker infra. You can test driver separately, and molecule.docker collection separately. Other content that is required to provision docker infra also has to be moved to collection. For instance https://github.com/ansible-community/molecule-plugins/tree/main/src/molecule_plugins/docker/playbooks/filter_plugins.

All what should be left in the driver is - driver name and cookiecutter templates to call for external collection content (playbook, playbooks will use other content). This way it is much more simple to create driver and test it. Same with molecule.docker (in this example) collection - this is collection you can test it more easily compared when it was in cookiecutter template.

Important note - I am not saying that molecule.docker collection should be the same as Community.Docker collection. It should depend on it, but the interface is completely different. Usually in collections roles are the interface, for molecule collections playbooks are the interface. Playbooks use molecule_yml dictionary to create all the infrastructure.

Find a way to expose in collection metadata the molecule support, so people could search for collections that can easily be used with molecule.

This can be as easy as agreeing that all playbooks that start with molecule_* are molecule playbooks and can be used in molecule. For instance molecule.docker.molecule_create. Also these playbooks can be added to community.docker collection (in this example). So no need to create another collection for molecule. But this might force to add additional dependencies to community.docker collection - investigation is required.

[tima]

In order to make it easier to write tests, we will investigate dropping current molecule drivers and replacing them with pure ansible collections.

You can do that already.

No doubt @kksat. One can use the delegated (not the best name admittedly) driver to do that. The point is to make using Ansible itself the undisputed preferred means of provisioning a test environment in a way that is accessible and as straightforward and easy for any content developer (especially new ones) to pick up and use. My hope is that there are a plethora of reusable options made available, but given this new found freedom of not having to write python code I have a feeling that a lot of content creators will opt to create their own tailored to their specific needs, preferences and standards.

So no need to create another collection for molecule.

But that assumes that a driver (provisioner) will not be shared beyond the collection it is embedded. I know you could in theory use the molecule content embedded with another collection, but don't you think managing that will get messy?

[kksat]

No doubt @kksat. One can use the delegated (not the best name admittedly) driver to do that.

My point is that you can do that already with any molecule driver. All the actual provisioner code in the driver can be moved to separate collection and imported from collection in the driver. create.yml is just a playbook and one can import playbook from any collection. See for example https://github.com/redhat-sap/molecule-driver-azure/blob/main/molecule_driver_azure/cookiecutter/%7B%7Bcookiecutter.molecule_directory%7D%7D/%7B%7Bcookiecutter.scenario_name%7D%7D/create.yml

We created molecule driver for azure once and when we update only respective collection to change drivers behaviour. Because driver delegates all provisioning to ansible collection while still doing its tasks.

But that assumes that a driver (provisioner) will not be shared beyond the collection it is embedded.

I think there is a misunderstanding. If all the provisioned code is in ansible collection, that exactly means it can be shared and reused outside of molecule driver. As soon as there is data similar to platforms list in molecule.yml, collection playbooks can be used to provision instances without molecule driver with ansible-playbook command.

I know you could in theory use the molecule content embedded with another collection, but don't you think managing that will get messy?

I am not sure what do you mean here by "use the molecule content embedded with another collection" we included playbooks from ansible collection to driver.

That was not messy to change ansible collection that driver uses, that was actually simple, because collection development is well understood (as well as testing and sharing content). But how to test cookie cutter templates inside driver? - this is a challenge from my point of view.

but given this new found freedom of not having to write python code

I believe we still need python code, because driver (python code) does something, that ansible collection cannot do. For instance installing python dependencies, running sanity checks, providing schema for the molecule.yml, etc.

But we can certainly make python code much easier to create. In our driver we only changed package name and driver class name from community driver - This is what needed in most of the cases to create driver, if other features needed - python development (extended molecule use case)

TLDR

My point of view
We need python code - molecule drivers - because they do staff ansible cannot do
Moving provisioned content out of driver to ansible collection is easy to do, easy to test and share outside of molecule content.

[tima]

@kksat:

My point is that you can do that already with any molecule driver.

What I'm getting it that we are agreement on the principle, but differ on the implementation.

Our concern is that multiple drivers written in python is additional (and significant) maintenance and support responsibility we want to remove in order to streamline future development and reduce what any user has to learn in addition to their core Ansible skills.

I am not sure what do you mean here by "use the molecule content embedded with another collection" we included playbooks from ansible collection to driver.

My understanding of what you wrote was that the molecule driver (provisioner) would be in the collection you are testing. Did I misunderstand what you meant?

We need python code - molecule drivers - because they do staff ansible cannot do

I've not been a coder for a few years now. I'll leave @ssbarnea and @cidrblock to discuss in more detail. Could you be more specific though? Is it just that python is far better suited for programming logic or is there something else you have in mind here? I'm just not seeing why consolidating all playbook driven provisioning cannot be consolidated into one default driver that everyone can learn and use.

Moving provisioned content out of driver to ansible collection is easy to do, easy to test and share outside of molecule content.

That I agree. Again, I think the difference here is implementation details. The current proposal is one default driver that implements all of the provision and deprovisioning tasks as Ansible content. What I get from your suggestion is that there should be some provision/deprovision content in Ansible but executed by an environment specific driver like docker, podman, aws, azure and so on.

Is that an accurate summarization?

[kksat]

Our concern is that multiple drivers written in python is additional (and significant) maintenance and support responsibility we want to remove in order to streamline future development and reduce what any user has to learn in addition to their core Ansible skills.

I fully agree with this statement. My point is that you do not need to create V6 for that and cause a lot of disruption. This is completely doable with current version.

My understanding of what you wrote was that the molecule driver (provisioner) would be in the collection you are testing. Did I misunderstand what you meant?

Not, not like this. Molecule driver is separate - https://github.com/redhat-sap/molecule-driver-azure

Collection that molecule driver uses is separate - https://github.com/redhat-sap/molecule.driver
We do not have to update molecule driver if we want to add more functionality how driver works, we update this ansible collection.
In the collection you may find tests folder, that has all the tests for the collection itself.
Also there is a third collection (the one we develop) - https://github.com/redhat-sap/rh_operations. We use molecule driver above to test content in that collection (not only roles, but modules as well), but test code is not public.

Could you be more specific though?

For instance installing python dependencies, running sanity checks, providing schema for the molecule.yml, etc. These are all tasks that drivers do. I do not see how it is possible to implement them with pure ansible. Maybe I am missing something.

What I get from your suggestion is that there should be some provision/deprovision content in Ansible but executed by an environment specific driver like docker, podman, aws, azure and so on.

Yes, but it is only because driver does something that ansible cannot do - installing pip dependencies, schema, etc.
On the other hand we can have something like pip install molecule[docker] - to install all dependencies for default (delegated) driver for docker, pip install molecule[azure] to install all dependencies for azure. So task to install dependencies is moved from driver to molecule package. Not sure if it possible to move all driver responsibilities away from driver to ansible content or to molecule itself - main concern with molecule.yml schema. Another concern is about collection dependencies. If we have one default (delegated) driver to manage all types of infra, its ansible collection dependencies will be huge - docker, podman, all cloud providers, all virtualisation providers, etc.

[ssbarnea]

Please check #3947 which is a full example of using podman directly. While this makes the test body little bit better it should provide a much better stability over time, as it would not need a driver that needs to be updated to keep up with all podman collection changes.

Podman and docker modules are known to change their arguments quite often, so they will benefit a lot from a pure ansible approach. Other benefits are that users will be able to mix multiple ways of provisioning hosts, not being limited to a single one.

[kksat]

@ssbarnea Sorry Sorin, I do not see how PR #3947 is different from current approach for the podman driver. Looks like a lot of functionality been stripped and with this approach it will be only possible to reference container image, not more. But there are a lot of parameters in podman collection. Ok, podman driver is not referenced anywhere, is this a benefit your are looking for? But now to add functionality to the driver, molecule itself has to be changed. And how podman itself will be installed, because it is needed for molecule with this approach, molecule will depend on podman?

It one wants to add parameter network for containers, so created containers are in the same or different networks, "driver" has to be changed anyways (create playbook actually has to be changed), because it is part of the driver.
On the other hand we recently added parameter for the driver to create public DNS records in Azure for created instances (redhat-sap/molecule.driver#27) and driver not changed (only molecule collection). And there is even a test scenario for the created change https://github.com/redhat-sap/molecule.driver/pull/27/files#diff-91b4676280dfb1c15c591cd774e710fa760d047d4701dca1db51bda1b717c6d9

Compare your approach with https://github.com/redhat-sap/molecule.driver

[ssbarnea]

@kksat As the entire content of molecule.yml file can be accessed as a variable inside the playbooks, one can stick any needed information inside and use it to for creating networks or other resources needed. My example is kinda minimal, but gives the nature of container testing, it should be enough for someone to build on top of.

The idea of not using a driver is that we don't need to maintain the feature pairing between the driver and the 3rd party collection module(s) that are effectively doing the work.

Obviously that collection authors could write modules that would make the use of molecule easier (less verbose playbooks).

[konstruktoid]

How should the plugin rewrite/replacement be done?

And even though there might not be 3rd party modules, the (vagrant|docker|podman|...) "glue" still needs to be maintained.

[cidrblock]

We are going to make every effort to preserve driver functionality for version 6.

An emphasis on using the default driver shouldn't need to include completely breaking the exisiting drivers :)

[zhan9san]

@kksat

I do not see how PR #3947 is different from current approach for the podman driver. Looks like a lot of functionality been stripped and with this approach it will be only possible to reference container image, not more.

IIUC, the idea is that after this issue is fixed, there is no molecule plugin any more.

Previously, if someone wants to test an Ansible role, they have to use a specific molecule plugin which acts as a collection proxy/middleware. A proxy can fix 90% problems. If not, add another one.

The molecule plugin makes the interface unified across different collections(e.g. docker, vagrant and podman). We just need to define platforms in molecule.yml and then molecule plugin would handles create and destroy logic. But like @ssbarnea mentioned, the underneath collections change often, and it pushes molecule plugin has to change as well.

In the future, it's end user's responsibility to handle these logic which provides more flexibility, but it introduces more code in end user's code base.

But there are a lot of parameters in podman collection. Ok, podman driver is not referenced anywhere, is this a benefit your are looking for?

IMHO, the answer is yes.

But now to add functionality to the driver, molecule itself has to be changed. And how podman itself will be installed, because it is needed for molecule with this approach, molecule will depend on podman?

It depends on how collection handle this issue. If one collection requires some tools, with/without molecule plugin the tools has to be installed in advance.

It one wants to add parameter network for containers, so created containers are in the same or different networks, "driver" has to be changed anyways (create playbook actually has to be changed), because it is part of the driver.
On the other hand we recently added parameter for the driver to create public DNS records in Azure for created instances (redhat-sap/molecule.driver#27) and driver not changed (only molecule collection). And there is even a test scenario for the created change https://github.com/redhat-sap/molecule.driver/pull/27/files#diff-91b4676280dfb1c15c591cd774e710fa760d047d4701dca1db51bda1b717c6d9

For this specific scenario, you can write your own create.yml by leveraging azure.azcollection.

Please correct me if I say anything wrong @ssbarnea

[kksat]

IIUC, the idea is that after this issue is fixed, there is no molecule plugin any more.

That clarifies it. Thank you

[cidrblock]

We may or may not look into this in the future. It really depends on the uptake of ansible as the provionioner and if examples/references exist that meet or exceed the capabilties and quality of the drivers.