molecule-driver-contrib: community managed drivers

[decentral1se]

Regarding #1473 (comment).

Hi @bandit145, I am hesitant to commit drivers which rely on community members maintaining drivers which most people do not use. To be honest, I think we already have too many drivers. Most people simply use Vagrant and Docker. Very few people use the others, and I don't actively test the others.

And since I am working on #1458, I am of course interested in this.

So, have you given any thought to a molecule-driver-contrib which is pluggable from the end-user side and introduces no maintenance burden on you all here? We could say that those drivers are maintained by the community and those community members are responsible for them. No guarantees are given back in molecule core. That would steer all the maintenance tickets away from here.

I would guess core devs here would know the best way that is possible, if this is feasible.

I'd be happy to QA test any effort here in plugging in my Linode driver functionality.

[wilmardo]

I am interested in this as well since I am working on making Molecule ready for Windows development (Hyper-V and WinRM support) since the shop I am working at is Windows orientated.
For my personal projects I am actively using LXD and willing to support this driver (and the Hyper-V solution in the future).
In my opinion community members supporting some drivers is a good idea, the contrib you suggest might be overkill. As long as the drivers use an Ansible native module there shouldn't be that many problems since there is an upstream maintainer.

If just the tickets get issued to the correct maintainer that should fix most of the problems (I think open for discussion of course). When a maintainer seems inactive over a longer period of time the driver should be removed from Molecule core.

Edit:
Another thing to consider is a contrib is a real stretch without a publicly visible CI. The community maintainers then must test all the changes locally, this takes a real time effort.

[decentral1se]

If just the tickets get issued to the correct maintainer that should fix most of the problems (I think open for discussion of course). When a maintainer seems inactive over a longer period of time the driver should be removed from Molecule core.

I am not sure if you are aware, but this is basically what is already happening. This ticket is about talking about other potential solutions because it seems this is not working for core devs (this may not be the case entirely but from the original motivating comment, I want to start this discussion in case that is so).

Another thing to consider is a contrib is a real stretch without a publicly visible CI. The community maintainers then must test all the changes locally, this takes a real time effort.

Each community driver maintains its own CI. This CI consists of: molecule init role -r foobar -d <community-driver> && cd foobar && molecule test. That's it, right? If the thing passes, the community driver is still working.

[bandit145]

Maybe an option would be to handle drivers like Ansible handles module discovery. As an example you could have a custom driver in a "drivers" folder in the cwd and molecule could pick that up and could use the driver(s) in that directory.

[retr0h]

So you can more or less do this today with the 'delegated' driver. The delegated driver defaults to managed: true, which means as the developer you bring your own create/destroy playbooks and conform to the instance_config API in create.yml.

We have had users already use this for a vmware driver. Community members simply drop in 3rd party provisioning playbooks, and set the driver to delegated, and you should be good to go.

[wilmardo]

So you can more or less do this today with the 'delegated' driver. The delegated driver defaults to managed: true, which means as the developer you bring your own create/destroy playbooks and conform to the instance_config API in create.yml.

The point with that approach is that the provisioning playbook resides with each role, whereas the drivers supplied with molecule keep the roles clean with only a molecule.yml. That was my main reason to update the LXD driver, I couldn't find a good workflow for custom playbooks within each role.

Also it might be a good idea to create some delegated examples to keep people from reinventing the wheel if this is the preferred approach.

[retr0h]

This is true, but its also creating excess work when its not really necessary. We want a plugable way to add drivers, that exists today.

[wilmardo]

Is it a possibility to do something like Ansible does with for example Azure (ansible[azure)]?
pip install molecule for Molecule with Docker and Vagrant
pip install molecule[coomunity_driver_name] and move these community drivers into a specific directory in the repo.

This keeps the knowledge and drivers within Molecule but lightens the burden on the core maintainers.

[decentral1se]

So you can more or less do this today with the 'delegated' driver. The delegated driver defaults to managed: true, which means as the developer you bring your own create/destroy playbooks and conform to the instance_config API in create.yml. We have had users already use this for a vmware driver. Community members simply drop in 3rd party provisioning playbooks, and set the driver to delegated, and you should be good to go.

You know, I'd be down to just do this and leave it at that. However, there is one thing this doesn't cover, I think. In my Linode PR, I had to add an actual driver python module (see https://github.com/metacloud/molecule/pull/1458/files#diff-f31994c777cf7153ef405f9b02ab920a) which did some fiddling with the SSH password and instance name for connection details. This is not covered by the delegated method, right?

[bandit145]

@lwm I believe you can set those bits in the delegated declaration (molecule.yml). I did consider using the delegated driver for mine originally but since my use case is company wide I thought it would be nicer to have it all in the main project if possible.

[watsonb]

I like the spirit of this discussion and can see both sides of the coin. We've got a boat load of Ansible roles and try very hard to leverage molecule for role development and automated CI with Jenkins. We're a mixed Linux/Windows shop and have to support both platforms. And, to make matters worse, we're a hybrid cloud shop using Azure and on-prem VMWare.

To date, I've leveraged the delegated driver and we can spin molecule instances up/down on-prem in VMWare when we write/test roles that don't play well with Docker. The downside of this is that each role has these drivers in them and keeping consistency across the growing number of roles we have is a royal pain. Something pluggable would be cool, but I see how it could be difficult to support a wide variety of use cases. I've had success adapting the delegated driver to spin up/down VMs in Azure as well.

My most difficult pain point at the moment is supporting Windows. I've got some Jenkins slaves with Virtualbox/Vagrant on them and built a Windows image with Packer, but the molecule vagrant driver with these Windows vbox images and the winrm stuff just isn't reliable enough. And then there's the whole verification piece that can be messy and without being able to RDP or access the Windows GUI it can be tough to visually check this stuff as well.

I think I'd like to try the delegated driver spinning up Windows VMs in VMWare (or even Azure), but I can't figure out how to coerce Molecule to retain/use Ansible/WinRM stuff to connect to these things for converge/verify steps after they've been created (the create/destroy is the easy part). Essentially I need to adapt the following to store the WinRM bits, but I have no idea how to make Molecule make use of it once it is stored:

- name: Populate instance config dict
      set_fact:
        instance_conf_dict: {
          'instance': "{{ item.instance.hw_name }}",
          'address': "{{ item.instance.ipv4 }}",
          'user': "{{ molecule_yml.driver.ssh_user }}",
          'port': "{{ molecule_yml.driver.ssh_port }}",
          'identity_file': "{{ molecule_yml.driver.ssh_identity_file }}"
        }
      with_items: "{{ server.results }}"
      register: instance_config_dict
      when: server.changed | bool

    - name: Convert instance config dict to a list
      set_fact:
        instance_conf: "{{ instance_config_dict.results | map(attribute='ansible_facts.instance_conf_dict') | list }}"
      when: server.changed | bool

    - name: Dump instance config
      copy:
        # NOTE(retr0h): Workaround for Ansible 2.2.
        #               https://github.com/ansible/ansible/issues/20885
        content: "{{ instance_conf | to_json | from_json | molecule_to_yaml | molecule_header }}"
        dest: "{{ molecule_instance_config }}"
      when: server.changed | bool

    - name: render ssh_config for instances
      template:
        src: 'ssh_config.j2'
        dest: "/tmp/molecule-ssh-config"
      when: server.changed | bool

Once created, I'd like to be able to molecule converge and molecule verify on these Windows-based VMs. As a bonus I could use native VMWare/Azure methods to get access to the GUI for manual/visual inspection.

I feel like I'm close between my experiments with Vagrant and the delegated driver, but I can seem to put them together into something reliable.

[decentral1se]

but I can't figure out how to coerce Molecule to retain/use Ansible/WinRM stuff to connect to these things for converge/verify steps after they've been created (the create/destroy is the easy part). ... but I have no idea how to make Molecule make use of it once it is stored

I think this is the same point as #1479 (comment)?

If we could extend the delegated driver concept to allow for a driver python module as well, then it would be more useful. However, this still doesn't get away from the pain point that when you have 100+ roles and you 'eject' your delegated driver plays into them and then you want to make a change to them all ...

#1479 (comment) is interesting!

[retr0h]

If we could extend the delegated driver concept to allow for a driver python module as well, then it would be more useful. However, this still doesn't get away from the pain point that when you have 100+ roles and you 'eject' your delegated driver plays into them and then you want to make a change to them all ...

They can be shared.

[decentral1se]

Right, so you could also submodule the delegated playbooks too. Perhaps not such a major issue.

Should we also try to hook in a driver module there as well?

[wilmardo]

The only issue I see with shared and delegated is that the knowledge stays with the user. Take for example @bandit145 who wants to use VMWare and starts building a solution. But there seem to be users having this working already as @retr0h states. This is time wasting and doesn't move the project forward.
Also for people looking in the docs search the driver section and see the current options. They might move on since there platforms isn't in this list ans consider it not supported.

Might be an idea to keep the delegated setup but add examples in the docs and questions about these examples get assigned to the one who wrote and uses the example?

I personally still don't see the issue with the solution proposed in #1479 (comment)

[watsonb]

I think it is a good idea to keep the delegated setup as a catch all and agree it might be a good idea to have a place for people to post their solutions so we're not all re-inventing the wheel. I certainly don't expect the molecule team to document every conceivable use-case as this delegated capability is very flexible but does come with some "gotchas" that aren't apparent to everybody.

I finally put some cycles into my Jenkins/Molecule/Vagrant/VirtualBox/Windows (whew!) setup and got it working!

[retr0h]

I personally still don't see the issue with the solution proposed in #1479 (comment)

Nobody said there was a problem with the issue, its just a matter of implementing it cleanly. I think we should create a table in the documentation that states the driver, the maintainer(s), and where the driver could be found (URL), for people to submodule or gilt it into the project. IMO, this solves the immediate problem.

How do we feel about this? Seems like a good starting point. Suppose we do implement in #1479 (comment) we can list the pypi package instead of the git repo.

[decentral1se]

This sounds great to me! 👏 👏 👏

[decentral1se]

Right, https://groups.google.com/forum/#!topic/molecule-users/O1a_YNAjy-Q probably solves this.

[wilmardo]

Wow that is a hefty step! Hopefully the transition goes smooth and the project will continue to grow 🎉

[gundalow]

Hi,
As detailed in https://groups.google.com/forum/#!topic/molecule-users/O1a_YNAjy-Q Ansible to adopt molecule and ansible-lint projects.

I'm in the process of doing the technical move. Feel free to reach out in #molecule-users for more info

So hopefully this solved this issue

[decentral1se]

🔥