Document molecule as a deployment tool

[grzegorznowak]

Proposal: Why not utilize Molecule in different ways than for ansible role-testing ?

Author: Grzegorz Nowak <@grzegorznowak>

Date: 2019-MM-DD

• Status: New
• Proposal type: branding, sort of ?

Motivation

Hey guys, I've been using molecule for about 2 years now, and while it initially indeed served me well for only integration/unit testing of Ansible roles I was brewing in-house for our servers, nowadays I'm actually using it for various infrastructure deployments. Including live services.

Bit, by bit, while familiarizing myself with the tool's philosophy and architecture it became clear the way it was built, outgrew the test-only scenario it's author announced it to be solving.
Big kudos for that btw.!
I truly appreciate the ability to define my servers in the language of Ansible roles, tied together with a sexy Molecule yaml and have that deployed/upgraded wherever needed.

I mean it's already clear that molecule can do all this if you know it enough, so why bother making it more obvious ? Well the thing is for a newcomer, who doesn't really understand the full potential of the tool, it will probably just sound too narrow and too complex a tool, to bother starting with.
Where in fact, at least for me, it's the only way I'm using LXC/LXD these days for dev machines, and can't even imagine going back to pre molecule days, and not having my teams' dev clusters provisioned using it's capabilities. I find it way easier and lighter than doing dockerfiles and composing them (yes I know not an objective metric but if works for me, might work for someone else too) .

I know many people will fallback to using dockers anyway, because in theory they are easier to go about... until you realize that in practice they aren't!
With only a few touches of ansible (galaxy), combined with molecule's config one can setup perfectly fitted environment, that is already on steroids with testing/linting/idempotence-checking etc. simply put, molecule'd!

If you guys agree that we should emphasize this side of the project more, and get it into boots of proper orchestration platform (that it already is behind the scenes), I'm happy to provide some not-trivial real-life configs that encapsulate clusters of servers for real-life development. And have them plug into documentation... somewhere fitting.

It's hard to tell if that's even on cards, so would appreciate being enlightened as to why we wouldn't want molecule be on par with tools like Docker Compose or Kubernetes, if we do.

Any thoughts on that lads ?

Problems

Possibly will need to be adding more test scenarios to cover this particular use case

Solution proposal

Basically the solution lies in documentation mostly, as the software is already capable of doing that easily.

[ssbarnea]

That is a good idea. I think that first step would be to pinpoint few such use cases so we can document them and link to a repository that contains them. I am really curious about the subject.

[grzegorznowak]

Hey @ssbarnea , I'm more than happy to hear that. Let me shove your way the simplest use case I could find in my (client's in this case) repos:
https://github.com/spottmedia/rust-wasm-starter

the key bit with setup instructions lie in the README.md under the Provisioning section.

I'm yet to prepare grander scale setup to be uploaded publicly (gotta strip those private bits first), but hopefully this will give you some overview of the approach I'm using on my end. Not the bleeding edge in term of the best configs' foldout, but close to.

[decentral1se]

Hey @grzegorznowak, nice write up and interesting to hear!

Concretely, I think you could focus on:

1. As @ssbarnea said, pick your (ideally simplest) use case ("hey kids, molecule can deploy too!")
2. Write a documentation page on it. Perhaps in the FAQ to start?

I would prefer to see a "here is my approach" narrative explanation so that the idea is first clear on what can be done. Then if you want, link to configs and more general nuts and bolts stuff.

A lot of people still struggle with the command line interface and the concepts of Molecule as it is for the purpose of testing roles. I don't think the idea will reach people if you dump configs and complicated setups.

Looking forward to see what you do!

[grzegorznowak]

hey @decentral1se, thanks buddy.

Personally I believe in the "simplest use case" only if one can back it up with a proper live-ish solution, to begin with. Like, I've seen too many basic samples that then become impossible to manage once you start trying to deliver production grade. And that's not Ops specific of course.
But so then I found molecule super flexible in that regard and easily scalable.

The link I'd given is actually already the simplest I can think of, with a real-dev-life application to back it up, so if you haven't already please have a sneak peek and see what parts might be unclear. Will appreciate any feedback.
I've tested it against a couple of devs my side, and the instructions seemed sufficiently clear to follow. Or at least they claimed they did end up with the dev machine spinned up as an LXD container. Maybe just didn't want to sound noob, I don't know ;)

To me concepts and CMD line is not any harder than other orchestration tools I played with, so that's interesting to hear people are struggling.
I mean, with this exercise, given we're re-purposing a tool for a task that it wasn't built around in the first place, it might feel a bit forceful at certain stages, but really not that much, which I hopefully will demonstrate with follow up samples.

I feel motivated due to you being open to giving the project a wiggle, so cheers boys!

[decentral1se]

I'm sorry I can't put more time into giving it a look but I'll happily review a PR 👍

[decentral1se]

Hey @grzegorznowak, after #2415 is merged, you could see about adding a "Advanced Use Cases" section and start to provide your configurations and tips on how to use Molecule in this way. Let me know if you have any issues, would be great to see this included from your end!

[grzegorznowak]

Thanks for letting me know @decentral1se.
Still mid pretty massive redo of our landscape using exactly the molecule-driven approach, so will have tons of real-life (or as close to as possible) examples to feed on here.
Hopefully may have some spare time this weekend, so let's assume me doing anything in theAdvanced Use Case section around that time at earliest. Cheers!

[grzegorznowak]

hey @decentral1se @ssbarnea ,
so let me actually get rolling this with addition of the real-world example repository, describing molecule as a provisioning tool in as tender words as I could come up with:
https://github.com/grzegorznowak/molecule-provisioning-example

I certainly em flawed by using this approach for so long I might not even know where the bits needing explanation lie anymore, so might need some help de-mudding bits there.

The giveaway of this particular write-up might be either copy-paste fragments we will want to use in docs here or just keeping the project stand-alone, and have it linked back.
Probably more happy to do the latter as this setup is more about actual codes and how they tie together, rather than describing what could be potentially done.
As I've mentioned before I didn't want to drift into the simplest use institute as it simply won't show the full benefits of molecule-driven projects, and I know people are more keen to use something when they can just copy-paste-tweak into their requirements.

On a margin, lack of examples is something that I think can be attributed to molecule's rate of adoption by newcomers, as I here had to find some answers myself too (around Goos verifies that I planned to use) and it was eventually a journey back and forth between google/docs/sources, until I finally understood where all bits should fall into.

Let me know how you see it coming all together.

As a side note no. 2, when putting the project up together I've stumbled upon https://blog.codecentric.de/en/2018/12/test-driven-infrastructure-ansible-molecule/
that describes very much the same concept as I'm proposing. Using simpler landscape and more elaborate words with possibly more-newcomer-friendly instructions.
I guess if we just had links to projects like this in docs, it might be enlightening already ?

[grzegorznowak]

Hey guys,

in case it's helpful in any way we have bootstrapped another molecule-oriented project for some simple cookie-cut containers:
https://github.com/spottmedia/molecule-cookie-cutter

It started as a by product of us wanting to test cover our in-house Ansible roles and configurations but we figured out it might benefit people (and project itself) more if spread out.

Personally I'm still a big believer in what molecule offers in terms of infra setups and em using it with increasing degree of confidence to deploy more and more complex clusters DEV or LIVE alike.
So hoping people will find this contribution useful too.

[ssbarnea]

Last month I created https://github.com/ansible-community/protogen for the sole purpose of using it as a reference implementation about how others should write and test ansible code.

Sadly I did not had time to work more on it, even if I was able to convince @geerlingguy to join the effort. Maybe others can help on making it a reality?

The goal of the project is not to become a template, at least not a cookiecutter template. Templates are good when you are new and start from scratch by they cannot be used on exiting projects. Existing projects need manual refactoring which can take many months, but when you have a model to follow is much easier to look at it for "inspiration" and to copy paste only the bits that do apply for your case.

[grzegorznowak]

I second what you said about templates @ssbarnea . And for that I've released the full-blown cluster setup a few months back to show off some of the advanced molecule techniques that we are using to build our stack. Happy days, looking behind...

Anyway! So I'm all for unifying whatever can be unified, but I won't say it will be easy given plethora of possible options one has here. If I was a newcomer I definitely would like to see some "this is how you might do it" kind of tease, but only to discover I won't be actually following it in short while, with growing confidence and understanding what my actual scenarios are.

I was exploring approach of wrapping creation of molecule configs with Wizards - ie. to create replicas of webservers then plug them into load balancer on our production environment - and I think this could be one of the entry-level applicable approaches, but also one that would become obsolete for many users pretty quickly.

Let me know if you see a field upon which any of this work can be merged with yours, be it by linking back, improving upon or else, and I'd be happy to make it happen. Provided those goals align with the vision of clients/projects that are main catalyst of any work I'm able to share with community eventually.
As a standard, any new Molecule/Ansible code I push is already equipped with end-to-end CI and is able to execute molecule test without a hiccup. OR at least should.