feat: replace ansible templating with makejinja

[onedr0p]

Succession to #1154 which tries to hand off templating to makejinja

Major benefit here is that the entire ./bootstrap/templates directory can be templated out in one command and it's done a lot faster than using Ansible. Another benefit is that the ./bootstrap/template dir can be 1:1 templated to ./

Ansible is still being used for validation for now and I don't see that changing unless there is a custom script built to handle that logic or there is another tool that makes it easy.

[mrueg]

I understand your motivation, I hope though it would be possible to undo this change.

A couple of arguments for ansible templating and not adding makejinja at the moment:

• Adding another templating engine adds more complexity
• Extending with other ansible playbooks and roles to the repository gets more difficult (e.g. sharing the same values is more difficult as well as then executing twice)

[onedr0p]

I wish Ansible could work the way I want to for templating, I've tried my best to make it faster but there's no async support or any other way to make it faster. The speed with makejinja is noticed faster. We're talking like < 2s compared to nearly 1.5 minutes for Ansible and it will only get slower as more files need templating.

Adding another templating engine adds more complexity

Both makejinja and Ansible use jinja so it's still the same templating engine and as the taskfile sets up a venv and pulls the deps it should be transparent for everyone.

Extending with other ansible playbooks and roles to the repository gets more difficult (e.g. sharing the same values is more difficult as well as then executing twice)

I'm not sure what you mean here, can you give an example? My hope is that I can also remove the need for Ansible for the config validation parts as well but I haven't found a tool I like to hand that off to yet.

[onedr0p]

I also wanted to say it's not just about the speed, maintaining the config templating in Ansible is cumbersome compared to makejinja. With Ansible you need a bunch of logic to make it happen, and separate out the add-ons and handle those independently. However with makejinja, it's very easy in that the templates directory gets rendered one to one with the output directory. The logic on if files or directories get included lives within the files and not some extra Ansible task.

[mrueg]

I wish Ansible could work the way I want to for templating, I've tried my best to make it faster but there's no async support or any other way to make it faster. The speed with makejinja is noticed faster. We're talking like < 2s compared to nearly 1.5 minutes for Ansible and it will only get slower as more files need templating.

How often would you execute it though? I assume it's only for the bootstrap.
If you're talking about the ansible execution against multiple hosts there's https://github.com/mutagen-io/mutagen which could be helpful

Adding another templating engine adds more complexity

Both makejinja and Ansible use jinja so it's still the same templating engine and as the taskfile sets up a venv and pulls the deps it should be transparent for everyone.

Extending with other ansible playbooks and roles to the repository gets more difficult (e.g. sharing the same values is more difficult as well as then executing twice)

I'm not sure what you mean here, can you give an example? My hope is that I can also remove the need for Ansible for the config validation parts as well but I haven't found a tool I like to hand that off to yet.

I use https://galaxy.ansible.com/ui/repo/published/devsec/hardening/ and https://galaxy.ansible.com/ui/standalone/roles/artis3n/tailscale/ in addition to this repository. I added them to my internal fork of this repo and the vars files include custom settings for them as well. I assumed I would need to move that over to makejinja as well

[onedr0p]

How often would you execute it though? I assume it's only for the bootstrap.
If you're talking about the ansible execution against multiple hosts there's mutagen-io/mutagen which could be helpful

It's not run a lot but as I mentioned with more addons and more files the time to execute this will only get worse. Also it is annoying that everytime I need to test things I need to wait that 1.5m+ for the results.

I looked into mutagen when I was trying to come up with a faster way of templating the config. It's really shines when you have multiple SSH hosts like you said. It won't help with speeding up templating out files on a single host, like here with localhost. Another issue with mutagen is that newer versions of Ansible may not be supported. The team behind that project are slow to make releases that are compatible with later Ansible versions.

I use galaxy.ansible.com/ui/repo/published/devsec/hardening and galaxy.ansible.com/ui/standalone/roles/artis3n/tailscale in addition to this repository. I added them to my internal fork of this repo and the vars files include custom settings for them as well. I assumed I would need to move that over to makejinja as well

For the most part adding that to the Ansible playbooks that get templated out should be as easy as adding the logic to ./bootstrap/templates/ansible. I don't see this being hard to migrate over, the major change is with the jinja templating delimiters/tags which I changed because renovate freaks out sometimes on parsing YAML files that have these in them.

• Instead of {% ... %} use #% ... %# and instead of {{ ... }} use #{ ... }# (this change also removes the need for the {% raw %} ... {% endraw %} blocks)

Edit: I see your point on having custom logic in the vars files, that might take a bit of migrating. I wasn't doing that here so it didn't come to mind that this would have been a problem. That logic could be migrated over to the templates instead of being in the vars files?

[onedr0p]

I also wanted to mention that if you are using any custom ansible utils for parsing / mutating vars, those will need to be added to the loader.py. It might be a bit cumbersome at first but shouldn't be too hard to update this file with some custom jinja functions / filters. This part took me the longest to understand but over all it's not that bad and makes it so you can even build functions or filters outside what ansible supports.