molecule-docker + systemd : what now?

[mrzor]

Greetings,

I believe I've experienced the effects of systemd/systemd#19245 (I believe) in that systemd containers will not function properly anymore with systemd 248.2-2-arch. This makes molecule-docker unsuitable for testing purposes for impacted hosts (Arch Linux starting last month, FC34 (?), soon more (?)).

While I can, and will, turn to other molecule plugins, I'm very much interested in ways to fix this or to workaround it.

Diagnostics

What happens is that there is only one systemd process in the container at PID 1 and that's it.

I wasn't able to get any logs out of systemd, or any kind of output. - so, no journal, and console doesn't print or write anything as far as I can tell.

Relevant issue

I wrongly posted this as an issue first, to which ssbarnea answered:

Use of systemd under docker was always problematic and not recommended use case. There is nothing molecule driver can do about that. If you need systemd, you are likely to want to use real VMs. Please use https://github.com/ansible-community/molecule/discussions next time.

[maxhoesel]

I know it's been a while since this has been posted, but I too am looking for a way forward with systemd-enabled molecule tests. Has anyone tried other container engines like podman?

I guess the other alternative would be full VMs using something like vagrant, but that seems awfully "fat" for simple test cases like the ones I have

[tadeboro]

We started replacing Docker with Podman and were able to port most of our test suites without issues. Running systems inside containers is one of the supported scenarios for Podman so things should be future-proof now.

[justin-f-perez]

Hello,

I have had good experience with podman once I figured out how to get it going... One place I had trouble with was "what can I pass to podman through molecule, and how?" This doesn't seem to be documented in the HTML docs anywhere, but the docstring for the driver is pretty comprehensive:
https://github.com/ansible-community/molecule-podman/blob/f12bb63cbdc745af8b530a462d7aea7df50ccb5e/src/molecule_podman/driver.py#L41

here's roughly what the platforms section of my molecule/default/molecule.yml looks like, it has some comments you may find useful. Please keep in mind that some of these options may be specific to my use case so you should adjust accordingly

platforms:
  # NOTE: platform options are defined by the driver. See class docstring:
  #   https://github.com/ansible-community/molecule-podman/blob/main/src/molecule_podman/driver.py
  - name: molecule_instance
    # i dont think we have anything too special in here, except maybe 'CMD [ "/sbin/init" ]'
    dockerfile: Containerfile.j2
    # A popular image choice: geerlingguy/docker-${MOLECULE_DISTRO:-ubuntu2004}-ansible
    # However, for our use case the plain ubuntu image seems to produce smaller builds
    image: ubuntu:20.04
    # by default molecule will pass an infinite sleep loop command to keep the
    # container running. We override that behavior with the init system (systemd)
    command: /sbin/init
    # podman _should_ auto-detect that we want systemd based on the command and
    # configure the container runtime appropriately, but just to be explicit:
    systemd: true
    # pre_build_image is a misnomer; this flag actually controls whether or not
    # the `dockerfile` template is expanded and built (false=build; true=skip)
    pre_build_image: false
    # we point these domains to the local ip so we can test
    # production URLs without hitting production services. docker/podman depend
    # on /etc/hosts for proper linking, so we can't just modify in converge.yml
    # We hard-code these here because I think the only way to pull them from somewhere
    # else is to override create.yml, e.g., to parse nginx templates, which is prob. not worth it.
    etc_hosts:
      example.com: 127.0.0.1
      anotherexample.com: 127.0.0.1
    # our use case is a bit unusual; typically you wouldn't run so
    # many webapps and processes in a single container, which can lead to many
    # open file handles (especially during initial setup). We adjust accordingly
    ulimits:
      - nofile=10000:10000

hope this is helpful!

[justin-f-perez]

PS: I don't think molecule_instance is the default name (default might have been just instance?) so you may want to change it back to the default if other parts of your project depend on it (e.g., if you have any host_vars defined in molecule.yml under provisioner:inventory:host_vars)

[netopsengineer]

New to Molecule, and noticed essentially every older article out there would test with a Geerling role like Docker, and use one of his systemd images, and it constantly failed for me on my Mac, but worked when testing on another machine. Come to find out, it seems the latest Docker Desktop after 4.3.0 changed some of the behaviors around cgroups, and what once worked before, now doesn't. I have not seen alot of guidance as a beginner, if systemd was never an intended use case, how one tests for service related items. I will attempt this Podman setup, but what other drivers are available that are recommended for properly testing something like this?

[justin-f-perez]

just FYI in case you were wondering if the example I pasted above will work for mac, I developed and run it on an arm64 mac. The vagrant driver wasn't a good fit for me due to lack of virtualbox support for arm64 and I was unable to get the libvirt plugin for vagrant working. Unfortunately, my notes are missing some details from the vagrant-libvirt attempt (e.g. what the exact error message I received was), but here's what I have for notes from that attempt (circa early Jan 2022):

---

Since virtual box doesn't work on m1 macbooks, I tried out a few other virtualization technologies.

There is a libvirt driver for molecule but per the README:

Please note that this driver is currently in its early stage of development, do not even try to install or use it until this message is removed.

Libvirt is an API/wrapper around qemu, which is a hardware emulator for VMs (see also: What is libvert, qemu, and kvm? Vagrant is a plugin-oriented CLI for managing VMs that can use libvirt as a backend. I tried using molecule-vagrant with the vagrant-libvirt plugin:

brew install vagrant qemu libvirt
brew services start libvirt

# Attempt 1 of installing the plugin; fails as it can't find libvirt header
CONFIGURE_ARGS='with-ldflags=-L/opt/vagrant/embedded/lib with-libvirt-include=/opt/homebrew/Cellar/libvirt/7.10.0_1/include/libvirt/ with-libvirt-lib=/opt/homebrew/Cellar/libvirt/7.10.0_1/lib/' \
  GEM_HOME=~/.vagrant.d/gems/2.7.4 \
  GEM_PATH=$GEM_HOME:/opt/vagrant/embedded/gems \
  PATH=/opt/vagrant/embedded/bin:$PATH \
  vagrant plugin install vagrant-libvirt

# Attempt 2 of installing the plugin
env CONFIGURE_ARGS="with-libvirt-lib=$(brew --prefix libvirt)/lib with-libvirt-include=$(brew --prefix libvirt)/include" vagrant plugin install vagrant-libvirt

Going any farther than this probably requires rebuilding vagrant itself or its dependencies from source. Maybe this was for the best. Molecule -> ansible -> vagrant -> libvirt -> qemu is a long dependency chain anyways.

[justin-f-perez]

Another potential pitfall for the folks in this thread to maybe be aware of... I had originally started with docker like this:

$ pip install molecule[lint,docker]

and then, I don't recall which, but I did one of these two:

$ pip install molecule-podman
$ pip install molecule[podman]

This was causing errors every time I did molecule reset. Unfortunately the exact cause is long gone from memory, but I want to say it was related to some setup/assumption related to installing w/ the docker extra first. In any case, the fix was to simply reinstall molecule by blowing away my virtual environment and making a new one with molecule[lint,podman].

[mrzor]

I recommend to most of my colleagues to stear clear of installing tooling using pip directly (even with virtualenvs) and to favor pipx, who will manage virtualenvs for you transparantly.

pipx install ansible then pipx inject ansible molecule, pipx inject ansible ansible-lint etc

Main benefit of pipx over venvs is that you have all your tools on $PATH. So you can use ansible and, say, pre-commit, in the same shell, while keeping their deps neatly isolated.

pipx upgrade-all is also super nice to keep everything up to date without activating a bunch of venvs and running pip.