docs: improve documentation around podman authentication credentials

[shoenig]

See hashicorp/nomad-driver-podman#265 (comment)

Also cleanup the docker authentication docs, there's room for some improved clarity.

[rwenz3l]

I think the most crucial bit is to mention it in the plugin options here: https://developer.hashicorp.com/nomad/plugins/drivers/podman#plugin-options

I went through a bunch of code, issues and pull-requests now and was confused if I have to tell my podman service to use the auth.json or if nomad (client-agent) is managing that or if it needs to be part of the job spec.

Some notes:

podman-login can use the auth.json to authenticate pulls with the EnvironmentVariable REGISTRY_AUTH_FILE. Unfortunately setting this for the service (systemd/sysconfig) does nothing, because it needs to be in the active working environment for podman to use.

This works:

$ export REGISTRY_AUTH_FILE=/etc/containers/auth.json
$ podman pull private-registry/image:tag
# or
$ podman run private-registry/image:tag

This does not and should not work,
because podman needs that in it's active env:

$ cat /etc/sysconfig/podman
REGISTRY_AUTH_FILE=/etc/containers/auth.json

$ systemctl restart podman
$ podman pull private-registry/image:tag
# pr
$ podman run private-registry/image:tag

Now for the interesting bit in nomad, this works:

# nomad.hcl on client-agents
plugin "nomad-driver-podman" {
  config {
    auth {
      config = "/etc/containers/auth.json"
    }
    socket_path = "unix://var/run/podman/podman.sock"
    gc {
      container = false
    }
    volumes {
      enabled = true
    }  
  }
}

This did not work:

$ cat /etc/sysconfig/nomad
REGISTRY_AUTH_FILE=/etc/containers/auth.json

# In your nomad-systemd-service:
# EnvironmentFile=-/etc/sysconfig/nomad

$ systemctl start|restart nomad

$ nomad run <job using podman driver + image from private registry>.hcl

I would think the last one should have worked too, but for me it did not.
Setting the auth in the podman driver is the most reasonable to me, it just needs to be actually documented.

Also worth showcasing is how the task actually gets to those credentials, because it was not very clear from the code.
My understanding now is that it resolves the registry-info from several places and found this to be most helpful:

https://github.com/hashicorp/nomad-driver-podman/blob/ff01fc9491ab5a0505424a330eac55530473173c/registry/authentication.go#L135-L146