Nested redhat podman containers don't get subscription from host (common/pkg/subscriptions)

[gardar]

When running nested redhat containers (container inside a container) the nested container does not get the redhat subscription from the host, whereas the container that runs directly on the host gets the subscription from the host.
Subscription manager does not run in container mode and the repositories don't exist inside the nested container.

I traced the issue down to the differences in /run/secrets and I'm actually able to get the subscription to work correctly in the nested container by simply bind mounting /run/secrets on the nested container from the parent container.
Upon closer inspection I found out that the yum repos from the host actually exist on the nested container under /run/secrets/redhat.repo but the certificates don't exist, so bind mounting /run/secrets/etc-pki-entitlement and /run/secrets/rhsm seems to be sufficient.

[rhatdan]

Not sure why this is not happening automatically.

This file is supposed to tell Podman to mount the directory into the container.

more /usr/share/containers/mounts.conf
/usr/share/rhel/secrets:/run/secrets

[gardar]

Strange, the file looks the same on both the host and the container

host:

root@host:~# cat /usr/share/containers/mounts.conf
/usr/share/rhel/secrets:/run/secrets

root@host:~# cat /etc/redhat-release
Red Hat Enterprise Linux release 8.8 (Ootpa)

root@host:~# podman --version
podman version 4.4.1

container (ubi8):

[root@c84bfc5e42c1 /]# cat /usr/share/containers/mounts.conf
/usr/share/rhel/secrets:/run/secrets

[root@c84bfc5e42c1 /]# cat /etc/redhat-release
Red Hat Enterprise Linux release 8.8 (Ootpa)

[root@c84bfc5e42c1 /]# podman --version
podman version 4.4.1

[gardar]

I can spot a difference by looking at the contents of /usr/share/rhel/secrets

host:

root@host:~# tree -l /usr/share/rhel/secrets/
/usr/share/rhel/secrets/
├── etc-pki-entitlement -> /etc/pki/entitlement
│   ├── 3654486719169445681-key.pem
│   ├── 3654486719169445681.pem
│   ├── 3868586155971816315-key.pem
│   ├── 3868586155971816315.pem
│   ├── 4647619488388425268-key.pem
│   ├── 4647619488388425268.pem
│   ├── 4918930088950278333-key.pem
│   ├── 4918930088950278333.pem
│   ├── 4930352730550253178-key.pem
│   ├── 4930352730550253178.pem
│   ├── 5695968016688669032-key.pem
│   ├── 5695968016688669032.pem
│   ├── 6339045514570628913-key.pem
│   ├── 6339045514570628913.pem
│   ├── 7856845756430013530-key.pem
│   ├── 7856845756430013530.pem
│   ├── 8623734886189020737-key.pem
│   ├── 8623734886189020737.pem
│   ├── 8811509968829781298-key.pem
│   └── 8811509968829781298.pem
├── redhat.repo -> /etc/yum.repos.d/redhat.repo
└── rhsm -> /etc/rhsm
    ├── ca
    │   ├── katello-default-ca.pem
    │   ├── katello-server-ca.pem
    │   ├── redhat-entitlement-authority.pem
    │   └── redhat-uep.pem
    ├── facts
    ├── logging.conf
    ├── pluginconf.d
    ├── rhsm.conf
    ├── rhsm.conf.kat-backup
    ├── rhsm.conf.rpmnew
    └── syspurpose
        ├── syspurpose.json
        └── valid_fields.json

6 directories, 31 files

container:

[root@c84bfc5e42c1 /]# tree -l /usr/share/rhel/secrets/
/usr/share/rhel/secrets/
├── etc-pki-entitlement -> /etc/pki/entitlement
├── redhat.repo -> /etc/yum.repos.d/redhat.repo
└── rhsm -> /etc/rhsm
    ├── ca
    │   ├── redhat-entitlement-authority.pem
    │   └── redhat-uep.pem
    ├── facts
    ├── logging.conf
    ├── pluginconf.d
    ├── rhsm.conf
    └── syspurpose
        └── valid_fields.json

6 directories, 6 files

The /etc/pki/entitlement is empty on the container and the entitlements from the host are located under /etc/pki/entitlement-host

Perhaps it's missing a logic which is something like this pseudo code:

if /etc/pki/entitlement-host then mount /usr/share/rhel/secrets/etc-pki-entitlement /etc/pki/entitlement-host
else mount /usr/share/rhel/secrets/etc-pki-entitlement /etc/pki/entitlement

[rhatdan]

Dnf is supposed to have a plugin that puts the entitlements back together, I am not sure how that works.

[gardar]

That might be what's enabling the repos/subscriptions on parent container, but I don't think that's going to work on the nested container, as neither /etc/pki/entitlement nor /etc/pki/entitlement-host gets populated on the nested container

[rhatdan]

It is best to add -v /run/secrets:/run/secrets in the internal container.

[rhatdan]

Perhaps we should modify the mounts.conf file inside of the container to do that by default.

[gardar]

It is best to add -v /run/secrets:/run/secrets in the internal container.

But won't cause potential conflicts with --secret ?

• target=target : Target of secret. For mounted secrets, this is the path to the secret inside the container. If a fully qualified path is provided, the secret is mounted at that location. Otherwise, the secret is mounted to /run/secrets/target. If the target is not set, the secret is mounted to /run/secrets/secretname by default. For env secrets, this is the environment variable key. Defaults to secretname.

I think
-v /run/secrets/etc-pki-entitlement:/run/secrets/etc-pki-entitlement -v /run/secrets/rhsm:/run/secrets/rhsm
would be a better solution.

[rhatdan]

Sure, You could add these to containers.conf.

[gardar]

But isn't this something that should be handled automatically?
Kind of surprised nobody has brought this issue up before.

[rhatdan]

You could make an argument that podman's containers image should do this automatically, but no way Podman in general should handle this.

[gardar]

Ok while I understand your concerns about Podman's scope, I think there's a case for consistency. If Podman already manages subscriptions for the initial container, logically, it should do so for nested containers too. Otherwise, it might be more consistent to not handle subscriptions at all and leave it entirely to the user.

Coming back to your point:

Perhaps we should modify the mounts.conf file inside of the container to do that by default.

and my concerns about mounting the whole /run/secrets, I realized that it is already being mounted in https://github.com/containers/common/blob/main/pkg/subscriptions/mounts.conf

So perhaps the best course of action would be to modify the mounts.conf as you suggested initially.

[rhatdan]

I am not sure if mounts.conf inside of the container would work correctly rather then just adding it to
volumes=...
In containers.conf

[gardar]

I tried changing mounts.conf inside of the "parent" container to /run/secrets:/run/secrets and I can confirm that it does fix the subscription/repos inside the nested container.
Adding /run/secrets:/run/secrets to volume in containers.conf works too.

[rhatdan]

Ok, lets add /etc/containers/storage.conf with
/run/secrets:/run/secrets. to the stable Containerfile used to build quay.io/podman/stable.