[BUG] Inside docker container, virtual detection is physical

[wica128]

Hi,
Description

I'm using docker containers, to develop salt states for our environment. Where I use salt to manage virtual and physical servers.
And I use grains:virtual to decide, what to run.

But in the running docker container, grains:virtual returns physical. salt-call --local grains.item virtual

A solution would be to add docker to https://github.com/saltstack/salt/blob/master/salt/grains/core.py#L841
Because systemd-detect-virt returns docker.

Setup
Docker container oraclelinux:8 with systemd enabled.

Steps to Reproduce the behavior
Install salt in the container and run salt-call --local grains.item virtual

local:
    ----------
    virtual:
        physical

Expected behavior
That grains:virtual returns docker when running salt-call --local grains.item virtual

Versions Report

[root@salt salt]# salt --versions-report
Salt Version:
          Salt: 3004.2

Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: 2.6.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.0.3
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.4
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: Not Installed
  pycryptodome: 3.15.0
        pygit2: Not Installed
        Python: 3.6.8 (default, Apr 11 2022, 08:20:28)
  python-gnupg: Not Installed
        PyYAML: 6.0
         PyZMQ: 21.0.2
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.3

System Versions:
          dist: oracle 8.6
        locale: UTF-8
       machine: aarch64
       release: 5.18.16-200.fc36.aarch64
        system: Linux
       version: Oracle Linux Server 8.6

[welcome]

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey.
Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

• Community Wiki
• Salt’s Contributor Guide
• Join our Community Slack
• IRC on LiberaChat
• Salt Project YouTube channel
• Salt Project Twitch channel

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar.
If you have additional questions, email us at saltproject@vmware.com. We’re glad you’ve joined our community and look forward to doing awesome things with you!

[lukasraska]

@OrangeDog Salt always provides container when inside Docker container

[root@78b3076435f1 salt-pkg]# salt-call --local grains.item virtual
local:
    ----------
    virtual:
        container

then more specifically Docker inside virtual_subtype grain

[root@78b3076435f1 salt-pkg]# salt-call --local grains.item virtual_subtype
local:
    ----------
    virtual_subtype:
        Docker

This however depends on systemd-detect-virt / virt-what (for LXC containers) or then on /proc/1/cgroup containing docker:

        if os.path.isfile("/proc/1/cgroup"):
            try:
                with salt.utils.files.fopen("/proc/1/cgroup", "r") as fhr:
                    fhr_contents = fhr.read()
                if ":/lxc/" in fhr_contents:
                    grains["virtual"] = "container"
                    grains["virtual_subtype"] = "LXC"
                elif ":/kubepods/" in fhr_contents:
                    grains["virtual_subtype"] = "kubernetes"
                elif ":/libpod_parent/" in fhr_contents:
                    grains["virtual_subtype"] = "libpod"
                else:
                    if any(
                        x in fhr_contents
                        for x in (":/system.slice/docker", ":/docker/", ":/docker-ce/")
                    ):  
                        grains["virtual"] = "container"
                        grains["virtual_subtype"] = "Docker"
            except OSError:
                pass

which it does in my container (CentOS, Ubuntu as a host)

[root@78b3076435f1 salt]# cat /proc/1/cgroup | grep -i docker
11:devices:/docker/78b3076435f10bf6b5130d3863be2e8ae64c5c2d98230a847499dc680dd3bad7
10:pids:/docker/78b3076435f10bf6b5130d3863be2e8ae64c5c2d98230a847499dc680dd3bad7
9:cpuacct,cpu:/docker/78b3076435f10bf6b5130d3863be2e8ae64c5c2d98230a847499dc680dd3bad7
8:hugetlb:/docker/78b3076435f10bf6b5130d3863be2e8ae64c5c2d98230a847499dc680dd3bad7
7:blkio:/docker/78b3076435f10bf6b5130d3863be2e8ae64c5c2d98230a847499dc680dd3bad7
6:cpuset:/docker/78b3076435f10bf6b5130d3863be2e8ae64c5c2d98230a847499dc680dd3bad7
5:freezer:/docker/78b3076435f10bf6b5130d3863be2e8ae64c5c2d98230a847499dc680dd3bad7
4:memory:/docker/78b3076435f10bf6b5130d3863be2e8ae64c5c2d98230a847499dc680dd3bad7
3:perf_event:/docker/78b3076435f10bf6b5130d3863be2e8ae64c5c2d98230a847499dc680dd3bad7
2:net_prio,net_cls:/docker/78b3076435f10bf6b5130d3863be2e8ae64c5c2d98230a847499dc680dd3bad7
1:name=systemd:/docker/78b3076435f10bf6b5130d3863be2e8ae64c5c2d98230a847499dc680dd3bad7

So I don't think this is "not a bug", but I don't think the bug is actually something caused by Salt. When I run any Docker container, it always contains cgroups with docker prefix/path.

When i run the specific oraclelinux:8, it contains docker in cgroups:

[root@bd6072c5d418 tmp]# cat /proc/1/cgroup   
13:pids:/docker/bd6072c5d418cad2677a1df053a3e92db82d0b226fc2352030c5eb5b74876976
12:hugetlb:/docker/bd6072c5d418cad2677a1df053a3e92db82d0b226fc2352030c5eb5b74876976
11:perf_event:/docker/bd6072c5d418cad2677a1df053a3e92db82d0b226fc2352030c5eb5b74876976
10:misc:/
9:blkio:/docker/bd6072c5d418cad2677a1df053a3e92db82d0b226fc2352030c5eb5b74876976
8:rdma:/
7:freezer:/docker/bd6072c5d418cad2677a1df053a3e92db82d0b226fc2352030c5eb5b74876976
6:cpuset:/docker/bd6072c5d418cad2677a1df053a3e92db82d0b226fc2352030c5eb5b74876976
5:devices:/docker/bd6072c5d418cad2677a1df053a3e92db82d0b226fc2352030c5eb5b74876976
4:cpu,cpuacct:/docker/bd6072c5d418cad2677a1df053a3e92db82d0b226fc2352030c5eb5b74876976
3:memory:/docker/bd6072c5d418cad2677a1df053a3e92db82d0b226fc2352030c5eb5b74876976
2:net_cls,net_prio:/docker/bd6072c5d418cad2677a1df053a3e92db82d0b226fc2352030c5eb5b74876976
1:name=systemd:/docker/bd6072c5d418cad2677a1df053a3e92db82d0b226fc2352030c5eb5b74876976
0::/system.slice/containerd.service

Although I don't run systemd there, so that might be the case.

@wica128 are you able to provide Dockerfile that reproduces this exact behaviour?

[kennybytes]

interesting, just running the debian:latest image gives me a different result; i'm using debian bullseye as my host:

$ docker run -it debian:latest  cat /proc/1/cgroup
0::/

[lukasraska]

Hmm, that's weird. This is on my laptop (Ubuntu 20.04)

$ docker run -it debian:latest  cat /proc/1/cgroup
13:perf_event:/docker/0ab1bdaaddebc2eca383e2c51966914b2e198fd44f9b15166dd731ef335420d1
12:freezer:/docker/0ab1bdaaddebc2eca383e2c51966914b2e198fd44f9b15166dd731ef335420d1
11:devices:/docker/0ab1bdaaddebc2eca383e2c51966914b2e198fd44f9b15166dd731ef335420d1
10:rdma:/
9:net_cls,net_prio:/docker/0ab1bdaaddebc2eca383e2c51966914b2e198fd44f9b15166dd731ef335420d1
8:cpuset:/docker/0ab1bdaaddebc2eca383e2c51966914b2e198fd44f9b15166dd731ef335420d1
7:blkio:/docker/0ab1bdaaddebc2eca383e2c51966914b2e198fd44f9b15166dd731ef335420d1
6:memory:/docker/0ab1bdaaddebc2eca383e2c51966914b2e198fd44f9b15166dd731ef335420d1
5:cpu,cpuacct:/docker/0ab1bdaaddebc2eca383e2c51966914b2e198fd44f9b15166dd731ef335420d1
4:misc:/
3:pids:/docker/0ab1bdaaddebc2eca383e2c51966914b2e198fd44f9b15166dd731ef335420d1
2:hugetlb:/docker/0ab1bdaaddebc2eca383e2c51966914b2e198fd44f9b15166dd731ef335420d1
1:name=systemd:/docker/0ab1bdaaddebc2eca383e2c51966914b2e198fd44f9b15166dd731ef335420d1
0::/system.slice/containerd.service

with

$ docker --version
Docker version 19.03.13, build 4484c46d9d

What kind of host OS it is and what is the Docker version?

[kennybytes]

@lukasraska

here's the host system:

$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

docker version:

$ docker --version
Docker version 20.10.5+dfsg1, build 55c4c88

was also able to repro on a macbook pro m1 with the following docker version:

 % docker --version
Docker version 20.10.17, build 100c701

[MartinMystikJonas]

@wica128 @kennybytes Did any of you find solution?

I just stubled upon the same bug. I run salt inside docker (build) inside VirtualBox 7.0 on Windows 11 but salt thinks it is in VirtualBox and not inside Docker.

    uid:
        0
    username:
        root
    uuid:
        a7135285-9416-a84b-a465-ecc8f50f1663
    virtual:
        VirtualBox
    zfs_feature_flags:
        False
    zfs_support:
        False
    zmqversion:
        4.3.1

Symptoms are exactly the same. No docker entry is in cgroups. Only:

0::/

VirtulBox OS:

PRETTY_NAME="Ubuntu 22.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.1 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

Docker version

Docker version 20.10.23, build 7155243

[kennybytes]

No, not yet unfortunately

[MartinMystikJonas]

Workaround to detect docker by interface names I used:

{% if salt['grains.get']('virtual_subtype', '') == "Docker" or salt['grains.get']('ip_interfaces:docker0', false) != false %}

[MartinMystikJonas]

This is probably the cause and it also suggest possible fixes:

• https://stackoverflow.com/questions/69002675/on-debian-11-bullseye-proc-self-cgroup-inside-a-docker-container-does-not-sho
• https://stackoverflow.com/questions/68816329/how-to-get-docker-container-id-from-within-the-container-with-cgroup-v2

[waltercodes]

I'm running Docker Desktop 4.28.0 (139021) on MacOS Sonoma v14.4.1. The container in question is running Rocky Linux 8 with systemd enabled and my Docker Desktop is running the default cgroups configuration, which I believe uses cgroups v2. We are seeing the behavior with the grain returning physical from within the container.

Both systemd-detect-virt and virt-what output docker within the Rocky Linux 8 container, so allowing for those values seems to be a reasonable way to correct the grain's return value. Our current workaround is to use the environment variable that we populate at build time, but some of our flows really would benefit from receiving proper output from the grain because they address all sorts of virtualization within a single state (not just docker vs physical). Here is an example of a patch against master as I found it this morning that would do what I'm envisioning:

core.py.patch