Process running within Docker Container throws OOM error but container doesn't report the error when exited

[mcregan23]

Description

I noticed an issue where a process running within a docker container on a Windows VM reported an Out of Memory error according to the logs emitted by docker, however the container itself reported as not exiting due to an OOM error (OOMKilled flag was false when running docker inspect <containerid>) and the exit code reported from the container never matched the expected exit code for an OOM error (Exit code 137). When trying to see if linux had the same behavior when it comes to the process running inside of the container reporting an OOM error, the docker container reported that it was killed due to OOM (OOMKilled flag was true when running docker inspect <containerid> and container reported exit code of 137). I always saw the same behavior in regards to the container reporting that itself wasn't killed due to OOM while the process was reporting OOM errors.

Reproduce

Windows Reproduction Steps:

1. Created simple DockerFile

FROM mcr.microsoft.com/windows/servercore:ltsc2019

COPY test.ps1 .

ENTRYPOINT ["powershell.exe"]

CMD [".\\test.ps1"]

test.ps1 contents

function Consume-Memory {
    $mem = @()

    while ($true) {
        # Allocate 10 MB of memory
        $data = New-Object byte[] (100*1024*1024)
        if (-not $data) {
            throw "Failed to allocate memory. Exiting..."
        }
        $mem += $data
        Write-Host "Allocated chunks: $($mem.Count)"
        Start-Sleep -Seconds 10  # Adjust sleep duration as necessary
    }
}

function Main {
    Consume-Memory
}

Main

2. Built an image using the above file docker build -t <yourtaghere> .
3. Ran a docker container using docker run -m 256 <tag here>
4. Monitored the container memory usage using docker stats
5. When container would eventually exit, I inspected the container using docker inspect <containeridhere>
6. Verified the container did not set the OOMKilled flag as True when the associated docker logs for the container through the path given in the above docker inspect command and process running within the container reported stopping due to an out of memory error.

Linux Reproduction Steps:

These steps are relatively short and simpler than the windows steps.

1. Created simple Dockerfile

FROM public.ecr.aws/amazonlinux/amazonlinux:latest

CMD ["python3", "-c", "foo=' '*1024*1024*512; import time; time.sleep(10)"]

2. Built the image and initiated a container from the image with a limit of 512mb for the container. I used docker build -t <yourtaghere> and docker run -m 512 <tag here>
3. The container would exit almost immediately, emitting exit code 137 which represents an OOM error and the process alongside reported an OutOfMemoryException.

Expected behavior

Docker container running on Windows when encountered with an OutOfMemory error from the process running within the container should report itself as being killed due to OOM error which is the behavior on linux.

docker version

Windows Client:
Client:
 Version:           22.06.0-beta.0-189-g4e05b45a90.m
 API version:       1.41 (downgraded from 1.42)
 Go version:        go1.19.2
 Git commit:        4e05b45a90
 Built:             Sat Oct 29 21:48:57 2022
 OS/Arch:           windows/amd64
 Context:           default

Server:
 Engine:
  Version:          20.10.23
  API version:      1.41 (minimum version 1.24)
  Go version:       go1.18.10
  Git commit:       6051f14291
  Built:            Wed Oct 25 18:58:05 2023
  OS/Arch:          windows/amd64
  Experimental:     false

Linux Version:
Client:
 Version:           20.10.25
 API version:       1.41
 Go version:        go1.19.9
 Git commit:        b82b9f3
 Built:             Wed Jul 12 19:37:13 2023
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.25
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.19.9
  Git commit:       5df983c
  Built:            Wed Jul  5 00:00:00 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.11
  GitCommit:        64b8a811b07ba6288238eefc14d898ee0b5b99ba
 runc:
  Version:          1.1.11
  GitCommit:        4bccb38cc9cf198d52bebf2b3a90cd14e7af8c06
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Linux Info:
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., 0.0.0+unknown)

Server:
 Containers: 9
  Running: 1
  Paused: 0
  Stopped: 8
 Images: 11
 Server Version: 20.10.25
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 64b8a811b07ba6288238eefc14d898ee0b5b99ba
 runc version: 4bccb38cc9cf198d52bebf2b3a90cd14e7af8c06
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 6.1.79-99.164.amzn2023.x86_64
 Operating System: Amazon Linux 2023.3.20240312
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 7.493GiB
 Name: ip-172-31-40-192.us-west-2.compute.internal
 ID: 4YDE:PK2F:GCNR:2KXX:4QIW:GJZP:AHPB:7P4V:5KKU:GPJO:IC4Y:UFT4
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Windows Info:
Client:
 Context:    default
 Debug Mode: false

Server:
 Containers: 23
  Running: 0
  Paused: 0
  Stopped: 23
 Images: 28
 Server Version: 20.10.23
 Storage Driver: windowsfilter
  Windows:
 Logging Driver: json-file
 Plugins:
  Volume: local
  Network: ics internal l2bridge l2tunnel nat null overlay private transparent
  Log: awslogs etwlogs fluentd gcplogs gelf json-file local logentries splunk syslog
 Swarm: inactive
 Default Isolation: process
 Kernel Version: 10.0 17763 (17763.1.amd64fre.rs5_release.180914-1434)
 Operating System: Windows Server 2019 Datacenter Version 1809 (OS Build 17763.5458)
 OSType: windows
 Architecture: x86_64
 CPUs: 4
 Total Memory: 7.707GiB
 Name: EC2AMAZ-8CRB3OK
 ID: JZVV:HI7E:KKII:ULNM:VNL7:KL22:QPAG:UKU5:Z4KF:IDJC:3IDC:HRU2
 Docker Root Dir: C:\ProgramData\docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional Info

Process logs from Windows:

{"log":"Allocated chunks: 104857600\n","stream":"stdout","time":"2024-03-22T15:51:37.3341515Z"}
{"log":"Array dimensions exceeded supported range.\r\n","stream":"stderr","time":"2024-03-22T15:52:04.7072964Z"}
{"log":"At C:\\test.ps1:11 char:9\r\n","stream":"stderr","time":"2024-03-22T15:52:04.7072964Z"}
{"log":"+         $mem += $data\r\n","stream":"stderr","time":"2024-03-22T15:52:04.7072964Z"}
{"log":"+         ~~~~~~~~~~~~~\r\n","stream":"stderr","time":"2024-03-22T15:52:04.7072964Z"}
{"log":"    + CategoryInfo          : OperationStopped: (:) [], OutOfMemoryException\r\n","stream":"stderr","time":"2024-03-22T15:52:04.7085132Z"}
{"log":"    + FullyQualifiedErrorId : System.OutOfMemoryException\r\n","stream":"stderr","time":"2024-03-22T15:52:04.7085132Z"}
{"log":" \r\n","stream":"stderr","time":"2024-03-22T15:52:04.7085132Z"}

docker inspect logs:

docker inspect exciting_rubin
[
    {
        "Id": "6fe653aeccb41201d50629356f057df650195f63bb16d0070fc31eefeb93a29b",
        "Created": "2024-03-22T15:50:54.4760729Z",
        "Path": "powershell.exe",
        "Args": [
            ".\\test.ps1"
        ],
        "State": {
            "Status": "exited",
            "Running": false,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false, `should be true`
            "Dead": false,
            "Pid": 0,
            "ExitCode": 3221225473,
            "Error": "",
            "StartedAt": "2024-03-22T15:50:56.2195812Z",
            "FinishedAt": "2024-03-22T16:03:36.7360532Z"
        },

[mcregan23]

Some added context in the form of a response when I submitted this issue to Moby: moby/moby#47618 (comment)