Docker stop hangs forever on sql image on Windows VM host

[shueybubbles]

using the image mcr.microsoft.com/mssql/server:2017-latest on Windows Server Version 10.0.18362.239,
Docker stop on the container instance hangs forever, other commands like docker exec and docker container top report "no such container" even though "docker logs" and "docker ps" seem to work fine.

Steps to reproduce the issue:

docker run -e "ACCEPT_EULA=1" -e "SA_PASSWORD=somepassword" -P -d --rm mcr.microsoft.com/mssql/server:2017-latest

Do some work with the sql instance. After a few days it will stop responding, it appears it decides to stop for reasons I can't discern. docker logs shows this:

2019-07-26 15:47:32.33 spid6s Always On: The availability replica manager is going offline because SQL Server is shutting down. This is an informational message only. No user action is required. 2019-07-26 15:47:32.36 spid6s SQL Server is terminating in response to a 'stop' request from Service Control Manager. This is an informational message only. No user action is required. 2019-07-26 15:47:33.38 spid23s Service Broker manager has shut down. 2019-07-26 15:47:34.54 spid6s .NET Framework runtime has been stopped. 2019-07-26 15:47:35.52 spid6s SQL Trace was stopped due to server shutdown. Trace ID = '1'. This is an informational message only; no user action is required.

C:\>docker container ls
CONTAINER ID        IMAGE                                                     COMMAND                  CREATED             STATUS              PORTS                     NAMES
9d71c434fe24        mcr.microsoft.com/mssql/server:2017-latest                "/opt/mssql/bin/sqls…"   2 days ago          Up 2 days           0.0.0.0:58819->1433/tcp   keep_sql2017test

C:\>docker container exec 9d71c434fe24 ls /
no such container

Output of docker version:

Client:
 Version:           master-dockerproject-2019-07-11
 API version:       1.40
 Go version:        go1.12.6
 Git commit:        6fc0dc91
 Built:             Fri Jul 12 00:07:05 2019
 OS/Arch:           windows/amd64
 Experimental:      false

Server:
 Engine:
  Version:          master-dockerproject-2019-07-11
  API version:      1.41 (minimum version 1.24)
  Go version:       go1.12.7
  Git commit:       2fc3480
  Built:            Fri Jul 12 00:19:30 2019
  OS/Arch:          windows/amd64
  Experimental:     true

Output of docker info:

Client:
 Debug Mode: false

Server:
 Containers: 4
  Running: 2
  Paused: 0
  Stopped: 2
 Images: 8
 Server Version: master-dockerproject-2019-07-11
 Storage Driver: windowsfilter (windows) lcow (linux)
  Windows:
  LCOW:
 Logging Driver: json-file
 Plugins:
  Volume: local
  Network: ics internal l2bridge l2tunnel nat null overlay private transparent
  Log: awslogs etwlogs fluentd gcplogs gelf json-file local logentries splunk syslog
 Swarm: inactive
 Default Isolation: process
 Kernel Version: 10.0 18362 (18362.1.amd64fre.19h1_release.190318-1202)
 Operating System: Windows Server Datacenter Version 1903 (OS Build 18362.239)
 OSType: windows
 Architecture: x86_64
 CPUs: 4
 Total Memory: 16GiB
 Name: <redacted>
 ID: 2P5Y:UUJ7:SGQZ:PH2D:KENL:567I:KDNZ:2CU2:DDHJ:VOWQ:UINZ:EWAS
 Docker Root Dir: C:\ProgramData\docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

This is an Azure VM

docker inspect shows
[
    {
        "Id": "9d71c434fe246a19aaa6e0de8f268cb5ef1fa3c8884f25702d4f7e78757570c2",
        "Created": "2019-07-26T14:40:07.5640464Z",
        "Path": "/opt/mssql/bin/sqlservr",
        "Args": [],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 363,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2019-07-26T14:40:11.9961699Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:a3e890278b9531355e6fffb44ec300dde1674f2978532820400fb1b6fe27a87e",
        "ResolvConfPath": "",
        "HostnamePath": "",
        "HostsPath": "",
        "LogPath": "C:\\ProgramData\\docker\\containers\\9d71c434fe246a19aaa6e0de8f268cb5ef1fa3c8884f25702d4f7e78757570c2\\9d71c434fe246a19aaa6e0de8f268cb5ef1fa3c8884f25702d4f7e78757570c2-json.log",
        "Name": "/keep_sql2017test",
        "RestartCount": 0,
        "Driver": "lcow",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": null,
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "default",
            "PortBindings": {
                "1433/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": ""
                    }
                ]
            },
            "RestartPolicy": {
                "Name": "no",
                "MaximumRetryCount": 0
            },
            "AutoRemove": true,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "Capabilities": null,
            "CgroupnsMode": "host",
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 0,
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "hyperv",
            "CpuShares": 0,
            "Memory": 3221225472,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": [],
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "KernelMemory": 0,
            "KernelMemoryTCP": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "MaskedPaths": null,
            "ReadonlyPaths": null
        },
        "GraphDriver": {
            "Data": {
                "dir": "C:\\ProgramData\\docker\\lcow\\9d71c434fe246a19aaa6e0de8f268cb5ef1fa3c8884f25702d4f7e78757570c2"
            },
            "Name": "lcow"
        },
        "Mounts": [],
        "Config": {
            "Hostname": "9d71c434fe24",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "1433/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "MSSQL_PID=Developer",
                "ACCEPT_EULA=Y",
                "SA_PASSWORD=MyFancyPassword01",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Cmd": [
                "/opt/mssql/bin/sqlservr"
            ],
            "Image": "mcr.microsoft.com/mssql/server:2017-latest",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": {
                "com.microsoft.product": "Microsoft SQL Server",
                "com.microsoft.version": "14.0.3192.2",
                "vendor": "Microsoft"
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "9d71c434fe246a19aaa6e0de8f268cb5ef1fa3c8884f25702d4f7e78757570c2",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {
                "1433/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "58819"
                    }
                ]
            },
            "SandboxKey": "9d71c434fe246a19aaa6e0de8f268cb5ef1fa3c8884f25702d4f7e78757570c2",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "nat": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "961e8c40d9ba72bd3d542d0018b0ae302f8c6c7ee5edb1a8b0cb221dfc27ac5f",
                    "EndpointID": "904318315b7687e5853c8aef1e96112dbb5b7c800608333c15dac45a0a06aeb2",
                    "Gateway": "172.31.32.1",
                    "IPAddress": "172.31.41.218",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "00:15:5d:cc:ad:76",
                    "DriverOpts": null
                }
            }
        }
    }
]

[starcraft66]

I'm experiencing the exact same issue with another windows container.

[dmitryRomanchenko]

i am having the same problem

[Iristyle]

I presume it shows up with hcsdiag list and you can kill the container with hcsdiag kill <id>?

[dmitryRomanchenko]

nope. hcsdiag shows nothing. i guess it might be related to #39864

[spelroth]

Hey I Have the same problem running on Windows server 2019 standard, as for now it happens when server is shut down unclean (like power loss or hard reset).
The dockers are in state running but in real the're not ;)
last time I've removed the C:/ProgramData/docker/containers/id_of_hanged_container and recreated them. But after today when the server failed I've got dozens of them...
Luckly I've found a workaround
I've found that the docker config.v2.json has state running set to true., so after stopping docker engine in services,msc and modifiing the state from true to false. I'm able to start the containers and they're back to working state. 👍