New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker container stuck in Created state on splunk driver failure #24376

Open
joshgarnett opened this Issue Jul 6, 2016 · 3 comments

Comments

Projects
None yet
4 participants
@joshgarnett

joshgarnett commented Jul 6, 2016

This is slightly related to related #21966. When starting a container, if the splunk server returns a 504 on start, the container is actually started, but docker reports the container as failed to start. Once in this state, the container no longer appears under docker ps, and the container process cannot be stopped with docker kill . The only way to stop the process is to manually kill it.

Below is the process and container information (some info has been redacted). 4790c6cb5a5e is the container that was in a bad state.

$ docker ps -a | grep 4790c6cb5a5e
4790c6cb5a5e  some_image  "/sbin/tini -g -- jav"   56 minutes ago  Created  some_name

$ ps aux | grep 4790c6cb5a5e
root     21347  0.0  0.0 200700  3380 ?        Sl   15:17   0:00 docker-containerd-shim 4790c6cb5a5e9b223c5fb2c9576c887c5bbe9206cd6f8763e24e820ca02f6552 /var/run/docker/libcontainerd/4790c6cb5a5e9b223c5fb2c9576c887c5bbe9206cd6f8763e24e820ca02f6552 docker-runc

$ ps aux | grep dev-ci
...+ 21378  0.0  0.0    740     4 ?        Ss   15:17   0:00 /sbin/tini -g -- java ....

$ docker inspect 4790c6cb5a5e
[
    {
        "Id": "4790c6cb5a5e9b223c5fb2c9576c887c5bbe9206cd6f8763e24e820ca02f6552",
        "Created": "2016-07-06T15:17:54.580010448Z",
        "Path": "/sbin/tini",
        "Args": [
            "-g",
            "--",
            "java"
        ],
        "State": {
            "Status": "created",
            "Running": false,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 0,
            "ExitCode": 128,
            "Error": "Failed to initialize logging driver: splunk: failed to verify connection - 504 GATEWAY_TIMEOUT - ",
            "StartedAt": "0001-01-01T00:00:00Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:5ec00ba4f5a3112fa3c2a0db63ae297e9bde33060da450aba749d08e74db75b8",
        "ResolvConfPath": "/data/docker/containers/4790c6cb5a5e9b223c5fb2c9576c887c5bbe9206cd6f8763e24e820ca02f6552/resolv.conf",
        "HostnamePath": "/data/docker/containers/4790c6cb5a5e9b223c5fb2c9576c887c5bbe9206cd6f8763e24e820ca02f6552/hostname",
        "HostsPath": "/data/docker/containers/4790c6cb5a5e9b223c5fb2c9576c887c5bbe9206cd6f8763e24e820ca02f6552/hosts",
        "LogPath": "",
        "Name": "...",
        "RestartCount": 0,
        "Driver": "overlay",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "...",
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "splunk",
                "Config": {
                    "splunk-index": "...",
                    "splunk-insecureskipverify": "true",
                    "splunk-source": "dev",
                    "splunk-sourcetype": "docker",
                    "splunk-token": "...",
                    "splunk-url": "...",
                    "tag": "\"{{.ImageName}}/{{.Name}}/{{.ID}}\""
                }
            },
            "NetworkMode": "default",
            "PortBindings": {
                "5000/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "6000"
                    }
                ]
            },
            "RestartPolicy": {
                "Name": "",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "Dns": null,
            "DnsOptions": null,
            "DnsSearch": null,
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "StorageOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 512,
            "Memory": 1879048192,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": null,
            "DiskQuota": 0,
            "KernelMemory": 0,
            "MemoryReservation": 0,
            "MemorySwap": 3758096384,
            "MemorySwappiness": -1,
            "OomKillDisable": false,
            "PidsLimit": 0,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "BlkioIOps": 0,
            "BlkioBps": 0,
            "SandboxSize": 0
        },
        "GraphDriver": {
            "Name": "overlay",
            "Data": {
                "LowerDir": "/data/docker/overlay/f9a61d8315eecf8a5f1784b34d99eb8f3730792359ced4a53918426637b3c96a/root",
                "MergedDir": "/data/docker/overlay/180c889928b3a3e5427e1a61e4ed4604137be8a5e27630f6ba1994482e2f99a2/merged",
                "UpperDir": "/data/docker/overlay/180c889928b3a3e5427e1a61e4ed4604137be8a5e27630f6ba1994482e2f99a2/upper",
                "WorkDir": "/data/docker/overlay/180c889928b3a3e5427e1a61e4ed4604137be8a5e27630f6ba1994482e2f99a2/work"
            }
        },
        "Mounts": [
            {
                "Source": "...",
                "Destination": "...",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],
        "Config": {
            "Hostname": "4790c6cb5a5e",
            "Domainname": "",
            "User": "...",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "5000/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/lib/jvm/jre/bin",
                "LANG=C.UTF-8",
                "JAVA_HOME=/usr/lib/jvm/jre"
            ],
            "Cmd": [
                "..."
            ],
            "Image": "...",
            "Volumes": {
                "...": {}
            },
            "WorkingDir": "/opt/docker",
            "Entrypoint": [
                "/sbin/tini",
                "-g",
                "--",
                "java"
            ],
            "OnBuild": null
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "0eb10d657b9e86ec6d2087f85a54d1061b11f2067a97781e046121e13d801aae",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": null,
            "SandboxKey": "/var/run/docker/netns/0eb10d657b9e",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "bridge": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "05ec2e2662c979538cb9f7ff8f6cdc1d461749c1a9108d71f6ac7be5d9562ad2",
                    "EndpointID": "",
                    "Gateway": "",
                    "IPAddress": "",
                    "IPPrefixLen": 0,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": ""
                }
            }
        }
    }
]
@joshgarnett

This comment has been minimized.

Show comment
Hide comment
@joshgarnett

joshgarnett Jul 6, 2016

Sorry, I forgot to include my docker info

$ docker info
Containers: 68
 Running: 7
 Paused: 0
 Stopped: 61
Images: 24
Server Version: 1.11.2
Storage Driver: overlay
 Backing Filesystem: extfs
Logging Driver: splunk
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null host bridge
Kernel Version: 4.6.3-1.el7.elrepo.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.858 GiB
Name: ...
ID: ...
Docker Root Dir: /data/docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

joshgarnett commented Jul 6, 2016

Sorry, I forgot to include my docker info

$ docker info
Containers: 68
 Running: 7
 Paused: 0
 Stopped: 61
Images: 24
Server Version: 1.11.2
Storage Driver: overlay
 Backing Filesystem: extfs
Logging Driver: splunk
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null host bridge
Kernel Version: 4.6.3-1.el7.elrepo.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.858 GiB
Name: ...
ID: ...
Docker Root Dir: /data/docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
@sidewinder12s

This comment has been minimized.

Show comment
Hide comment
@sidewinder12s

sidewinder12s Nov 2, 2017

I've also seen this when Splunk returns a 500.

sidewinder12s commented Nov 2, 2017

I've also seen this when Splunk returns a 500.

@sharonx

This comment has been minimized.

Show comment
Hide comment
@sharonx

sharonx Mar 23, 2018

pass in --log-opt splunk-verify-connection=false should get around this issue.

Note that when you see errors like that, it means there is something wrong with your splunk connection.

sharonx commented Mar 23, 2018

pass in --log-opt splunk-verify-connection=false should get around this issue.

Note that when you see errors like that, it means there is something wrong with your splunk connection.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment