Hitting panic() in lxc/driver.go:288

[farcaller]

I got a docker with btrfs backend.

Client version: 0.8.0
Go version (client): go1.2
Git commit (client): cc3a8c8-dirty

After stopping docker with a live running container via systemctl stop docker it never comes up again, hitting the panic() in lxc driver.

Context:

info{
  ID: "0e8982087f82bc2017fc0de3e6102718dde24b3d31dc65dab02516fe3b62d21c",
  driver: &driver{
    root: "/var/lib/docker",
    apparmor: false,
    sharedRoot: false,
  }
}

Returned output/error from lxc-info:
0e8982087f82bc2017fc0de3e6102718dde24b3d31dc65dab02516fe3b62d21c doesn't exist, exit code 1.

Config from the corresponding container (/var/lib/docker/containers/0e8982087f82bc2017fc0de3e6102718dde24b3d31dc65dab02516fe3b62d21c/config.json):

{
  "ID": "0e8982087f82bc2017fc0de3e6102718dde24b3d31dc65dab02516fe3b62d21c",
  "Created": "2014-02-23T17:38:07.338616974Z",
  "Path": "/opt/app/server.rb",
  "Args": [
    "-o",
    "0.0.0.0"
  ],
  "Config": {
    "Hostname": "0e8982087f82",
    "Domainname": "",
    "User": "",
    "Memory": 0,
    "MemorySwap": 0,
    "CpuShares": 0,
    "AttachStdin": false,
    "AttachStdout": false,
    "AttachStderr": false,
    "PortSpecs": null,
    "ExposedPorts": {
      "4567/tcp": {}
    },
    "Tty": false,
    "OpenStdin": false,
    "StdinOnce": false,
    "Env": [
      "HOME=/",
      "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
    ],
    "Cmd": [
      "/opt/app/server.rb",
      "-o",
      "0.0.0.0"
    ],
    "Dns": null,
    "Image": "farcaller/influxdb-browser",
    "Volumes": {},
    "VolumesFrom": "",
    "WorkingDir": "",
    "Entrypoint": null,
    "NetworkDisabled": false,
    "OnBuild": null
  },
  "State": {
    "Running": true,
    "Pid": 871,
    "ExitCode": 0,
    "StartedAt": "2014-02-23T17:38:11.261828761Z",
    "FinishedAt": "0001-01-01T00:00:00Z",
    "Ghost": false
  },
  "Image": "3b8ccb21cb4e739edff403b7d5c9fd710fc0ceaa4be9cbd5b2d26d1f3a8cae04",
  "NetworkSettings": {
    "IPAddress": "172.17.0.2",
    "IPPrefixLen": 16,
    "Gateway": "172.17.42.1",
    "Bridge": "docker0",
    "PortMapping": null,
    "Ports": {
      "4567/tcp": [
        {
          "HostIp": "0.0.0.0",
          "HostPort": "4567"
        }
      ]
    }
  },
  "ResolvConfPath": "/etc/resolv.conf",
  "HostnamePath": "/var/lib/docker/containers/0e8982087f82bc2017fc0de3e6102718dde24b3d31dc65dab02516fe3b62d21c/hostname",
  "HostsPath": "/var/lib/docker/containers/0e8982087f82bc2017fc0de3e6102718dde24b3d31dc65dab02516fe3b62d21c/hosts",
  "Name": "/insane_bardeen",
  "Driver": "btrfs",
  "Volumes": {},
  "VolumesRW": {}
}

I'm not sure why docker expects it to be running at this state (I guess docker doesn't tear container down, but systemd kills everything in the docker cgroup). Anyway, it must not panic on docker-lxc state inconsistency.

[crosbymichael]

Can you post the panic?

[farcaller]

panic: exit status 1

goroutine 1 [running]:
runtime.panic(0x42ddf0, 0x10cd51b0)
    /usr/lib/go/src/pkg/runtime/panic.c:266 +0x134
github.com/dotcloud/docker/execdriver/lxc.(*info).IsRunning(0x10ceb6e0, 0x10c2f7d0)
    /home/farcaller/docker/src/src/github.com/dotcloud/docker/execdriver/lxc/driver.go:289 +0x304
github.com/dotcloud/docker.(*Runtime).Register(0x10c2f1e0, 0x10c59f00, 0x0, 0x0)
    /home/farcaller/docker/src/src/github.com/dotcloud/docker/runtime.go:165 +0x560
github.com/dotcloud/docker.func·049(0x10c59f00)
    /home/farcaller/docker/src/src/github.com/dotcloud/docker/runtime.go:299 +0x38
github.com/dotcloud/docker.(*Runtime).restore(0x10c2f1e0, 0x4128a8, 0x942074)
    /home/farcaller/docker/src/src/github.com/dotcloud/docker/runtime.go:311 +0x9a0
github.com/dotcloud/docker.NewRuntimeFromDirectory(0x10c44ae0, 0x10c8c1b0, 0x1, 0x1, 0x10c44ae0)
    /home/farcaller/docker/src/src/github.com/dotcloud/docker/runtime.go:667 +0xfa4
github.com/dotcloud/docker.NewRuntime(0x10c44ae0, 0x10c8c1b0, 0x13, 0x10c5d32c, 0x0)
    /home/farcaller/docker/src/src/github.com/dotcloud/docker/runtime.go:615 +0x34
github.com/dotcloud/docker.NewServer(0x10c8c1b0, 0x10c44ae0, 0xf, 0x0, 0x0)
    /home/farcaller/docker/src/src/github.com/dotcloud/docker/server.go:2340 +0x34
github.com/dotcloud/docker.jobInitServer(0x10c2f410, 0x10c00df8)
    /home/farcaller/docker/src/src/github.com/dotcloud/docker/server.go:44 +0x98
github.com/dotcloud/docker/engine.(*Job).Run(0x10c2f410, 0x0, 0x0)
    /home/farcaller/docker/src/src/github.com/dotcloud/docker/engine/job.go:66 +0x5a8
main.main()
    /home/farcaller/docker/src/docker/docker/docker.go:55 +0x12a0

[orospakr]

I'm seeing this as well (also with the btrfs backend). I'm not sure, but I think it may have started happening after I went from 3.12 to 3.13. I'm on Arch.

[farcaller]

FWIW, I've been previously running docker on 3.13.1 for a while, and this one is triggered on 3.13.4. I don't feel that it's somehow related to kernel though.

[petrosagg]

I had the same issue on Arch Linux. It has to do with the version of lxc. The problem was fixed when I downgraded from lxc-1.0 to lxc-0.9.

[orospakr]

Oh, interesting. It's not the only docker/lxc 1.0 interaction regression: #4298.

[vieux]

@farcaller can you post the commit id you are using, your title says line 288 and your panic 289 I want to be sure the exact line.

[farcaller]

The correct line is 282 at cc3a8c8, others were added by me to do
logging.

On Mon, Feb 24, 2014 at 11:04 PM, Victor Vieux notifications@github.comwrote:

@farcaller https://github.com/farcaller can you post the commit id you
are using, your title says line 288 and your panic 289 I want to be sure
the exact line.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/4295#issuecomment-35951479
.

Sincerely,
Vladimir "Farcaller" Pouzanov
http://farcaller.net/

[vieux]

@farcaller the panic was removed by #4013 could you try with docker 0.8.1 ?

[farcaller]

Yes, I noticed that I'm not running on the bleeding edge. I guess, this issue is not relevant now.