Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Hitting panic() in lxc/driver.go:288 #4295

Closed
farcaller opened this Issue Feb 23, 2014 · 10 comments

Comments

Projects
None yet
5 participants
Contributor

farcaller commented Feb 23, 2014

I got a docker with btrfs backend.

Client version: 0.8.0
Go version (client): go1.2
Git commit (client): cc3a8c8-dirty

After stopping docker with a live running container via systemctl stop docker it never comes up again, hitting the panic() in lxc driver.

Context:

info{
  ID: "0e8982087f82bc2017fc0de3e6102718dde24b3d31dc65dab02516fe3b62d21c",
  driver: &driver{
    root: "/var/lib/docker",
    apparmor: false,
    sharedRoot: false,
  }
}

Returned output/error from lxc-info:
0e8982087f82bc2017fc0de3e6102718dde24b3d31dc65dab02516fe3b62d21c doesn't exist, exit code 1.

Config from the corresponding container (/var/lib/docker/containers/0e8982087f82bc2017fc0de3e6102718dde24b3d31dc65dab02516fe3b62d21c/config.json):

{
  "ID": "0e8982087f82bc2017fc0de3e6102718dde24b3d31dc65dab02516fe3b62d21c",
  "Created": "2014-02-23T17:38:07.338616974Z",
  "Path": "/opt/app/server.rb",
  "Args": [
    "-o",
    "0.0.0.0"
  ],
  "Config": {
    "Hostname": "0e8982087f82",
    "Domainname": "",
    "User": "",
    "Memory": 0,
    "MemorySwap": 0,
    "CpuShares": 0,
    "AttachStdin": false,
    "AttachStdout": false,
    "AttachStderr": false,
    "PortSpecs": null,
    "ExposedPorts": {
      "4567/tcp": {}
    },
    "Tty": false,
    "OpenStdin": false,
    "StdinOnce": false,
    "Env": [
      "HOME=/",
      "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
    ],
    "Cmd": [
      "/opt/app/server.rb",
      "-o",
      "0.0.0.0"
    ],
    "Dns": null,
    "Image": "farcaller/influxdb-browser",
    "Volumes": {},
    "VolumesFrom": "",
    "WorkingDir": "",
    "Entrypoint": null,
    "NetworkDisabled": false,
    "OnBuild": null
  },
  "State": {
    "Running": true,
    "Pid": 871,
    "ExitCode": 0,
    "StartedAt": "2014-02-23T17:38:11.261828761Z",
    "FinishedAt": "0001-01-01T00:00:00Z",
    "Ghost": false
  },
  "Image": "3b8ccb21cb4e739edff403b7d5c9fd710fc0ceaa4be9cbd5b2d26d1f3a8cae04",
  "NetworkSettings": {
    "IPAddress": "172.17.0.2",
    "IPPrefixLen": 16,
    "Gateway": "172.17.42.1",
    "Bridge": "docker0",
    "PortMapping": null,
    "Ports": {
      "4567/tcp": [
        {
          "HostIp": "0.0.0.0",
          "HostPort": "4567"
        }
      ]
    }
  },
  "ResolvConfPath": "/etc/resolv.conf",
  "HostnamePath": "/var/lib/docker/containers/0e8982087f82bc2017fc0de3e6102718dde24b3d31dc65dab02516fe3b62d21c/hostname",
  "HostsPath": "/var/lib/docker/containers/0e8982087f82bc2017fc0de3e6102718dde24b3d31dc65dab02516fe3b62d21c/hosts",
  "Name": "/insane_bardeen",
  "Driver": "btrfs",
  "Volumes": {},
  "VolumesRW": {}
}

I'm not sure why docker expects it to be running at this state (I guess docker doesn't tear container down, but systemd kills everything in the docker cgroup). Anyway, it must not panic on docker-lxc state inconsistency.

Contributor

crosbymichael commented Feb 23, 2014

Can you post the panic?

Contributor

farcaller commented Feb 23, 2014

panic: exit status 1

goroutine 1 [running]:
runtime.panic(0x42ddf0, 0x10cd51b0)
    /usr/lib/go/src/pkg/runtime/panic.c:266 +0x134
github.com/dotcloud/docker/execdriver/lxc.(*info).IsRunning(0x10ceb6e0, 0x10c2f7d0)
    /home/farcaller/docker/src/src/github.com/dotcloud/docker/execdriver/lxc/driver.go:289 +0x304
github.com/dotcloud/docker.(*Runtime).Register(0x10c2f1e0, 0x10c59f00, 0x0, 0x0)
    /home/farcaller/docker/src/src/github.com/dotcloud/docker/runtime.go:165 +0x560
github.com/dotcloud/docker.func·049(0x10c59f00)
    /home/farcaller/docker/src/src/github.com/dotcloud/docker/runtime.go:299 +0x38
github.com/dotcloud/docker.(*Runtime).restore(0x10c2f1e0, 0x4128a8, 0x942074)
    /home/farcaller/docker/src/src/github.com/dotcloud/docker/runtime.go:311 +0x9a0
github.com/dotcloud/docker.NewRuntimeFromDirectory(0x10c44ae0, 0x10c8c1b0, 0x1, 0x1, 0x10c44ae0)
    /home/farcaller/docker/src/src/github.com/dotcloud/docker/runtime.go:667 +0xfa4
github.com/dotcloud/docker.NewRuntime(0x10c44ae0, 0x10c8c1b0, 0x13, 0x10c5d32c, 0x0)
    /home/farcaller/docker/src/src/github.com/dotcloud/docker/runtime.go:615 +0x34
github.com/dotcloud/docker.NewServer(0x10c8c1b0, 0x10c44ae0, 0xf, 0x0, 0x0)
    /home/farcaller/docker/src/src/github.com/dotcloud/docker/server.go:2340 +0x34
github.com/dotcloud/docker.jobInitServer(0x10c2f410, 0x10c00df8)
    /home/farcaller/docker/src/src/github.com/dotcloud/docker/server.go:44 +0x98
github.com/dotcloud/docker/engine.(*Job).Run(0x10c2f410, 0x0, 0x0)
    /home/farcaller/docker/src/src/github.com/dotcloud/docker/engine/job.go:66 +0x5a8
main.main()
    /home/farcaller/docker/src/docker/docker/docker.go:55 +0x12a0

I'm seeing this as well (also with the btrfs backend). I'm not sure, but I think it may have started happening after I went from 3.12 to 3.13. I'm on Arch.

Contributor

farcaller commented Feb 24, 2014

FWIW, I've been previously running docker on 3.13.1 for a while, and this one is triggered on 3.13.4. I don't feel that it's somehow related to kernel though.

I had the same issue on Arch Linux. It has to do with the version of lxc. The problem was fixed when I downgraded from lxc-1.0 to lxc-0.9.

Oh, interesting. It's not the only docker/lxc 1.0 interaction regression: #4298.

Collaborator

vieux commented Feb 24, 2014

@farcaller can you post the commit id you are using, your title says line 288 and your panic 289 I want to be sure the exact line.

Contributor

farcaller commented Feb 25, 2014

The correct line is 282 at cc3a8c8, others were added by me to do
logging.

On Mon, Feb 24, 2014 at 11:04 PM, Victor Vieux notifications@github.comwrote:

@farcaller https://github.com/farcaller can you post the commit id you
are using, your title says line 288 and your panic 289 I want to be sure
the exact line.


Reply to this email directly or view it on GitHubhttps://github.com/dotcloud/docker/issues/4295#issuecomment-35951479
.

Sincerely,
Vladimir "Farcaller" Pouzanov
http://farcaller.net/

Collaborator

vieux commented Feb 25, 2014

@farcaller the panic was removed by dotcloud#4013 could you try with docker 0.8.1 ?

Contributor

farcaller commented Feb 25, 2014

Yes, I noticed that I'm not running on the bleeding edge. I guess, this issue is not relevant now.

@farcaller farcaller closed this Feb 25, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment