start docker failed, error is panic: invalid freelist page: 48, page type is leaf

[gosoon]

Description
start docker failed，the error log is:

Mar 01 12:54:59 localhost systemd[1]: Starting Docker Application Container Engine...
Mar 01 12:55:03 localhost dockerd[4230]: time="2021-03-01T12:55:03.630985606+08:00" level=warning msg="The \"-g / --graph\" flag is deprecated. Please use \"--data-root\" instead"
Mar 01 12:55:54 localhost dockerd[4230]: time="2021-03-01T12:55:54.856750579+08:00" level=warning msg="Your kernel does not support cgroup blkio weight"
Mar 01 12:55:54 localhost dockerd[4230]: time="2021-03-01T12:55:54.856793736+08:00" level=warning msg="Your kernel does not support cgroup blkio weight_device"
Mar 01 12:55:55 localhost dockerd[4230]: time="2021-03-01T12:55:55.097560330+08:00" level=error msg="Failed to load container 332bfd2303f1ee0dfd0cb3cb925e7b19dc5e40cf9405e9c63ef78b8e761cfce3: open /media/disk1/docker/containers/332bfd2303f1ee0dfd0cb3cb925e7b19dc5e40cf9405e9c63ef78b8e761cfce3/config.v2.json: no such file or directory"
Mar 01 12:55:55 localhost dockerd[4230]: time="2021-03-01T12:55:55.546134668+08:00" level=error msg="Failed to load container c424a4dccd49e1437ba37a640ac9b1390f74fee81b639a71f2e91a139177057f: json: cannot unmarshal number into Go value of type container.Container"
Mar 01 12:55:55 localhost dockerd[4230]: time="2021-03-01T12:55:55.546371935+08:00" level=error msg="Failed to load container aaa08c96a1911aed9f1e0a2e54efb358849ab0b5f5c29cbe0f158e3e1547ffd8: invalid character 'l' looking for beginning of value"
Mar 01 12:55:55 localhost dockerd[4230]: time="2021-03-01T12:55:55.111425237+08:00" level=error msg="Failed to load container 633e7416f211156752f61dc459bede9eec55bd4d996e376141e7713a9d8b152f: open /media/disk1/docker/containers/633e7416f211156752f61dc459bede9eec55bd4d996e376141e7713a9d8b152f/config.v2.json: no such file or directory"
Mar 01 12:55:55 localhost dockerd[4230]: time="2021-03-01T12:55:55.560490512+08:00" level=error msg="Failed to load container b224983ba9f2ebf2d0936b4abc006639d9d8675971fdd0d7bc962f54b25d341c: json: cannot unmarshal number into Go value of type container.Container"
Mar 01 12:55:55 localhost dockerd[4230]: time="2021-03-01T12:55:55.564080280+08:00" level=error msg="Failed to load container efdb793c18f83af85fd4b95e30469542d9fd9aa1f0a905108c31a2fb05bd6e60: json: cannot unmarshal number into Go value of type container.Container"
Mar 01 12:55:55 localhost dockerd[4230]: time="2021-03-01T12:55:55.566406244+08:00" level=error msg="Failed to load container b10500a3fcbfeb60fd33e5e62d04030f77b06ad5a5df47e0fb0d8944893cf31b: unexpected EOF"
Mar 01 12:55:58 localhost dockerd[4230]: time="2021-03-01T12:55:58.287237972+08:00" level=error msg="9134a5cad5d0c25638ff9ccebd5507f706c0183fedfcb8a6343e786ee9ae8c69 cleanup: failed to delete container from containerd: no such container"
Mar 01 12:55:58 localhost dockerd[4230]: time="2021-03-01T12:55:58.569738561+08:00" level=error msg="092bea231813038b94f26264f346f0e2115e945ab806dd060985cc67c52d5de9 cleanup: failed to delete container from containerd: no such container"
Mar 01 12:56:01 localhost dockerd[4230]: panic: invalid freelist page: 48, page type is leaf
Mar 01 12:56:01 localhost dockerd[4230]: goroutine 1 [running]:
Mar 01 12:56:01 localhost dockerd[4230]: github.com/docker/docker/vendor/go.etcd.io/bbolt.(*freelist).read(0xc000823800, 0x7f005c329000)
Mar 01 12:56:01 localhost dockerd[4230]: /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/go.etcd.io/bbolt/freelist.go:266 +0x339
Mar 01 12:56:01 localhost dockerd[4230]: github.com/docker/docker/vendor/go.etcd.io/bbolt.(*DB).loadFreelist.func1()
Mar 01 12:56:01 localhost dockerd[4230]: /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/go.etcd.io/bbolt/db.go:316 +0xd1
Mar 01 12:56:01 localhost dockerd[4230]: sync.(*Once).doSlow(0xc00017c568, 0xc0010d42b8)
Mar 01 12:56:01 localhost dockerd[4230]: /usr/local/go/src/sync/once.go:66 +0xe5
Mar 01 12:56:01 localhost dockerd[4230]: sync.(*Once).Do(...)
Mar 01 12:56:01 localhost dockerd[4230]: /usr/local/go/src/sync/once.go:57
Mar 01 12:56:01 localhost dockerd[4230]: github.com/docker/docker/vendor/go.etcd.io/bbolt.(*DB).loadFreelist(0xc00017c400)
Mar 01 12:56:01 localhost dockerd[4230]: /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/go.etcd.io/bbolt/db.go:309 +0x6c
Mar 01 12:56:01 localhost dockerd[4230]: github.com/docker/docker/vendor/go.etcd.io/bbolt.Open(0xc001902a20, 0x2d, 0x1a4, 0xc0010d43f8, 0xc0017e7560, 0x203000, 0x203000)
Mar 01 12:56:01 localhost dockerd[4230]: /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/go.etcd.io/bbolt/db.go:286 +0x36c
Mar 01 12:56:01 localhost dockerd[4230]: github.com/docker/docker/vendor/github.com/docker/libkv/store/boltdb.(*BoltDB).getDBhandle(0xc0013c5950, 0x11, 0x55fbf8c1a2a0, 0x55fbf5dcf7ad)
Mar 01 12:56:01 localhost dockerd[4230]: /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/docker/libkv/store/boltdb/boltdb.go:113 +0xd1
Mar 01 12:56:01 localhost dockerd[4230]: github.com/docker/docker/vendor/github.com/docker/libkv/store/boltdb.(*BoltDB).List(0xc0013c5950, 0xc0017e7580, 0x1b, 0x0, 0x0, 0x0, 0x0, 0x0)
Mar 01 12:56:01 localhost dockerd[4230]: /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/docker/libkv/store/boltdb/boltdb.go:274 +0xd0
Mar 01 12:56:01 localhost dockerd[4230]: github.com/docker/docker/vendor/github.com/docker/libnetwork/datastore.(*cache).kmap(0xc001638260, 0x55fbf8dc3dc0, 0xc0019260d0, 0x0, 0x1b, 0x0)
Mar 01 12:56:01 localhost dockerd[4230]: /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/datastore/cache.go:43 +0x286
Mar 01 12:56:01 localhost dockerd[4230]: github.com/docker/docker/vendor/github.com/docker/libnetwork/datastore.(*cache).list(0xc001638260, 0x55fbf8dc3dc0, 0xc0019260d0, 0x0, 0x0, 0x0, 0x0, 0x0)
Mar 01 12:56:01 localhost dockerd[4230]: /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/datastore/cache.go:164 +0x7e
Mar 01 12:56:01 localhost dockerd[4230]: github.com/docker/docker/vendor/github.com/docker/libnetwork/datastore.(*datastore).List(0xc00182d980, 0xc0017e7540, 0x1b, 0x55fbf8dc3dc0, 0xc0019260d0, 0x0, 0x0, 0x0, 0x0, 0x0)
Mar 01 12:56:01 localhost dockerd[4230]: /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/datastore/datastore.go:517 +0x205
Mar 01 12:56:01 localhost dockerd[4230]: github.com/docker/docker/vendor/github.com/docker/libnetwork/drivers/bridge.(*driver).populateNetworks(0xc0014fb380, 0x5, 0x55fbf79935b6)
Mar 01 12:56:01 localhost dockerd[4230]: /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/drivers/bridge/bridge_store.go:50 +0x1d7
Mar 01 12:56:01 localhost dockerd[4230]: github.com/docker/docker/vendor/github.com/docker/libnetwork/drivers/bridge.(*driver).initStore(0xc0014fb380, 0xc000ca4990, 0x0, 0xc00182d3c0)
Mar 01 12:56:01 localhost dockerd[4230]: /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/drivers/bridge/bridge_store.go:35 +0x211
Mar 01 12:56:01 localhost dockerd[4230]: github.com/docker/docker/vendor/github.com/docker/libnetwork/drivers/bridge.(*driver).configure(0xc0014fb380, 0xc000ca4990, 0x55fbf8892560, 0xc000ca4690)
Mar 01 12:56:01 localhost dockerd[4230]: /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/drivers/bridge/bridge.go:439 +0x20d
Mar 01 12:56:01 localhost dockerd[4230]: github.com/docker/docker/vendor/github.com/docker/libnetwork/drivers/bridge.Init(0x55fbf8d66d00, 0xc001d83800, 0xc000ca4990, 0xc000ca4990, 0x0)
Mar 01 12:56:01 localhost dockerd[4230]: /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/drivers/bridge/bridge.go:169 +0x9c
Mar 01 12:56:01 localhost dockerd[4230]: github.com/docker/docker/vendor/github.com/docker/libnetwork/drvregistry.(*DrvRegistry).AddDriver(...)
Mar 01 12:56:01 localhost dockerd[4230]: /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/drvregistry/drvregistry.go:72
Mar 01 12:56:01 localhost dockerd[4230]: github.com/docker/docker/vendor/github.com/docker/libnetwork.New(0xc0014fb300, 0x9, 0x10, 0xc0008d9290, 0xc0013b7a70, 0xc0014fb300, 0x9)
Mar 01 12:56:01 localhost dockerd[4230]: /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/controller.go:221 +0x4f0
Mar 01 12:56:01 localhost dockerd[4230]: github.com/docker/docker/daemon.(*Daemon).initNetworkController(0xc00000c1e0, 0xc000155b80, 0xc0013b7a70, 0xc000334af0, 0xc00000c1e0, 0xc000830f18, 0xc0013b7a70)
Mar 01 12:56:01 localhost dockerd[4230]: /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/daemon/daemon_unix.go:854 +0xa9
Mar 01 12:56:01 localhost dockerd[4230]: github.com/docker/docker/daemon.(*Daemon).restore(0xc00000c1e0, 0xc0009570c0, 0xc000964000)
Mar 01 12:56:01 localhost dockerd[4230]: /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/daemon/daemon.go:467 +0x50b
Mar 01 12:56:01 localhost dockerd[4230]: github.com/docker/docker/daemon.NewDaemon(0x55fbf8d95fc0, 0xc0009570c0, 0xc000155b80, 0xc0008d9290, 0x0, 0x0, 0x0)
Mar 01 12:56:01 localhost dockerd[4230]: /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/daemon/daemon.go:1117 +0x2be7
Mar 01 12:56:01 localhost dockerd[4230]: main.(*DaemonCli).start(0xc0008d8660, 0xc0002fcf00, 0x0, 0x0)
Mar 01 12:56:01 localhost dockerd[4230]: /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/cmd/dockerd/daemon.go:195 +0x743
Mar 01 12:56:01 localhost dockerd[4230]: main.runDaemon(...)
Mar 01 12:56:01 localhost dockerd[4230]: /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/cmd/dockerd/docker_unix.go:13
Mar 01 12:56:01 localhost dockerd[4230]: main.newDaemonCommand.func1(0xc00021adc0, 0xc0002fd200, 0x0, 0x6, 0x0, 0x0)
Mar 01 12:56:01 localhost dockerd[4230]: /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/cmd/dockerd/docker.go:34 +0x7c
Mar 01 12:56:01 localhost dockerd[4230]: github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).execute(0xc00021adc0, 0xc00004e080, 0x6, 0x6, 0xc00021adc0, 0xc00004e080)
Mar 01 12:56:01 localhost dockerd[4230]: /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:850 +0x462
Mar 01 12:56:01 localhost dockerd[4230]: github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc00021adc0, 0x0, 0x0, 0x10)
Mar 01 12:56:01 localhost dockerd[4230]: /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:958 +0x34b
Mar 01 12:56:01 localhost dockerd[4230]: github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).Execute(...)
Mar 01 12:56:01 localhost dockerd[4230]: /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:895
Mar 01 12:56:01 localhost dockerd[4230]: main.main()
Mar 01 12:56:01 localhost dockerd[4230]: /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/cmd/dockerd/docker.go:97 +0x191
Mar 01 12:56:01 localhost systemd[1]: docker.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Mar 01 12:56:01 localhost systemd[1]: Failed to start Docker Application Container Engine.
Mar 01 12:56:01 localhost systemd[1]: Unit docker.service entered failed state.
Mar 01 12:56:01 localhost systemd[1]: docker.service failed.
Mar 01 12:56:01 localhost systemd[1]: docker.service holdoff time over, scheduling restart.

Output of docker version:

# docker --version
Docker version 20.10.1, build 831ebea

# cat /proc/version
Linux version 4.18.0-xxxx (gcc version 8.2.1 20180905 (Red Hat 8.2.1-3) (GCC)) #1 SMP Tue Nov 17 17:11:24 CST 2020

[thaJeztah]

Are there more lines in the error log?

Was this immediately after updating? Was this a fresh install of docker 20.10 or updating an existing installation?

[thaJeztah]

Looking at the output, I highly suspect some files inside /var/lib/docker got corrupted; some container state files seem to not match the actual state, and the panic in boltdb / libnetwork indicates that libnetwork's database got corrupted;

Is it possible your machine ran out of disk space at some point, or was there an unclean shutdown?

Assuming you use the default configuration (and docker's storage is in /var/lib/docker) you could try renaming the /var/lib/docker/network/files/local-kv.db file to (e.g.) local-kv.db.old, and see if docker is able to start up then.

I see there's a warning in the log output, which indicates you have some custom options set (and are using old/deprecated options), so possibly your storage is located elsewhere;

level=warning msg="The \"-g / --graph\" flag is deprecated. Please use \"--data-root\" instead"

[gosoon]

There may be a problem with the machine's disk. I will try to fix the disk first.