daemon ungraceful shutdown during starting make daemon failed to start next time with Error initializing network controller

[coolljt0725]

During daemon starting, if the daemon was shutdown ungracefully, the daemon may will failed to start next time with error
FATA[0001] Error starting daemon: Error initializing network controller: Error creating default "bridge" network: failed to allocate gateway (172.17.0.1): Address already in use
we hit this problem on one of our server, on this server, the daemon failed to start with timeout(because there is a process take out all of the cup resource, 400% on a 4-core server)

and after several times of timeout failure, the failure became Error starting daemon: Error initializing network controller: Error creating default "bridge" network: failed to allocate gateway (172.17.0.1): Address already in use. I investigated and found that this may due to the the ungraceful shutdown happened between https://github.com/docker/docker/blob/master/vendor/src/github.com/docker/libnetwork/controller.go#L507 to https://github.com/docker/docker/blob/master/vendor/src/github.com/docker/libnetwork/controller.go#L535, the controller has allocated ip pool and stored the bitmap to store, but the network failed to finish the initialization and didn't store the network to store. so this ip pool doesn't belong to any network, so it can't be cleaned up.
It's hard to reproduce, to produce you can add

if name == "bridge" {
       log.Infof("after ipamAllocate, please kill the daemon")
       time.Sleep(10* time.Second)
       log.Infof("after sleep")
}

after https://github.com/docker/docker/blob/master/vendor/src/github.com/docker/libnetwork/controller.go#L506 and the kill the daemon when we get message after ipamAllocate, please kill the daemon
and then start again, the daemon will failed with FATA[0001] Error starting daemon: Error initializing network controller: Error creating default "bridge" network: failed to allocate gateway (172.17.0.1): Address already in use

To fix this, I think we should have a way to clean up unused allocated ip pool or synchronized the store of bit map and network.

[thaJeztah]

@coolljt0725 could you open an issue in the libnetwork repository as well? Looks like this is a libnetwork issue