Local restore of LocalScope drivers such as bridge and host driver

[chenchun]

I've took a look at the current store/datastore usage in libnetwork code base. I think libkv is mainly used for service discovery of GlobalScope network such as overlay and remote driver.
In order to implement docker daemon hot upgrade moby/moby#13884, lots of states of bridge driver like ip/port need to be preserved and recovered from data store after restart of daemon. And I think its inappropriate to use etcd/consol/zookeeper to store the states of LocalScope drivers.
I propose to add a new backend like leveldb to libkv so that we can use two kinds of state store to persist states of both GlobalScope and LocalScope driver.
@abronan @mavenugo WDYT?

[mavenugo]

@chenchun Yes. @sanimej is actively working on boltdb as a libkv backend.
We feel that it works best for the LocalScope drivers.

[chenchun]

@mavenugo, thanks. I believe I can just wait for boltdb backend and keep going implementing libnetwork LocalScope restore with libkv api.

[mavenugo]

@chenchun in order to avoid duplicating the work, it would be good to sync up with @sanimej on the current state of the implementation. I believe he also got the LocalScope integration done with boltdb as a backend in libkv.

[chenchun]

@mavenugo, thanks. @sanimej, can you share your current branch of the implementation?

[sanimej]

@chenchun I am making some changes and testing it.. will share the branch shortly.

[sanimej]

@chenchun here is the branch with my changes.. https://github.com/sanimej/libkv/tree/boltdb

I will be making some additional changes. But you can started with the libnetwork integration.

[mavenugo]

@sanimej Thanks.
@chenchun I synced up with @sanimej & he hasn't started on the libnetwork integration with the libkv based boltdb backend yet. If you are interested, please push a PR.
It would be helpful if you can share your design idea so that we can review it before you start to integrate the libkv changes that @sanimej provided.

From a design standpoint, we were thinking of keeping it simple, by having 2 libkv stores (one for GlobalScope and another for LocalScope) managed by the NeworkController. For any locally scoped objects, we can use the locally scoped db that is backed by boltdb. The rest of the code can just stay as is without any changes. WDTY ?

[chenchun]

I'm not started too. Yes, I planned to keep two objects of libkv stores. I'll think about it and post my idea here.

[mavenugo]

@chenchun Thanks

[chenchun]

@sanimej, thanks for providing the branch.

[mavenugo]

@chenchun can you please share the design ? Also, the local-scope persistence is of priority for other issues that we are seeing. Is this something that you can handle soon ?

[chenchun]

Sorry for the delay. My design is as below

Add a localStore datastore.DataStore field in controller.
Add a LocalDatastore DatastoreCfg field in Config.
We can choose the correct store in updateNetworkToStore/deleteNetworkFromStore/updateEndpointToStore/deleteEndpointFromStore according to the scope of the network.
For localStore there is no need to watchNetworks/watchEndpoints

The rest of the code can just stay as is without any changes

I didn't see the endpoints are restored after restart daemon restart. Does restoring endpoints have to include in?

[mavenugo]

@chenchun thanks. The design is inline with my thought process as well.

I didn't see the endpoints are restored after restart daemon restart. Does restoring endpoints have to include in?

Well, we depend on watchEndpoints to do the restore during daemon restart and it does work for global datastore. But, if we don't support watch for localStore, then we will have to implement a c.getEndpointsFromStore just like c.processNetworkUpdate and use the existing c.processEndpointUpdate to trigger the update in initDataStore. I think that should be simple to do.
WDYT ?

[chenchun]

OK, I will make a PR when I go to the office shortly.

[sanimej]

@chenchun I noticed an issue with AtomicPut.. I am in the process of committing a change. You can pick it up once its in for your integration.

[mavenugo]

Just to be clear, the libkv side changes are being tracked under docker/libkv#46