simulation code - step2, decentralized SGD as a reference implementation, static graph

[martinjaggi]

depends on #6

get an easy to use reference code for training (simulated decentralized, so just running locally without any p2p backend) on any given communication graph, on a standard/toy dataset. this will be useful to later to compare the p2p version to it.

for simplicity we'll first assume all nodes perform one step of SGD (gradient and communication) per clock step, and that the underlying communication graph remains fixed (the code should allow giving an arbitrary graph as an input)

[Saipraneet]

1e208c4

I think this simulation solves this issue (at least for CNN on MNIST). Uses an user defined communication graph which can be set each step. I think this also solves #2 with some minor adaptation. Additionally, we can control number of local updates in each node before communication as well (like in federated learning).

Please feel free to re-open issue/unroll commit.