Successive docker starts do not start container every time

[hmahmood]

One docker start call after another in quick succession on the same container sometimes will return 125 exit code from the docker cli.

Steps to reproduce:

1. docker create --name test busybox echo foo
2. run in a tight loop: docker start -a test; should return "foo" and exit code 0 every time, but sometimes will return 125 with or without the correct output

[hmahmood]

For at least the case where we get a 125 and no output, I found out that the start command does not start the container. We get the powered off event for the previous start call late, and set the container state to stopped for the current call before we can actually power on the VM, which results in the container not getting started.

[hickeng]

Where do we abort a start call based on the container state? It's supposed to be an idempotent operation, so portlayer would just do nothing if the container is already in the desired state - but that should be being determined based on the VM runtime structure, not the container cache state, which is primarily for faking the transition states.

[mhagen-vmware]

@hmahmood Please prioritize and estimate and move to backlog.

[caglar10ur]

vspc holds the connection even after the vm powers off up to 10 sec. so starting the vm again before this loops run causes issues.

267 // monitorVMConnections is to monitor connections and delete VM recoreds from the vmManager map when the containerVM exits
268 func (vspc *Vspc) monitorVMConnections() {
269     ticker := time.NewTicker(VMHealthCheckPeriod)
270     go func() {
271         for {
272             select {
273             case <-ticker.C:
274                 vspc.vmManagerMu.Lock()
275
276                 for k, vm := range vspc.vmManager {
277                     vm.Lock()
278                     if !vm.inVmotion && vm.containerConn.IsClosed() { // vm just shut down
279                         log.Debugf("(vspc) detected closed connection for VM %s", k)
280                         log.Debugf("(vspc) closing connection to the AttachServer")
281                         vm.remoteConn.Close()
282                         log.Debugf("(vspc) deleting vm records from the vm manager %s", k)
283                         delete(vspc.vmManager, k)
284                     }
285                     vm.Unlock()
286                 }
287                 vspc.vmManagerMu.Unlock()
288             case <-vspc.doneCh:
289                 ticker.Stop()
290                 return
291             }
292         }
293     }()
294 }

[stuclem]

@mhagen-vmware should this go in the Release Notes for 1.1?