Running exec during container create results in a `ConcurrentAccess` error #4881

jakedsouza · 2017-04-25T19:37:25Z

VIC version:
vic-machine-linux version v0.8.0-0-91097e2 on master

Deployment details:
Single VCH deployed on a cluster

What was the vic-machine create command used to deploy the VCH?

./bin-master/vic-machine-linux create -t 10.192.250.50 -u="administrator@vsphere.local" -p="pass" --image-store="datastore11/${vchName}-imagestore" --volume-store="datastore11/${vchName}-volumestore:volumestore" --volume-store="datastore11/${vchName}-volumestore:default"  --tls-cname="*.eng.vmware.com"  --timeout 6m --name ${vchName} --thumbprint=08:87:A6:D6:B8:99:70:51:F5:1F:42:1B:52:C2:59:97:8A:2A:0C:34 --compute-resource=cls --bridge-network=bridge --insecure-registry=10.192.236.96 --debug=2 --no-tls

Steps to reproduce:

Deploy VCH
In terminal 1 run exec command repeatedly

watch  -n 1 docker exec -d c1  ls /

In terminal 2 start and run a container

docker run -it --name c1 -v test17:/etc/example alpine /bin/ash

Actual behavior:

We do see a valid error Error response from daemon: No such container while the container vm is being created
After this, we see another error

Server error from portlayer: [PUT /containers    /{handle}][409] commitConflict  &{Code:0 Message:Cannot complete operation due to concurrent modification by another operation.}

Expected behavior:

Expect to see an error Error response from daemon: No such container while the container vm is being created
See no error once the container is created

NOTE

While we do see an error, it is tricky to reproduce consistently as it seems to occur during tether startup.
Also AFAIK the error does not seem to affect the container creation. All functionality seems to work fine.

Logs:

Attached logs. The container is named large in the logs

container-logs.zip

The text was updated successfully, but these errors were encountered:

caglar10ur · 2017-04-25T23:01:39Z

Added retry support to exec branch (#4339) to handle this (regular attach path already does that)

@hmahmood

* Attach support for exec'ed session Includes @hmahmood's and @hickeng's various fixed to dio/persona and portlayer Fixes #4263 Fixes #4367 Fixes #4598 Fixes #4410 Fixes #4339 Fixes #4881 Requires #4288 Implements exec functionality for personality Those changes live in lib/apiservers/engine/backends/ directory. There are changes in container_proxy functions since exec and attach shares some common code paths. We started to pass ids instead of viccontainer structs around to them to support both workflows. AttachStreams function splits the cancelation logic into different parts. Closing stdin cancels other active streams. We also ensure that the API client connection is shutdown on exit. Implements exec support for portlayer rest api Those changes live in lib/apiservers/portlayer/ directory. One important change there is context changes. In the past we were calculating the timeouts manually and then passing timeouts as a parameter to Get call. session, err := attachServer.Get(context.Background(), params.ID, timeout) now each stream is creating its own context and passing it to the function - eliminating manual calculations and redundant timeout value. session, err := i.server.Interaction(ctx, params.ID) Portlayer attach subsystem re-organization Those changes live in lib/portlayer/attach/. Some of them just renames and splitting monolithic file (they are now bind.go/join.go etc.) into pieces. Connector has new set of features. We now have a different unblock signal for signaling tether (called unblock) for starting the processes inside the container. We also now have a SSH channel mux running on client side to receive container ids from tether. Normally portlayer initiates the connection and asks container ids. This is still the case for the primary session. But after that primary session start event, any exec'ed session will cause tether to restart and as a part of that tether will push container ids to the portlayer. Additionally we now started to use channel mux to evict closed connections from the connection map (#4367). Lastly we started to use singleflight to make sure that there is only one call in-flight for each ID at any given time. tether support for exec As noted above, tether now pushes container ids to portlayer when it receives reload signal. It supports new unblock signal. Fixes a bug caused by wrong usage of cleanup functions. Fixes races causing incorrect behaviors. dio changes dio reader now has a PropagateEOF function toggles whether to return EOF when all readers return EOF. Setting this to true will result in an EOF if there are no readers available when Read is next called. dio writer now calls CloseWrite if the writes implements CloseWriter.

@hmahmood

* Attach support for exec'ed session Includes @hmahmood's and @hickeng's various fixed to dio/persona and portlayer Fixes vmware#4263 Fixes vmware#4367 Fixes vmware#4598 Fixes vmware#4410 Fixes vmware#4339 Fixes vmware#4881 Requires vmware#4288 Implements exec functionality for personality Those changes live in lib/apiservers/engine/backends/ directory. There are changes in container_proxy functions since exec and attach shares some common code paths. We started to pass ids instead of viccontainer structs around to them to support both workflows. AttachStreams function splits the cancelation logic into different parts. Closing stdin cancels other active streams. We also ensure that the API client connection is shutdown on exit. Implements exec support for portlayer rest api Those changes live in lib/apiservers/portlayer/ directory. One important change there is context changes. In the past we were calculating the timeouts manually and then passing timeouts as a parameter to Get call. session, err := attachServer.Get(context.Background(), params.ID, timeout) now each stream is creating its own context and passing it to the function - eliminating manual calculations and redundant timeout value. session, err := i.server.Interaction(ctx, params.ID) Portlayer attach subsystem re-organization Those changes live in lib/portlayer/attach/. Some of them just renames and splitting monolithic file (they are now bind.go/join.go etc.) into pieces. Connector has new set of features. We now have a different unblock signal for signaling tether (called unblock) for starting the processes inside the container. We also now have a SSH channel mux running on client side to receive container ids from tether. Normally portlayer initiates the connection and asks container ids. This is still the case for the primary session. But after that primary session start event, any exec'ed session will cause tether to restart and as a part of that tether will push container ids to the portlayer. Additionally we now started to use channel mux to evict closed connections from the connection map (vmware#4367). Lastly we started to use singleflight to make sure that there is only one call in-flight for each ID at any given time. tether support for exec As noted above, tether now pushes container ids to portlayer when it receives reload signal. It supports new unblock signal. Fixes a bug caused by wrong usage of cleanup functions. Fixes races causing incorrect behaviors. dio changes dio reader now has a PropagateEOF function toggles whether to return EOF when all readers return EOF. Setting this to true will result in an EOF if there are no readers available when Read is next called. dio writer now calls CloseWrite if the writes implements CloseWriter.

@hmahmood

* Attach support for exec'ed session Includes @hmahmood's and @hickeng's various fixed to dio/persona and portlayer Fixes vmware#4263 Fixes vmware#4367 Fixes vmware#4598 Fixes vmware#4410 Fixes vmware#4339 Fixes vmware#4881 Requires vmware#4288 Implements exec functionality for personality Those changes live in lib/apiservers/engine/backends/ directory. There are changes in container_proxy functions since exec and attach shares some common code paths. We started to pass ids instead of viccontainer structs around to them to support both workflows. AttachStreams function splits the cancelation logic into different parts. Closing stdin cancels other active streams. We also ensure that the API client connection is shutdown on exit. Implements exec support for portlayer rest api Those changes live in lib/apiservers/portlayer/ directory. One important change there is context changes. In the past we were calculating the timeouts manually and then passing timeouts as a parameter to Get call. session, err := attachServer.Get(context.Background(), params.ID, timeout) now each stream is creating its own context and passing it to the function - eliminating manual calculations and redundant timeout value. session, err := i.server.Interaction(ctx, params.ID) Portlayer attach subsystem re-organization Those changes live in lib/portlayer/attach/. Some of them just renames and splitting monolithic file (they are now bind.go/join.go etc.) into pieces. Connector has new set of features. We now have a different unblock signal for signaling tether (called unblock) for starting the processes inside the container. We also now have a SSH channel mux running on client side to receive container ids from tether. Normally portlayer initiates the connection and asks container ids. This is still the case for the primary session. But after that primary session start event, any exec'ed session will cause tether to restart and as a part of that tether will push container ids to the portlayer. Additionally we now started to use channel mux to evict closed connections from the connection map (vmware#4367). Lastly we started to use singleflight to make sure that there is only one call in-flight for each ID at any given time. tether support for exec As noted above, tether now pushes container ids to portlayer when it receives reload signal. It supports new unblock signal. Fixes a bug caused by wrong usage of cleanup functions. Fixes races causing incorrect behaviors. dio changes dio reader now has a PropagateEOF function toggles whether to return EOF when all readers return EOF. Setting this to true will result in an EOF if there are no readers available when Read is next called. dio writer now calls CloseWrite if the writes implements CloseWriter.

jakedsouza added component/portlayer component/portlayer/interaction component/tether kind/defect Behavior that is inconsistent with what's intended labels Apr 25, 2017

jakedsouza mentioned this issue Apr 25, 2017

Copy existing volume data to mounts [full ci] #4835

Merged

caglar10ur mentioned this issue Apr 25, 2017

[full ci] Attach support for exec'ed session #4339

Merged

caglar10ur closed this as completed in #4339 Apr 27, 2017

mdubya66 added this to the Sprint 8 milestone May 1, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Running exec during container create results in a `ConcurrentAccess` error #4881

Running exec during container create results in a `ConcurrentAccess` error #4881

jakedsouza commented Apr 25, 2017

caglar10ur commented Apr 25, 2017

Running exec during container create results in a ConcurrentAccess error #4881

Running exec during container create results in a ConcurrentAccess error #4881

Comments

jakedsouza commented Apr 25, 2017

caglar10ur commented Apr 25, 2017

Running exec during container create results in a `ConcurrentAccess` error #4881

Running exec during container create results in a `ConcurrentAccess` error #4881