Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The Vagrant Docker provider fails to stop Docker containers when using Sysbox #72

Open
ctalledo opened this issue Sep 17, 2020 · 2 comments
Labels
bug Something isn't working
Projects

Comments

@ctalledo
Copy link
Member

ctalledo commented Sep 17, 2020

Playing around with Vagrant, I was easily able to create a container using the Docker provider + Sysbox. Here is the Vagrant file:

   1 │ # encoding: utf-8                                                                                                                                                                                                                                                                                                      
   2 │ # -*- mode: ruby -*-                                                                                                                                                                                                                                                                                                   
   3 │ # vi: set ft=ruby :                                                                                                                                                                                                                                                                                                    
   4 │                                                                                                                                                                                                                                                                                                                        
   5 │ Vagrant.configure("2") do |config|                                                                                                                                                                                                                                                                                     
   6 │   config.vm.provider "docker" do |d|                                                                                                                                                                                                                                                                                   
   7 │     d.image = "alpine"                                                                                                                                                                                                                                                                                                 
   8 │     d.cmd = ["tail", "-f", "/dev/null"]                                                                                                                                                                                                                                                                                
   9 │     d.create_args = ["--runtime=sysbox-runc"]                                                                                                                                                                                                                                                                         
  10 │   end                                                                                                                                                                                                                                                                                                                  
  11 │ end    

Running vagrant up works without problem.

However, vagrant halt or vagrant destroy -f fail with a cryptic error:

ctalledo@nestybox-srv-01:~/vagrant/sysbox$ vagrant destroy -f                                                                                                                                                                                                                                                       [228/9101]
==> default: Stopping container...                                                                                                                             
==> default: Deleting the container...                                                                                                                         
Traceback (most recent call last):                                                                                                                             
        84: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/batch_action.rb:86:in `block (2 levels) in run'
        83: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/machine.rb:198:in `action'         
        82: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/machine.rb:198:in `call'                          
        81: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/environment.rb:613:in `lock'         
        80: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/machine.rb:212:in `block in action'  
        79: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/machine.rb:240:in `action_raw'                    
        78: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/runner.rb:89:in `run'       
        77: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/util/busy.rb:19:in `busy'                         
        76: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/runner.rb:89:in `block in run'
        75: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/builder.rb:116:in `call'                   
        74: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/warden.rb:48:in `call'  
        73: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/builtin/call.rb:53:in `call'                                  
        72: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/runner.rb:89:in `run'                      
        71: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/util/busy.rb:19:in `busy'                  
        70: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/runner.rb:89:in `block in run'
        69: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/builder.rb:116:in `call'         
        68: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/warden.rb:48:in `call'          
        67: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/warden.rb:107:in `block in finalize_action'
        66: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/warden.rb:48:in `call'
        65: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/builtin/call.rb:53:in `call'
        64: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/runner.rb:89:in `run'
        63: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/util/busy.rb:19:in `busy'
        62: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/runner.rb:89:in `block in run'
        61: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/builder.rb:116:in `call'
        60: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/warden.rb:48:in `call'
        59: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/warden.rb:107:in `block in finalize_action'
        58: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/warden.rb:48:in `call'
        57: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/warden.rb:107:in `block in finalize_action'
        56: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/warden.rb:48:in `call'
        55: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/builtin/call.rb:53:in `call'
        54: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/runner.rb:89:in `run'
        53: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/util/busy.rb:19:in `busy'
        52: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/runner.rb:89:in `block in run'
        51: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/builder.rb:116:in `call'
        50: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/warden.rb:48:in `call'
        49: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/warden.rb:107:in `block in finalize_action'
        48: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/warden.rb:48:in `call'
        47: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/builtin/call.rb:53:in `call'
        46: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/runner.rb:89:in `run'
        45: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/util/busy.rb:19:in `busy'
        44: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/runner.rb:89:in `block in run'
        43: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/builder.rb:116:in `call'
        42: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/warden.rb:48:in `call'
        41: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/warden.rb:107:in `block in finalize_action'
        40: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/warden.rb:48:in `call'
        39: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/builtin/config_validate.rb:25:in `call'
        38: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/warden.rb:48:in `call'
        37: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/builtin/provisioner_cleanup.rb:25:in `call'
        36: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/warden.rb:48:in `call'
        35: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/builtin/env_set.rb:19:in `call'
        34: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/warden.rb:48:in `call'
        33: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/builtin/call.rb:53:in `call'
        32: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/runner.rb:89:in `run'
        31: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/util/busy.rb:19:in `busy'
        30: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/runner.rb:89:in `block in run'
        29: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/builder.rb:116:in `call'
        28: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/warden.rb:48:in `call'
        27: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/warden.rb:107:in `block in finalize_action'
        26: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/warden.rb:48:in `call'
        25: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/warden.rb:107:in `block in finalize_action'
         24: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/warden.rb:48:in `call'
        23: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/builtin/call.rb:53:in `call'
        22: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/runner.rb:89:in `run'
        21: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/util/busy.rb:19:in `busy'
        20: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/runner.rb:89:in `block in run'
        19: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/builder.rb:116:in `call'
        18: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/warden.rb:48:in `call'
        17: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/warden.rb:107:in `block in finalize_action'
        16: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/warden.rb:48:in `call'
        15: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/plugins/providers/docker/action/stop.rb:16:in `call'
        14: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/warden.rb:48:in `call'
        13: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/warden.rb:107:in `block in finalize_action'
        12: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/warden.rb:48:in `call'
        11: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/plugins/providers/docker/action/host_machine_sync_folders_disable.rb:18:in `call'
        10: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/action/warden.rb:48:in `call'
         9: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/plugins/providers/docker/action/destroy.rb:23:in `call'
         8: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/plugins/providers/docker/driver.rb:174:in `rm'
         7: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/plugins/providers/docker/driver.rb:120:in `created?'
         6: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/plugins/providers/docker/driver.rb:280:in `execute'
         5: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/plugins/providers/docker/executor/local.rb:16:in `execute'
         4: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/util/busy.rb:19:in `busy'
         3: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/plugins/providers/docker/executor/local.rb:17:in `block in execute'
         2: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/util/subprocess.rb:22:in `execute'
         1: from /opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/util/subprocess.rb:62:in `execute'
/opt/vagrant/embedded/gems/2.2.9/gems/vagrant-2.2.9/lib/vagrant/util/subprocess.rb:62:in `pwd': No such file or directory - getcwd (Errno::ENOENT)

I am not quite sure what's going on, but in Linux this type of error generally occurs when a command is executed from within a directory that no longer exists. That does not appear to be the case here, at least on the surface.

Speculating here: I noticed that when Vagrant runs the container, it mounts the Vagrant file in the host into the container's /vagrant directory. This causes Sysbox to mount shiftfs on the host directory where the Vagrant file is located. This in turn makes that directory non-exec. I wonder if this is playing a role somehow.

Also: after the error occurs, the container is stopped but not removed. It must be removed explicitly with "docker rm".

@ctalledo ctalledo added the bug Something isn't working label Sep 17, 2020
@ctalledo ctalledo added this to To do in Sysbox Dev via automation Sep 17, 2020
@ctalledo
Copy link
Member Author

Another hint: I noticed that if I launch multiple sys containers with vagrant, and then destroy them, the destruction goes well for all but the last remaining sys container. In the prior comment I speculated on the shiftfs mount being the culprit, and this latest observation adds weight to that since shiftfs would only be unmounted when destroying the last remaining container.

@ctalledo
Copy link
Member Author

I also noticed another problem: when provisioning 16 sys containers with Vagrant (using the Docker provider with Sysbox), normally one of the containers fails to provision, with the following error:

An error occurred while executing multiple actions in parallel.                                                                                                
Any errors that occurred are shown below.                                                                                                                      
                                                                                                                                                               
An error occurred while executing the action on the 'server14'                                                                                                 
machine. Please handle this error then try again:                                                                                                              
                                                                                                                                                               
A Docker command executed by Vagrant didn't complete successfully!                                                                                             
The command run along with the output from the command is shown                                                                                                
below.                                                                                                                                                         
                                                                                                                                                               
Command: ["docker", "run", "--name", "syscont_server14_1600385625", "-d", "-v", "/home/ctalledo/vm-vs-syscont-perf/provision-time/syscont:/vagrant", "--runtime=sysbox-runc", "--cpus=2", "-m=2g", "benchmark-syscont", {:notify=>[:stdout, :stderr]}]
                                                                                                                                                               
Stderr: WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.                               

/usr/bin/docker: Error response from daemon: OCI runtime create failed: error in the container spec: invalid mount config: failed to request mounts from sysbox-mgr: failed to invoke ReqMounts via grpc: rpc error: code = DeadlineExceeded desc = context deadline exceeded: unknown.
                                                                                                                                                                                                                                                                                                                                                 
Stdout: b151c7e0164214b46f4f058a6a33d50f1e95c5acd7e99616f979b0082850afe5         

It seems that sysbox-runc is complaining that sysbox-mgr has taken too long to respond to a request issued via grpc. We need to investigate why sysbox-mgr is taking long enough to exceed the grpc deadline, and whether the grpc deadline is set correctly or should be increased.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Sysbox Dev
  
To do
Development

No branches or pull requests

1 participant