/etc/hostname Device or resource busy

[ocxo]

I am using Chef to update /etc/hostname and /etc/hosts which fails with this error:

Errno::EBUSY
------------
Device or resource busy - (/tmp/.hostname20141123-16-kyb4i1, /etc/hostname)

This was supposedly fixed in 1.2.0. Any ideas what I could be doing wrong or if this is a regression?

I am running on OS X 10.10 via boot2docker on Ubuntu 12.04.

Here are some more environment details:

$ uname -a
Darwin serafina.local 14.0.0 Darwin Kernel Version 14.0.0: Tue Sep 23 23:01:41 PDT 2014; root:xnu-2782.1.97~6/RELEASE_X86_64 x86_64

$ docker version
Client version: 1.3.0
Client API version: 1.15
Go version (client): go1.3.3
Git commit (client): c78088f
OS/Arch (client): darwin/amd64
Server version: 1.3.1
Server API version: 1.15
Go version (server): go1.3.3
Git commit (server): 4e9bbfa

$ docker -D info
Containers: 0
Images: 11
Storage Driver: aufs
 Root Dir: /mnt/sda1/var/lib/docker/aufs
 Dirs: 11
Execution Driver: native-0.2
Kernel Version: 3.16.4-tinycore64
Operating System: Boot2Docker 1.3.1 (TCL 5.4); master : 9a31a68 - Fri Oct 31 03:14:34 UTC 2014
Debug mode (server): true
Debug mode (client): true
Fds: 10
Goroutines: 13
EventsListeners: 0
Init Path: /usr/local/bin/docker

[vbatts]

@fromonesrc could you provide more information according to https://github.com/docker/docker/blob/master/CONTRIBUTING.md#reporting-issues
thanks

[ocxo]

Sorry about that, will follow up soon as I can.

[eigood]

The issue is that even though you can "write" to those files, you can't rename them. This is due to both files being bind mounted. I am using ansible to automatically update /etc/hosts with references to either other machines, or to localhost. I then place lines like the following in the file:

x.x.x.x $app_service.config

The method that ansible uses to update a file, is to write it to a temporary location, then do an atomic rename. This is a fairly standard way to update files, while ensuring safety that the new file has been saved to disk.

[ocxo]

@vbatts I've updated the issue with more detail about my env.

[epcim]

+1

(the ability to write/modify /etc/hosts, /etc/resolv.conf and possibbly other tools is key for any deployment CI testing on docker on any image and any tooling (chef, ansible, puppet, salt)).

@vbatts - The ability to specify --dns --add-host is fine. But the use case is to run deployment testing within a docker container. So since the deployment tools running test modify filesystem withing the image they must have an option to modify any part of the FS. Even the performance will be affected.

[epcim]

I have found the same issue on Docker 1.3, 1.4. The cause was not in docker. The chef hostfile cokbook was using a file resource (which ootb enables atomic_update: true) and that caused the issue on /etc/hosts file update.

When I logged in the new docker image
sudo docker run -i -t epcim/docker-chefdk:latest /bin/bash I could manually modify /etc/hosts (rw). I could do it also using recipe and chef-apply. However whent I extended a recipe to match 1:1 the failed deployment - so it was also including a diff attribute from file resource then the chef-apply failed with the "Resource Busy" clausule while the atomic_update was true:

However, with false it works:

test-cookbook# cat modhosts.rb 


file("/etc/hosts") do       
        action "create"       
        retries 0       
        retry_delay 2       
        path "/etc/hosts"       
        backup 5       
        atomic_update false
    diff "--- /etc/hosts\t2015-01-26 12:13:31.629974265 +0000\\n+++ /etc/.hosts20150126-113-1d8sxon\t2015-01-26 12:19:43.104093084 +0000\\n@@ -1,8 +1,17 @@\\n+#\\n+# This file is managed by Chef, using the hostsfile cookbook.\\n+# Editing this file by hand is highly discouraged!\\n+#\\n+# Comments containing an @ sign should not be modified or else\\n+# hostsfile will be unable to guarantee relative priority in\\n+# future Chef runs!\\n+#\\n+\\n+127.0.0.1\tlocalhost\\n 10.0.0.10\t6e48fccf181d\\n-ff00::0\tip6-mcastprefix\\n ff02::1\tip6-allnodes\\n ff02::2\tip6-allrouters\\n-127.0.0.1\tlocalhost\\n+fe00::\tip6-localnet\\n+ff00::\tip6-mcastprefix\\n ::1\tlocalhost ip6-localhost ip6-loopback\\n-fe00::0\tip6-localnet"       
        content "#\n# This file is managed by Chef, using the hostsfile cookbook.\n# Editing this file by hand is highly discouraged!\n#\n# Comments containing an @ sign should not be modified or else\n# hostsfile will be unable to guarantee relative priority in\n# future Chef runs!\n#\n\n127.0.0.1\tlocalhost\n10.0.0.10\t6e48fccf181d\nff02::1\tip6-allnodes\nff02::2\tip6-allrouters\nfe00::\tip6-localnet\nff00::\tip6-mcastprefix\n::1\tlocalhost ip6-localhost ip6-loopback\n"       
end

The solution could be to use chef_client config option on .kitchen.yml:

suites:
  - name: default
    run_list:
        - recipe[lab_base-srv::default]
    attributes:
      chef_client:
        config:
          file_atomic_update: false

However I guess, chef_client cookbook do not set this option at the current state: chef-boneyard/chef-client#248

Also I run immediately to another issue while modifing /etc/resolv.conf with template resource:

 ================================================================================       
    Error executing action `create` on resource 'template[/etc/resolv.conf]'       
    ================================================================================       

    Errno::EBUSY       
    ------------       
    Device or resource busy @ sys_fail2 - (/tmp/chef-rendered-template20150126-113-10b6x14, /etc/resolv.conf)       

    Resource Declaration:       
    ---------------------       
    # In /tmp/kitchen/cache/cookbooks/resolver/recipes/from_server_role.rb       

     31: template "/etc/resolv.conf" do       
     32:   source "resolv.conf.erb"       
     33:   owner "root"       
     34:   group "root"       
     35:   mode 0644       
     36:   variables(       
     37:     'search' => node['resolver']['search'],       
     38:     'nameservers' => nameservers.sort,       
     39:     'options' => node['resolver']['options']       
     40:   )       
     41: end       
     42:        

but due the chef-apply issue with template resource I did not proceed further - https://tickets.opscode.com/browse/CHEF-4002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel

[jessfraz]

closed see comment about problem not within docker but chef #9295 (comment)

[jdmulloy]

@epcim I found out that the correct place to put the setting in .kitchen.yml is in the provisioner section like so.

suites:
  - name: default
    run_list:
      - recipe[lab_base-srv::default]
    provisioner:
      client_rb:
        file_atomic_update: false

You can also put this in the global provisioner section if you wish.

[hdeadman]

For future people that come across this issue, I thought I would leave link to the documentation in Docker that explains why Puppet, Chef, etc can't update /etc/hosts, /etc/resolv.conf, & /etc/hostname in a container (at least not if they are doing an atomic rename that they would normally do).

-- https://docs.docker.com/articles/networking/#dns

That documentation explains how those three files are special overlays managed by Docker as part of its networking support. Essentially your provisioning scripts ought to leave them alone and you can pass command-line arguments to docker to influence their contents. From the docs -

... you should leave the files themselves alone and use the following Docker options instead.

[lamont-granquist]

@jdmulloy Hi, I'm the software developer at chef who actually wrote the file_atomic_update code. You should not be abusing the global setting in that way. The right way to fix this is to update only the specific file resources that need to use it. Your workaround will work right up until it invalidates your testing and you deploy code to production that fails which passed in your preproduction testing. And if you deploy this setting to production and flip every file resource in your production runs to not be atomic you incur a severe risk of incurring data loss, corruption of files and crashing.

I'll be updating the Chef::Config internal documentation and the chef.docs.io website with big warnings not to tweak that global setting, along with updating the docs.chef.io site with warnings against doing that.

https://github.com/chef/chef/pull/4696/files

I've also released a chef_hostname cookbook which has the right setting and supports docker containers:

https://supermarket.chef.io/cookbooks/chef_hostname

[marzzz21]

@lamont-granquist , so how do we properly rectify this issue?

[lamont-granquist]

set atomic_updates false only on the resource which is updating /etc/hosts or /etc/hostname or whatever in the docker container and /when/ it is running inside a docker container.

you can even re-open file or template resources used by other resources and patch them to apply it in a wrapper recipe:

resources(template: "/etc/hosts").atomic_update true

throw that in a recipe somewhere after template "/etc/hosts" do ... end has been created on the resource collection.

for editing /etc/hostname and/or /etc/hosts to set the hostname the chef_hostname resource does that for you, just use it:

https://supermarket.chef.io/cookbooks/chef_hostname

if you are using some other cookbook to edit /etc/hosts then that cookbook should get patched, but you can use the previous technique to open up the resource collection and dynamically patch the resource it creates.

[lamont-granquist]

The resolver cookbook issue that @epcim mentioned above is also being fixed here:

sous-chefs/resolver#22

[lamont-granquist]

The chef/chef#4949 issue has been opened to solve this in chef core for all file resources and detect docker guests and add the atomic_updates flag by default -- only for /etc/hosts, /etc/hostname and /etc/resolv.conf and only in docker guests.

[Martin1982]

The page mentioned in comment #9295 (comment) is no longer available