chef_zero provisioner fails to load data into chef-zero server

[drags]

Split out of #5339:

I am experiencing issues with the provisioner not loading data into the chef-zero server, here's a paste of the (non debug) log output to give some context:

$ vagrant destroy -f; time vagrant up --provision
==> default: VM not created. Moving on...
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Importing base box 'ubuntu_nocm'...
==> default: Matching MAC address for NAT networking...
==> default: Setting the name of the VM: chef-repo_default_1439861960499_8145
==> default: Fixed port collision for 22 => 2222. Now on port 2200.
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
    default: Adapter 1: nat
==> default: Forwarding ports...
    default: 22 => 2200 (adapter 1)
==> default: Running 'pre-boot' VM customizations...
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 127.0.0.1:2200
    default: SSH username: vagrant
    default: SSH auth method: private key
    default: Warning: Connection timeout. Retrying...
    ==> default: Machine booted and ready!
==> default: Checking for guest additions in VM...
==> default: Mounting shared folders...
    default: /vagrant => /Users/tim/src/chef-repo
    default: /tmp/vagrant-chef/8addf10b12dfed40403da41012b18717/roles => /Users/tim/src/chef-repo/roles
    default: /tmp/vagrant-chef/f9ae443f73e3f920af0c78cbbb967ef4/cookbooks => /Users/tim/src/chef-repo/dep_cookbooks
    default: /tmp/vagrant-chef/6fd3da73972d018f85ba5733c51be7a2/cookbooks => /Users/tim/src/chef-repo/cookbooks
    default: /tmp/vagrant-chef/f5904b8c9b12a4eb12e865c109951859/data_bags => /Users/tim/src/chef-repo/data_bags
==> default: Running provisioner: chef_zero...
    default: Installing Chef (latest)...
Generating chef JSON and uploading...
==> default: Running chef-client (local-mode)...
==> default: stdin: is not a tty
==> default: [2015-08-18T01:40:23+00:00] INFO: Started chef-zero at chefzero://localhost:8889 with repository at /tmp/vagrant-chef/6fd3da73972d018f85ba5733c51be7a2, /tmp/vagrant-chef/f9ae443f73e3f920af0c78cbbb967ef4
==> default:   One version per cookbook
==> default:   data_bags at /tmp/vagrant-chef/f5904b8c9b12a4eb12e865c109951859/data_bags
==> default:   roles at /tmp/vagrant-chef/8addf10b12dfed40403da41012b18717/roles
==> default: [2015-08-18T01:40:23+00:00] INFO: Forking chef instance to converge...
==> default: Starting Chef Client, version 12.4.1
==> default: [2015-08-18T01:40:23+00:00] INFO: *** Chef 12.4.1 ***
==> default: [2015-08-18T01:40:23+00:00] INFO: Chef-client pid: 1808
==> default: [2015-08-18T01:40:29+00:00] INFO: HTTP Request Returned 404 Not Found: Object not found: chefzero://localhost:8889/nodes/vagrant-ubuntu-trusty-64
==> default: [2015-08-18T01:40:29+00:00] INFO: Setting the run_list to ["role[basev2]"] from CLI options
==> default: [2015-08-18T01:40:29+00:00] INFO: HTTP Request Returned 404 Not Found: Object not found: chefzero://localhost:8889/roles/basev2
==> default: [2015-08-18T01:40:29+00:00] ERROR: Role basev2 (included by 'top level') is in the runlist but does not exist. Skipping expand.
==> default:
==> default: ================================================================================
==> default: Error expanding the run_list:
==> default: ================================================================================
==> default:
==> default:
==> default: Missing Role(s) in Run List:
==> default: ----------------------------
==> default: * basev2 included by 'top level'
==> default:
==> default:
==> default: Original Run List
==> default: -----------------
==> default: * role[basev2]
==> default:
==> default:
==> default: Running handlers:
==> default: [2015-08-18T01:40:29+00:00] ERROR: Running exception handlers
==> default: Running handlers complete
==> default: [2015-08-18T01:40:29+00:00] ERROR: Exception handlers complete
==> default: Chef Client failed. 0 resources updated in 6.134546019 seconds
==> default: [2015-08-18T01:40:29+00:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
==> default: [2015-08-18T01:40:29+00:00] ERROR: The expanded run list includes nonexistent roles: basev2
==> default: [2015-08-18T01:40:31+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
Chef never successfully completed! Any errors should be visible in the
output above. Please fix your recipes so that they properly complete.

real    1m20.281s
user    0m4.126s
sys     0m1.749s

If I attempt to run curl -H "Accept: application/json" localhost:8889/roles on the box during the chef-client -z run then I get this output:

{

}

If I attempt to run curl -H "Accept: application/json" localhost:8889/cookbooks during the chef-client -z run then I get a 0 length response.

I was trying to debug from the chef-zero side, and I'm unclear on how this is supposed to work (especially with the chef_zero provisioner accepting an array of cookbook paths). From what I can tell chef-zero will allow either 1) serving from a directory which it expects to be a server repo directory (with ./roles. ./cookbooks, etc), or 2) act as a local listener where knife <subcommand> -z is used to upload items to the chef-zero server.

For reference I've put a full vagrant up --provision --debug output here: https://gist.github.com/drags/d6d4c0a7e8b3e13b8bca (oh goodness, I almost posted my AWS keys since vagrant debug output includes full env.. that is.. not a safe default)

I've tried both with latest chef as well as 11.18

My Vagrantfile:

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu_nocm"

  config.vm.provider "virtualbox" do |vb|
     vb.customize ["modifyvm", :id, "--memory", "4096"]
  end

  config.vm.provision "chef_zero" do |chef|
    chef.cookbooks_path = ["cookbooks", "dep_cookbooks"]
    chef.data_bags_path = "data_bags"
    chef.roles_path = "roles"
    chef.add_role "basev2"

  end

end

I'm running into roadblocks on all fronts trying to debug, all pointers/suggestions greatly appreciated.

Thanks,
Tim

[sethvargo]

Hi Tim,

What are the contents of your roles directory?

[drags]

The roles directory contains the ruby role definitions (ex: basev2.rb)

Additionally, here's the layout of where Vagrant places all of the mounted files when using the given Vagrantfile:

vagrant@vagrant-ubuntu-trusty-64:/tmp$ tree -L 2 vagrant-chef/
vagrant-chef/
├── 6fd3da73972d018f85ba5733c51be7a2
│   ├── cookbooks
│   └── nodes
├── 8addf10b12dfed40403da41012b18717
│   └── roles
├── client.rb
├── dna.json
├── f5904b8c9b12a4eb12e865c109951859
│   └── data_bags
└── f9ae443f73e3f920af0c78cbbb967ef4
    ├── cookbooks
    └── nodes

[sethvargo]

Hi @drags

The Chef Zero provisioner does not support Ruby roles. It never has 😄 - that's your issue. There's at least a few open tickets on the Chef tracker.

[drags]

@sethvargo

Thanks, that got me going. Pardon the noise

[joelbernstein]

@sethvargo I've been using the chef-zero provisioner successfully with a ruby role for months, this doesn't seem so plausible. If it never has, why has it been working for me? It's only since Vagrant 1.7.4 and VirtualBox 5 that it breaks.

[joelbernstein]

there's also nothing obvious on https://github.com/chef/chef/issues?utf8=%E2%9C%93&q=roles+chef-zero+
please advise...

[sethvargo]

@joelbernstein aside from the fact that I used to work on Chef Zero and many other Chef tools 😄 there are a number of places on the Internet:

• https://www.chef.io/blog/2014/06/24/from-solo-to-zero-migrating-to-chef-client-local-mode/#comment-1463181584
• knife diff / ChefFS / chef-zero and JSON vs ruby DSL files chef/chef#2502

Depending on how far down the rabbit hole you go, the issue isn't actually Chef Zero, it's a dependency of zero called ChefFS (Chef File System) that takes files on disk and turns them into a Chef Server. Chef Zero uses this to create a fake Chef Server in memory and then use the disk for persistence.

Now, the reason this used to work is that Vagrant was not using Chef Zero before. You can see more information in #5072 and #5619, but basically we were running Chef Solo for the Chef Zero integration since the initial launch. This was fixed in 4b1847a.

So, the reason it's been working for you for the past few months is that you weren't actually running Chef Zero. I realize this can be confusing and I apologize for the back-and-forth.

[joelbernstein]

@sethvargo thank you for the patient response! It hadn't sunk in before but a journey down the rabbit hole (thanks for the very useful links) has been informative.

I guess, then, if my Chef code has been working fine for the last few months with the chef-zero provisioner, it must be because it works fine with chef-solo. So I should continue using that, at least until I rewrite my roles as json?

[sethvargo]

@joelbernstein it depends on your use case. IMO, you should rewrite them as JSON because they are static data (has nothing to do with Vagrant). That being said, if you want to continue using Ruby, just switch to Solo. If your recipes worked in Vagrant 1.7.3, they don't actually need Chef Zero to run.

[joelbernstein]

Thanks @sethvargo, that makes good sense. For now I've switched back to -solo but I'd rather be using -zero and will make time to rewrite the roles. Cheers.

[drags]

Hey @joelbernstein,

I had the same issue, but one thing I found for a quick fixup is:

• boot up the vagrant vm in solo mode and let it mount your chef.roles_path into /tmp/vagrant-<something>,
• in one shell in the Vagrant VM start the basic chef-zero server with just: chef-zero.
• In another shell navigate to the roles directory and run knife role -z from file *.rb.

This will cause knife to "upload" the roles to the local chef-zero server, and the result is that in whatever directory you run the knife role -z from file command in will have a new sub directory called roles/ with the JSON converted output of all of your roles.

Hope this helps
-T