How does ChefDK interoperate (or break) normal ruby gem workflows?

[rberger]

As a long time user of Chef, I am finding that installing ChefDK a very scary prospect. This may be due to some lack of documentation.

• How is this not going to cause the same pain and grief as when Vagrant when from Gems to package?
• How does this interact with normal bundle install in cookbooks, or other app development that might use gems that are associated with chef, berkshelf, etc.?
• How does it interact with (or break) rvm/rbenv workflows?

An indication of this confusion is that Berkshelf says to put /opt/chefdk/embedded/bin in your $PATH

[rberger]

Also rvm info complains. Obviously its using the ruby in /opt/chefdk and not rvm's if I put /opt/chefdk/embedded/bin at the head of $PATH:

Warning! PATH is not properly set up, '/Users/rberger/.rvm/gems/ruby-2.1.0/bin' is not at first place,
         usually this is caused by shell initialization files - check them for 'PATH=...' entries,
         it might also help to re-add RVM to your dotfiles: 'rvm get stable --auto-dotfiles',
         to fix temporarily in this shell session run: 'rvm use ruby-2.1.0'.

[danielsdeleo]

Hi @rberger

We'll add some documentation to address this. But to briefly respond:

How is this not going to cause the same pain and grief as when Vagrant when from Gems to package?

For one, Vagrant dropped all support for other installation mechanisms as part of that switch, so if it broke your workflow, you had no way to get new versions of vagrant until all of the libraries, plugins, and other parts of your workflow were updated also. All of the projects in ChefDK can still be installed the old way, no one is forcing you to switch if you can't.

How does this interact with normal bundle install in cookbooks, or other app development that might use gems that are associated with chef, berkshelf, etc.?

There's two ways to use and think about ChefDK. One, it's a set of applications that happen to ship with all of their dependencies and install to an isolated location on your system. If this is how you choose to use ChefDK, then you can use chef/knife, berks and test-kitchen that come bundled with ChefDK alongside your other development environment. Any TK drivers, knife plugins, etc. you need, install them with chef gem and other ruby tools you install normally. If you are already familiar with ruby, have tools that require a particular ruby version, etc., this might be best for you. I'll also note that this is what we've optimized thus far.

The other way to use/think about ChefDK is that it provides a working ruby development environment out of the box. In this case, you and your team members would not use rvm/rbenv to get a working base ruby, you just use the one that comes with ChefDK. To do this currently, you need to add /opt/chefdk/embedded/bin to your PATH. If you do this, the applications that are bundled with ChefDK, like knife, berks, and kitchen are protected from gem conflicts and the like by the appbundler system (which also ensures that the command line applications perform well even as you install more gems), but the same is not true for other ruby applications you might install. This is a use case that some people want ChefDK to support and we think it's reasonable, but it will probably remain opt-in. We are looking at some features to make it easier, like having a chef subcommand to initialize your shell so you could place chef init-shell bash (or something like that) in your .bashrc and have the necessary components added to your path.

How does it interact with (or break) rvm/rbenv workflows?

First of all, ChefDK uses the appbundler system to make rvm-safe binaries, so if you use ChefDK in the first usage scenario described above, everything will work just fine. As to how much your workflow is affected, this is up to you. If you want to just use chef/knife, berks, and kitchen as provided by ChefDK, then you just need to make sure that ChefDK's copy of those tools is first in your path and you can use other ruby tools as normal. If you want to use ChefDK as your primary ruby environment, then you would probably uninstall rvm/rbenv and use ChefDK's ruby environment instead.

[danielsdeleo]

3c00b83 Updated the README with some of this information (note that it refers to the code on master which isn't yet released).

[geewiz]

As soon as I prepend my PATH with /opt/chefdk/embedded/bin, the shell will use the gem command that comes with the DK. Installing new non-Chef gems then will either fail (because Mac OS DK doesn't install to the home dir yet) or create a complete mess of gems installed into the DK environment instead of my rbenv one.

[danielsdeleo]

@geewiz as the read me notes, you can either just use ChefDK to run chef/knife/berks/kitchen alongside your existing ruby dev environment or as a replacement for your dev environment. There's not really a middle ground. Tl;dr, don't add /opt/chefdk/embedded/bin to your PATH if you want to keep using rvm.

[petracvv]

@geewiz If you prepend /opt/chefdk/bin to your PATH it will do what you want. That directory has only the chef related commands in it. While the chefdk install does create correct symlinks in /usr/bin to /opt/chefdk, if you have rvm those are overwritten by the rvm versions as rvm puts itself at the front of the PATH.

[ringods]

@danielsdeleo I would rather see some investigation to integrate the Chef-DK embedded ruby into rbenv/rvm similar to how a system ruby is handled: does rvm mount ring a bell?

https://github.com/wayneeseguin/rvm/blob/master/help/mount.md

You can't uninstall the system ruby, but you can add global gems and specific gemsets to the setup. Installing 'global' gems could still be done in the users specific gem folder. A good mix of your appbundler setup and rvm features.

rbenv and rvm are already very well supported by a plethora of other tools. Having a shell wrapper that sets environment variables correct is much harder to get correct in other automation cases. E.g. how to get your suggestion for chef init-shell bash combined with the Chef script provider?

[danielsdeleo]

@ringods Firstly, a huge portion of our community doesn't really want to deal with a full ruby development environment with support for multiple ruby versions, etc. So ChefDK is kinda like a ruby version manager that only supports one version of ruby for that use case.

As for running scripts with ChefDK environment variables, that's what chef exec is for. This should be fine for one-offs or automation/scripts that you only write once, but obviously it's annoying to type it all the time, so chef shell-init gives you a convenient way to make that a permanent part of your environment. Though this feature is aimed at people who want ChefDK to provide their one and only ruby development environment.

Also, you say "rbenv and rvm are already very well supported by a plethora of other tools." What kind of features or integration are you missing? I use rbenv for day-to-day development work, but I don't integrate it with any other tools, I just pick which ruby I want from the command line and that's it.

[Toady00]

@ringods Something thats worked really well for me so far is https://github.com/zimbatm/direnv . I'm primarily a rails dev, but I also manage 20+ servers with chef. So I use both a lot. I have rvm installed just as http://rvm.io suggests. All my code for work is in ~/code/work, all my personal code is in ~/code/personal, and all my chef code is in ~/code/chef. Using direnv, I have a ~/code/chef/.envrc file that looks like this:

PATH_add /opt/chefdk/embedded/bin
PATH_add /opt/chefdk/bin

When I cd into ~/code/chef or any subdir, direnv prepends these paths to my $PATH, and as soon as I cd out of that directory, direnv removes those from my path. So rvm takes control when I'm not using chef, and chef takes control when I am using chef.

And if you are worried about performance, direnv is written in go, and has been completely unnoticeable so far. Hope this was helpful.

[wickett]

@Toady00 this is really great

@rberger does this answer what you needed?

[lamont-granquist]

FWIW, I just 'rvm use system' and make rvm go away when i want to use chefdk. Based on a little testing it looks like rvm supports 'system' being in .ruby-version of a project as well.

[rberger]

@wickett The combination of the direnv and setting my GEM_HOME to
export GEM_HOME="/Users/rberger/.chefdk/gem/ruby/2.1.0" gets me pretty close to there. I have to also take ~/.rvm out of my path. So my direnv .envrc file looks like:

export PATH=/opt/chefdk/bin:/Users/rberger/.chefdk/gem/ruby/2.1.0/bin:/opt/chefdk/embedded/bin:/Users/rberger/anaconda/bin:/Users/rberger/bin:/usr/local/bin:/usr/local/sbin:/usr/local/share/npm/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/usr/local/MacGPG2/bin:/usr/texbin:/usr/local/opt/go/libexec/bin
export GEM_ROOT="/opt/chefdk/embedded/lib/ruby/gems/2.1.0"
export GEM_HOME="/Users/rberger/.chefdk/gem/ruby/2.1.0"
export GEM_PATH=/Users/rberger/.chefdk/gem/ruby/2.1.0:/opt/chefdk/embedded/lib/ruby/gems/2.1.0

The only thing is I seem to have to now type bundle exec while I'm in this direnv where as in my normal rvm environment I don't have.

All in all I still feel dirty using ChefDK. But I'm back to being productive and using it for all Chef work.

[lamont-granquist]

rvm use system will remove the path, it also removes all the gem environment variables. In the general case you'd need to eliminate all these variables at least in order to be certain that your .envrc was completely cleaning your environment:

https://github.com/opscode/omnibus/blob/cf4b18fb94f724956b28592cde0890fe42cf6fbe/lib/omnibus/builder.rb#L747-L757

[rberger]

@lamont-granquist Interesting idea.
But it looks like if I put system in .ruby-version in the same directory as .envrc, everything works fine when I cd into that directory, but when I cd out, I have lost my normal rvm path.

I.E. my PATH normally starts with:
/Users/rberger/.rvm/gems/ruby-2.1.2/bin:/Users/rberger/.rvm/gems/ruby-2.1.2@global/bin:/Users/rberger/.rvm/rubies/ruby-2.1.2/bin:

But after I put system in .ruby-version, and I cd out of the directory with the .ruby-version and .envrc, my rvm rich path items aren't put back in the front of the PATH and instead I have /Users/rberger/.rvm/bin at the end of my PATH.

So ether I'm doing it wrong or its not doing what I expect.

[thomas-riccardi]

When using rvm and chefdk with chefdk just for /opt/chefdk/bin executables (so not as a ruby development environment), I get an error when runnnig foodcritic:

$ foodcritic 
/opt/chefdk/embedded/lib/ruby/site_ruby/2.1.0/rubygems/dependency.rb:313:in `to_specs': Could not find 'foodcritic' (>= 0) among 38 total gem(s) (Gem::LoadError)                                                                            
Checked in 'GEM_PATH=/home/xxx/.rvm/gems/ruby-1.9.3-p547:/home/xxx/.rvm/gems/ruby-1.9.3-p547@global', execute `gem env` for more information                                                                                       
        from /opt/chefdk/embedded/lib/ruby/site_ruby/2.1.0/rubygems/dependency.rb:322:in `to_spec'
        from /opt/chefdk/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_gem.rb:58:in `gem'

This seems to be in contradiction with what was said here and on the README: chefdk as I understand it should create its own totally isolated workspace at least when running /opt/chefdk/bin executable, shouldn't it?

The current workaround is to execute rvm use system before using chefdk executables. This should not be necessary. /opt/chefdk/bin executable should probably clear the GEM_* environment variables before execution.

[lamont-granquist]

foodcritic isn't 'appbundled' so its not getting the environment cleaning and you need to either use 'rvm use system' or 'chef exec foodcritic'

this is kind of an unactionable catch-all bikeshed of a thread, so you probably should open a concrete issue against omnibus-chef to add appbunde'ing to foodcritic: https://github.com/opscode/omnibus-chef/issues/new

[thomas-riccardi]

@lamont-granquist it seems to have already been fixed by PR chef-boneyard/omnibus-chef#254.
Thank you.

[mrjcleaver]

http://lists.opscode.com/sympa/arc/chef/2014-11/msg00338.html and https://dwradcliffe.com/2014/09/19/chefdk-with-rbenv.html bring this up too.

[martinb3]

a huge portion of our community doesn't really want to deal with a full ruby development environment with support for multiple ruby versions, etc.

Just want to chime in and say there is also a large part of the community that does want ruby to work outside of chef-dk's installation. Do we have any real data on this point, since it is driving some design decisions that are disruptive to the user base?

[lamont-granquist]

So we have a hard requirement which is not up for debate to support ruby naive users who have little ruby experience, do not use rvm/chruby/rbenv/etc and want to use chef-dk as a development environment. It should be obvious why we have that requirement and why it isn't up for debate.

The question here is if anyone who uses chruby or rbenv has a concrete productive change to apply to chef-dk to make it easier to work with that environment. So far I haven't read one. There's this strawman argument which has been constructed that fixing chruby/rbenv isn't being done because we want to support the ruby naive users, and I don't see the inevitable connection there. In fact it looks more like the people we'd hose are the people who want to run system ruby side-by-side with omnibus ruby packages that would get more hosed. It seems like we could eliminate the distinction between /opt/chef/bin and /opt/chef/embedded/bin and make omnibus more aggressive about taking over the role of your system ruby and that would actually be better for chruby. And the /opt/chef/bin and /opt/chef/embedded/bin distinction was made LONG before we came up with the ruby-naive-user use case, and were focused more on not stomping on the role of /usr/bin/ruby.

So, instead of arguing about wishy-washy use cases, make a concrete technical proposal, bonus points if there's a real pull request we can look at, and we can debate technical issues.

And like I posted before I use rvm and because of that it injects all kinds of awful gem variables into my environment which there's very little hope of getting chefdk to play nice with, so I just 'rvm use system' to make rvm go away and it works fine for me. So I use multiple version ruby switching on my laptop with chef-dk and don't have a problem with that mechanism. I can't design what we need for chruby/rbenv support because I don't have years of experience with those switchers and don't understand what they need.

So, nobody is saying that we can't support your use case, but nobody has presented any concrete ideas about how to fix it.

[danielsdeleo]

Likewise, I use rbenv, and I just run the command from the README to initialize a shell session for ChefDK and then throw away that session when I'm done ( this one: eval "$(chef shell-init SHELL_NAME)"). It puts all the ChefDK stuff in front of the rbenv stuff in my PATH, and works.

[mrjcleaver]

My concern (see https://dwradcliffe.com/2014/09/19/chefdk-with-rbenv.html) was that gem executables were not working. I confess I know just enough Ruby for Chef, but also do some small time Ruby coding for other purposes. I switched from rvm recently, and am generally happier on chruby.

This is not a pull request, but (per http://lists.opscode.com/sympa/arc/chef/2014-11/msg00338.html) @dawilliams kindly shared some code: https://gist.github.com/dawilliams/1674b7d7744d95c1b781

[lamont-granquist]

Right but that's a patch to chruby, so again its not clear what we need to do here to support the use case.

If that's the solution then it clearly fine, but there's nothing to merge in our repos, and the fix seems to lie in a patch to chruby itself.

And the claim is made that we're "driving some design decisions that are disruptive to the user base" and I'm still not seeing where that is. We still have nothing concrete that the chruby-using chef community needs from patches to this repo (or chef, or omnibus-chef, appbundler, etc).

[rberger]

I should chime in and say that ever since #160 (Set GEM_HOME to Gem.user_dir instead of Gem.paths.home) and I've been using direnv in all my cookbook repos that I want to use Chef-DK with,

I have been very happily using Chef-DK for all my chef development while in chef cookbooks with an .envrc file and then doing regular Ruby development with rvm in all other directories with no problems.

Note that direnv runs the envrc file as bash even if you are using zsh. So that chef shell-init should use bash in the .envrc even if you are using zsh (should be true with other shells, but only tested with zsh)

Here is what I use as an .envrc that will set your PATH while in the direnv'd directory with no rvm directories in it and with chefdk paths at the front of the final PATH the chef shell-init makes sure the GEM* Variables are set right as well.

eval "$(chef shell-init bash)"
# Update path to have chefdk first and remove any rvm directories from your original PATH
# directory removal hint from http://stackoverflow.com/a/370255/38841
export PATH=$(IFS=:; t=($PATH); unset IFS; t=(${t[@]%%*rvm*}); IFS=:; echo "${t[*]}";)

[kwilczynski]

I also use rbenv and I've been doing what @danielsdeleo is doing (see his comment above), and had no issues with any clashes, etc. In some cases, I also have direnv added, also no problem there.

[danjellesma]

Seconding @martinb3 on this topic. I use RVM and having ChefDK (latest) bundle Ruby with it's plugins, like Berks, have made developing cookbooks more difficult than I think it needs to be. I'd like to think anyone who is at the point of using Chef / configuration management can handle installing Ruby on their own, and probably already have it. Direnv seems like a bandaid/workaround for what would be standard usage. Am I missing something obvious?

Vagrant throws up when trying to use any of the Chef plugins (again, Berks) when using RVM.

/Users/dan/.rvm/gems/ruby-2.1.2@chef/bin/berks --version --format json
Could not find 'berkshelf' (>= 0) among 127 total gem(s) (Gem::LoadError). 

It appears that you are not using the ChefDK. Please note that Vagrant Berkshelf
works best when used with the ChefDK, and other installation methods are not
officially supported.

[lamont-granquist]

So, again, concretely, what is the proposed fix to this repo? I don't know if i'm the one missing something obvious but how do you expect a non-rvm ruby to interact with an rvm ruby? That seems like you're inherently crossing the streams -- what do we do about it?

[martinb3]

This doesn't address @lamont-granquist, but it might give the other commenters something to look over: https://github.com/martinb3/chefdk-demo

I added test suites for rvm and rbenv, and serverspec tests demonstrating how to switch back and forth from rbenv rubies and chef-dk as well as from rvm rubies and chef-dk.

[danielsdeleo]

I'd like to think anyone who is at the point of using Chef / configuration management can handle installing Ruby on their own, and probably already have it.

There's a very large contingent of users who don't care about Ruby development, etc. So without something like ChefDK, their experience is: install whatever ruby their system has, find out it's the wrong version (the hard way), google around, find out they should use rvm or whatever, install that, and then have no idea what to do when it breaks, because they don't know how to correlate a seemingly unrelated error with the fact that some GEM_* is set wrong, because they don't have years of experience developing ruby. Going through that whole process when you just wanted to configure NTP on your servers frankly sucks. I've heard this exact story from a ton of people who were extremely frustrated with using Chef before ChefDK came along. I think we tend to forget what a struggle it was to learn things like debugging ruby version managers, debugging gem compilation/install issues, etc., once we have that skill.

Now, you probably have a good reason why the complexity of rvm is worth it for you. I understand, because I have the same needs and I use a similar tool to address them. That said, there's a limit to what ChefDK can do to accommodate you. You're installing other versions of the tools that ChefDK packages and putting them first in your PATH (or GEM_PATH or whatever). There's nothing ChefDK can do about this, just as there's no way a C executable can work if you set your LD_LIBRARY_PATH to something broken.

As for the exact error you're reporting:

Vagrant throws up when trying to use any of the Chef plugins (again, Berks) when using RVM.

Nothing in your error seems to indicate any problem with ChefDK. Presumably you're running vagrant up, it's trying to load some plugin, and it's looking in the wrong place because of how your environment is configured.

[ringods]

Since my comment early August (see higher), I have been testing a lot of scenarios. Eventually, I just settled with Chef-DK and stripping some gems out of my Gemfile:

• chef
• berkshelf
• test-kitchen

All the other Ruby needing tools are running under RVM.

[docwhat]

Well, how about working with rvm, rbenv, etc. to allow switching to chefdk? E.g. rvm use chefdk-1.2

[lamont-granquist]

I just use rvm use system to do that. I have no idea how to patch it in so that rvm 'knows' about an external ruby that rvm itself didn't install -- are there any other examples of integrating rvm like that?

[ringods]

Not that I would need it, but doesn't rvm mount support this?

https://github.com/wayneeseguin/rvm/blob/master/help/mount.md

[daxgames]

I use rvm and use the following to activate ChefDK as my default ruby for all shells.

In my profile setup

eval "$(chef shell-init `basename $SHELL`)"

# Load RVM into a shell session *as a function
[[ -s "$HOME/.rvm/scripts/rvm" ]] && source "$HOME/.rvm/scripts/rvm"

Set rvm default ruby to system one time.

rvm --default use system

This opens all subsequent shells using ChefDK. My final env contains the following:

GEM_ROOT=/opt/chefdk/embedded/lib/ruby/gems/2.1.0
PATH=/opt/chefdk/bin:/Users/daxgames/.chefdk/gem/ruby/2.1.0/bin:/opt/chefdk/embedded/bin:.:/Users/daxgames/bin:/opt/chefdk/bin:/Users/daxgames/.chefdk/gem/ruby/2.1.0/bin:/opt/chefdk/embedded/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:.:/Users/daxgames/bin:/opt/chefdk/bin:/Users/daxgames/.chefdk/gem/ruby/2.1.0/bin:/opt/chefdk/embedded/bin:/Users/daxgames/.yadr/bin:/Users/daxgames/.yadr/bin/yadr:/Users/daxgames/.rvm/bin
GEM_HOME=/Users/daxgames/.chefdk/gem/ruby/2.1.0
GEM_PATH=/Users/daxgames/.chefdk/gem/ruby/2.1.0:/opt/chefdk/embedded/lib/ruby/gems/2.1.0

To switch back to an rvm installed Ruby I do the following:

rvm use ruby-1.9.3-p545

My environment now contains:

GEM_ROOT=/opt/chefdk/embedded/lib/ruby/gems/2.1.0
PATH=/Users/daxgames/.rvm/gems/ruby-1.9.3-p545/bin:/Users/daxgames/.rvm/gems/ruby-1.9.3-p545@global/bin:/Users/daxgames/.rvm/rubies/ruby-1.9.3-p545/bin:/Users/daxgames/.rvm/bin:/opt/chefdk/bin:/Users/daxgames/.chefdk/gem/ruby/2.1.0/bin:/opt/chefdk/embedded/bin:/opt/chefdk/bin:/Users/daxgames/.chefdk/gem/ruby/2.1.0/bin:/opt/chefdk/embedded/bin:/opt/chefdk/bin:/Users/daxgames/.chefdk/gem/ruby/2.1.0/bin:/opt/chefdk/embedded/bin:.:/Users/daxgames/bin:/opt/chefdk/bin:/Users/daxgames/.chefdk/gem/ruby/2.1.0/bin:/opt/chefdk/embedded/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:.:/Users/daxgames/bin:/opt/chefdk/bin:/Users/daxgames/.chefdk/gem/ruby/2.1.0/bin:/opt/chefdk/embedded/bin:/Users/daxgames/.yadr/bin:/Users/daxgames/.yadr/bin/yadr

To switch back to an ChefDK I throw away the current shell and open another or do the following:

rvm use system ; eval "$(chef shell-init `basename $SHELL`)"

Executing 'gem list -d' from either env setup does not show any gem install overlaps. I can work with cookbooks that work either with ChefDK under the default env setup or the same cookbook with rvm ruby 1.9.3-p545 and bundle exec using Gemfile/Gemfile.lock files.

Seems to work for me. Anyone see any issues with this approach?

I am wondering if 'rbenv global system' might produce similar results for rbenv users but I don't use rbenv. Can someone else test it?

[docwhat]

For those that use rbenv, I have a solution which seems to work for me at the moment: an rbenv plugin that combined with ln -nsf /opt/chefdk/embedded/ $RBENV_ROOT/versions/chefdk makes everything work.