rbenv must be tested

[bramswenson]

Personally I found working with rvm to be good until you upgraded rvm itself. It seemed every release would have odd quirks with failures and unpredictable exit codes (ie 0 on error). Personally I found rvm terribile to script against (think chef), as each release it seemed to change its exit codes and/or improperly handle failure. This among many other issues could be mostly eradicated with a proper test suite.

I realize that rbenv is a shell based application, but I strongly suggest cucumber/aruba features that focus on ensuring that rbenv returns proper exit codes and is scriptable, unix style. Simply ensuring the external command line interface behaves consistently across releases.

I would be happy to start on something like this if the approach is agreeable.

Thanks again for rbenv! It is much needed!

[sstephenson]

I'm absolutely in favor of testing the rbenv interface and establishing a scriptable API. Thanks for opening the discussion.

Implementation-wise, I'm not a fan of Cucumber-style tests, and I'd prefer to avoid any external dependencies. I suspect we can do just fine with a directory of shell scripts as test cases.

[bramswenson]

Sure. In fact, that is the path that rvm took as well. Custom assert_* style bash functions and such. I kinda felt like this was a barrier to contribution, but this is your rodeo. Either way I would love to help.

[josh]

Sam, you should actually give Test::Unit a shot. It works surprisingly well for unix commands, not so much for that stuff that require shell functions. But thats something we're trying to avoid.

[telemachus]

Something that I keep meaning to try on my own shell scripts is tap-functions. The idea is to give you TAP-style tests for shell scripts. That gives you a little more structure than just scripts to test scripts, but not very much in the way of dependencies. (You source one file to get the TAP-ish framework.)

[thinkerbot]

Here is a related pull request adding the test scaffold to run shell-based tests on multiple VMs, using assert_* style bash functions. I'm not sure how they relate to what rvm uses but this is something I've been working on for an separate project: #29

[ghost]

shunit2 might be worth looking at for this.

[bramswenson]

Cranking up a bunch of VM's seems a bit much for this. +1 for a shunit2

[thinkerbot]

I disagree about the VMs, although it depends on the intended scope of this project. Unix boxes differ in surprising ways. If this is to be for wide use, then I think it should be tested in as many environments as possible. If this is for just ubuntu + bash + 'one special environment' then who cares, do whatever is expedient for that one environment and warn everyone else off. I'm guessing this is intended for wide use, but maybe I'm wrong!

Setting up and testing on multiple VMs truly isn't that hard. Like any testing strategy it just takes a bit of forethought. Moreover I don't think the authors need to test on every VM, just make it simple to add new VMs so whoever cares about it can setup, test, and fix for their environment.

[ghost]

@thinkerbot I'm no expert when it comes to shell scripting, but from my experiences if you are following posix guidelines the scripts are going to be as portable as possible. Many Linux distributions have very strict requirements that any system shell scripts be fully POSIX compliant for this very reason. This also has the added benefit, of being more likely to work out of the box on BSD systems without additional work.

Problem is many people get used to bashisms as it's their default shell and do not check to be sure what they are doing is actually posix compliant. While bash is posix compliant it has many additions, as well as most modern shells. This can be eliminated by running bash --posix or if it's the default shell, which it is on many systems, running sh runs bash in posix mode. The man page has a lot of extra information.

I'm not positive if shunit2 is posix compliant or not, but if it is, running all the tests through dash would ensure compatibility one would think.

Running them through multiple shells would also have the same effect, but obviously going to take a lot longer. Still cheaper than firing up multiple VMs, and less setup.

You can find the docs on posix here. bashcritic may be of use as well.

_Update_ Dash is the default shell (/bin/sh) on Debian Lenny and up, and on Ubuntu starting at 6.10.

[thinkerbot]

@gregf thanks for your thoughtful response. I'm not an expert on shell testing either, but I have some recent experience and what is surprising to me is how tricky it can be to be POSIX compliant.

It's not impossible by any means, but you'd be surprised how many commands are not a part of POSIX and/or how many common options aren't a part of POSIX or even the Linux standard. And then there are bugs and occasional ambiguity. For example there is a debate whether dash is POSIX compliant because it doesn't set LINENO. You can confirm on ubuntu 11.04 with a script like this:

$ echo 'echo $LINENO' > example.sh
$ sh example.sh 

$ bash example.sh
1

Surprising right? I got bit by that when I was trying to add a debugging trace to some scripts. Of course something like this may not be important but the point is you just never know unless you test.

The reason why I like VMs is that you can reset them. I don't like testing shell stuff locally because if it goes wrong it can change something and suddenly it's hard to recover. With VMs, there's no such hassle. If it goes wrong you can reset to a base snapshot. You do need to plan where you take your snapshots but it's often not hard to make good decisions about that. Moreover the snapshots provide good references for identifying what is different between a test environment and what happens in the wild.

Regarding overhead, yeah there is some but it's manageable in my opinion. I know it's a lot less than I thought before I got into it. On a 2.66 GHz 4GB RAM macbook pro I can reset and start 3 VMs using VirtualBox in less than a minute with a rake task. Initial set up is maybe 30 min each... whatever the install disk takes. And if you're smart about your ssh settings, specifically if you setup a master socket, then the overhead to transfer and run your scripts remotely is totally manageable (~100ms).

As a practical reference, I can start 2 ubuntu VMs using the rbenv code at my pull request in ~43s, run the tests remotely in ~1s, and shutdown in ~1s. If you treat the initial setup as an occasional refresh and construct your tests so you can run them multiple times before a refresh then it really isn't too bad (IMO).

See here for the output from when I timed my pull request code: https://gist.github.com/1143394

[ghost]

@thinkerbot Not sure how to respond to your comments on the LSB really, as I'm afraid to start a flame war with certain people. So I'm just going to say as little as possible on it and hope for the best.

My way of avoiding using options that are not POSIX, is by using the BSD man pages, when I want to make a script portable. GNU, tends to sneak options in there that are not considered POSIX compatible, thus the whole issue with the LSB. I guess it's something I just learned to work around over the years.

I also agree it's no small task, learning the in's and out's, I just feel it's the only sane way of solving the problem. BSD/Linux developers have been dealing with the issue for many years, so I do see it as a responsible learning curve. I think we have to think of POSIX in the same way you would rubyspec, not every implementation of the ruby vm has passed rubyspec either. We learn to work around those problems though.

The bug you speak of in dash was fixed, I feel in a reasonable amount of time. Not sure how recent Ubuntu 11.04 is, but the bug in dash was fixed in October of 2010. Not that you don't have a valid point, anyone running 11.04 would have potentially hit a bug.

Don't get me wrong, I love VMs. I live in vagrant working with chef most of the day. For running a test suite, on some bash scripts, I just feel there has to be a better way. Not saying POSIX has to be the solution, I just think there can be a better one. I was also not trying to imply that running multiple VMs is really that heavy. Just in the comparison of not using them when we don't really have what I consider a real need for a full VM. I think people go directly to a VM these days and forget about all the other tools around like chroot for example.

I'm not sure the tests really have to stomp all over the file system either. Lets say for some reason they have to though, using something like tmpfs could write file system based tests in ram very fast and you have no tear down, even doing some work in /tmp might be a good solution with little to no tear down.

Not sure if any of this adds any value at this point, so I'm going to leave it at that. I commend you on actually taking the time to make a patch for the tests regardless. I was also surprised how fast it actually ran.

[ghost]

@thinkerbot I forget everyone is a mac user these days and you guys might not have tmpfs as a option. I did some searching for you in case anyone did want to look into using something similar. I found this blog post where a guy made a ram disk under OSX that would replicate a tmpfs setup.

_Update_ Better yet

[sstephenson]

The plan is to use Bats: https://github.com/sstephenson/bats

[bramswenson]

Fantastic tool. Thank you for yet another one. Also thanks to @telemachus for suggesting TAPS. Sam clearly heard you there. Been using rbenv since my initial post and have been very happy with it. Thanks again.

[bramswenson]

Didnt mean to close, but I suppose since I opened it...