Browse files

Added Cucumber/Rcov post

  • Loading branch information...
1 parent 9a5e6d1 commit 9352d2d8d794913f15032556c97499fc1d040719 @nickh committed Jan 15, 2012
Showing with 291 additions and 4 deletions.
  1. +13 −2 _layouts/default.html
  2. +198 −0 _posts/2012-01-13-cucumber-rcov-with-external-client.md
  3. +20 −2 css/blog.css
  4. +60 −0 css/syntax.css
View
15 _layouts/default.html
@@ -5,6 +5,7 @@
<title>{{ page.title }}</title>
<meta name="author" content="Nick Hengeveld" />
<link rel="stylesheet" href="/css/blog.css" type="text/css" media="screen, projection" />
+ <link rel="stylesheet" href="/css/syntax.css" type="text/css" media="screen, projection" />
</head>
<body>
@@ -18,14 +19,24 @@
<div class="dogs">
<h3>Dog Stuff</h3>
<ul>
+ <li>2012.01.11 - <a href="http://www.everytrail.com/view_trip.php?trip_id=1415354">Tracking with Fezzik</a></li>
+ <li>2012.01.11 - <a href="http://www.everytrail.com/view_trip.php?trip_id=1415335">Tracking with Maverick</a></li>
+ <li>2012.01.09 - Fezzik Tracking Class</li>
+ <li>2012.01.07 - <a href="http://www.everytrail.com/view_trip.php?trip_id=1415335">Tracking with Maverick</a></li>
<li>2011.12.22 - <a href="http://www.everytrail.com/view_trip.php?trip_id=1398119">Tracking with Fezzik</a></li>
<li>2011.12.15 - <a href="http://www.everytrail.com/view_trip.php?trip_id=1388577">Tracking with Maverick</a></li>
</ul>
</div>
<div class="reading">
- <h3>Reading</h3>
- <a href="http://www.amazon.com/Jesus-Interrupted-Revealing-Contradictions-ebook/dp/B001TKD4XA/ref=tmm_kin_title_0?ie=UTF8&m=AG56TWVU5XWC2&qid=1325011489&sr=8-1">Jesus Interrupted</a> - Bart D. Ehman
+ <h3>Currently Reading</h3>
+ <ul>
+ <li><a href="http://www.amazon.com/Inside-of-a-Dog-ebook/dp/B002NT3B52/ref=tmm_kin_title_0?ie=UTF8&m=AG56TWVU5XWC2&qid=1326652725&sr=8-1">Inside of a Dog</a> - Alexandra Horowitz</li>
+ </ul>
+ <h3>Just Finished</h3>
+ <ul>
+ <li><a href="http://www.amazon.com/Jesus-Interrupted-Revealing-Contradictions-ebook/dp/B001TKD4XA/ref=tmm_kin_title_0?ie=UTF8&m=AG56TWVU5XWC2&qid=1325011489&sr=8-1">Jesus Interrupted</a> - Bart D. Ehman</li>
+ </ul>
</div>
<div class="footer">
View
198 _posts/2012-01-13-cucumber-rcov-with-external-client.md
@@ -0,0 +1,198 @@
+---
+layout: post
+permalink: cucumber-rcov-with-external-client.html
+title: Cucumber Test Coverage With an External Client
+category: work
+tags: github
+---
+
+I'm currently working on a Sinatra app that emulates a Subversion HTTP
+server. Much of the automated test suite is driven by Cucumber, which
+sets up repository and checkout scenarios and exercises them using
+various `svn` commands. I recently wanted to get an idea of how well
+these Cucumber tests exercised the application, and this is how I got it
+working using rcov.
+
+Cucumber does have rcov integration, but expects to use a driver like
+Webrat or Capybara to handle HTTP requests. In that environment
+Cucumber is able to create an instance of your application and use mock
+request and response objects to communicate between it and the driver.
+
+Since I have cucumber calling out to `svn`, the application needs to be
+listening for connections on a port that the svn client can connect to.
+I previously had that working with a few changes to my `env.rb` file:
+
+{% highlight ruby %}
+`bin/rackup config.ru -D -P tmp/test_server.pid -p 5205`
+sleep(2)
+
+at_exit do
+ Process.kill('KILL', File.read('tmp/test_server.pid').to_i)
+end
+{% endhighlight %}
+
+The `-D` rackup parameter caused the app to start up in the background.
+After giving the server a couple of seconds to start, there would be a
+listener ready for the svn client to hit. The `at_exit` block would
+take care of shutting down the server when the test suite finished.
+This wasn't very elegant, but it got the job done. That is, until I
+tried to introduce rcov.
+
+rcov does its thing by loading a bunch of ruby code, tracking which
+lines are executed until it exits, and then reporting on what was and
+wasn't executed. `bin/rackup` is loadable ruby code, so I was able to
+make a small modification to get it working under rcov:
+
+{% highlight ruby %}
+`bin/rcov bin/rackup -- config.ru -D -P tmp/test_server.pid -p 5205`
+{% endhighlight %}
+
+I ran `rake features` and rcov happily wrote out coverage reports.
+However, the reports were all wrong - barely any of the application code
+was reported as covered, and I knew that much of it was definitely hit
+during the test run. The problem was the `-D` parameter; the code that
+rcov was tracking loaded the application classes, fired up a detached
+child to do the real work, and then quit. Turns out the reports had
+been written out before most of the tests had run.
+
+Fixing this was pretty straightforward; rather that relying on rack to
+run itself in the background, I moved that logic into `env.rb`:
+
+{% highlight ruby %}
+fork do
+ exec 'bin/rcov bin/rackup -- config.ru -P tmp/test_server.pid -p 5205'
+end
+sleep(2)
+
+at_exit do
+ Process.kill('KILL', File.read('tmp/test_server.pid').to_i)
+end
+{% endhighlight %}
+
+Success! But there was something that bothered me a bit. Cucumber has
+better integration with rcov that I wasn't using. Ideally, I would use
+a rake task instead of my cucumber environment setup to control rcov:
+
+{% highlight ruby %}
+Cucumber::Rake::Task.new(:features) do |t|
+ t.cucumber_opts = "--format pretty"
+ t.rcov = true
+end
+{% endhighlight %}
+
+I updated my rake task and changed the background rack app to run
+without rcov, and again things ran but the coverage reports were way
+off. This time, rcov had measured coverage in the parent process where
+all the cucumber things were happening and nothing in the child process
+where the app was running
+([rcov doesn't work with Kernel.fork](https://github.com/relevance/rcov/issues/77).)
+
+There was no reason that the cucumber things needed to happen in the
+parent, I just needed to switch things around so the rack app ran in the
+parent process and the testing continued in the child. This required a
+bit of research as I needed the rack app to start in the current process
+instead of a subshell. Fortunately, rackup is ruby so I was able to
+start the app in a more ruby-friendly way:
+
+{% highlight ruby %}
+Rack::Server.start(
+ :app => My::App.new,
+ :environment => 'none',
+ :Port => 5205,
+)
+{% endhighlight %}
+
+Tests ran successfully but the output was full of Webrick debug and
+access logs. Passing some extra options to `Rack::Server.start` took
+care of that:
+
+{% highlight ruby %}
+Rack::Server.start(
+ :app => My::App.new,
+ :environment => 'none',
+ :Port => 5205,
+ :Logger => Logger.new('/dev/null'),
+ :AccessLog => [],
+)
+{% endhighlight %}
+
+This time the tests ran and looked great, but I didn't get any coverage
+reports. rcov was politely waiting for the call to Rack::Server.start
+to return and it got SIGKILLed. Further research turned up a couple of
+interesting things: you can stop the server with a SIGINT, and the server
+has callbacks. I had never been happy with `sleep(2)` as a reliable way of
+waiting for the server to start; in fact it had been responsible for a
+few false negatives.
+
+Since the rcov/server and tests were running in separate processes, I
+needed a way for the callback in the server process to let the tester
+process know it was OK to proceed. I opted to use a pipe, where the
+tester process would wait on a read and the server would write when the
+callback triggered.
+
+Also, the SIGINT from the child was now causing a return from
+`Rack::Server.start` rather than killing the server process,
+so I had to add an explicit exit to prevent the server process from
+running all the tests again.
+
+Now the rcov coverage reports were correct, and the server was starting
+and stopping cleanly. There was only one subtle problem left. The exit
+status of the parent process is used to indicate whether the test suite
+passed, and it was always returning a 0 (pass) status even if a test
+failed:
+
+{% highlight bash %}
+$ bin/cucumber features/log.feature
+Scenario: Log # features/empty_repo.feature:45
+ When I run svn log in the root of my checkout # features/step_definitions/log.rb:13
+ Then I get the error "you suck" # features/step_definitions/repo.rb:331
+ Expected the svn command to fail, but it succeeded (RuntimeError)
+ ./features/step_definitions/repo.rb:332:in `/^I get the error "([^"]*)"$/'
+ features/empty_repo.feature:47:in `Then I get the error "you suck"'
+
+Failing Scenarios:
+cucumber features/empty_repo.feature:45 # Scenario: Log
+
+5 scenarios (1 failed, 4 passed)
+29 steps (1 failed, 1 skipped, 27 passed)
+0m12.935s
+
+$ echo $?
+0
+{% endhighlight %}
+
+This was a simple fix; the parent/server process just needed to
+relay the child/tester exit status. My final `env.rb` looks like this:
+
+{% highlight ruby %}
+rd, wr = IO.pipe
+server_pid = $$
+tester_pid = fork
+if tester_pid.nil?
+ wr.close
+ rd.read
+ rd.close
+
+ at_exit do
+ Process.kill('INT', server_pid)
+ end
+else
+ rd.close
+
+ Rack::Server.start(
+ :app => My::App.new,
+ :environment => 'none',
+ :Port => 5205,
+ :Logger => Logger.new('/dev/null'),
+ :AccessLog => [],
+ :StartCallback => lambda { wr.write('started'); wr.close }
+ )
+
+ Process.waitpid(tester_pid, 0)
+ exit $?.exitstatus
+end
+{% endhighlight %}
+
+I'm pretty happy with the way this ended up. I can use the existing
+Cucumber/rcov integration and I learned some interesting details
+about Rack servers. Now to work on the missing coverage...
View
22 css/blog.css
@@ -13,8 +13,8 @@ h1,h2 > a {
div.blog {
width: 100%;
padding-top: 30px;
- background: -webkit-linear-gradient(top, #bbb, white);
- background: -moz-linear-gradient(top, #bbb, white);
+ background: -webkit-linear-gradient(top, #bbb, white 150px);
+ background: -moz-linear-gradient(top, #bbb, white 150px);
}
div.blog > div.overview {
float: left;
@@ -37,6 +37,14 @@ div.blog > div.overview > div.dogs > ul {
padding-left: 0px;
list-style-type: none;
}
+div.blog > div.overview > div.reading > h4 {
+ margin: 0px;
+}
+div.blog > div.overview > div.reading > ul {
+ list-style-type: none;
+ margin: 3px 0px;
+ padding-left: 5px;
+}
div.blog > div.overview > div.footer {
padding-top: 20px;
font-size: 0.9em;
@@ -46,6 +54,9 @@ div.blog > div.posts {
margin-left: 260px;
width: 640px;
}
+div.blog > div.posts > div.post {
+ margin-bottom: 15px;
+}
div.blog > div.posts > div.post > div.header {
}
div.blog > div.posts > div.post > div.header > a {
@@ -62,3 +73,10 @@ div.blog > div.posts > div.post > div.content > p {
div.blog > div.posts > div.teaser > div.content > p {
display: inline;
}
+div.blog > div.posts > div.post > div.content > div.highlight {
+ background: #ddd;
+ border: 1px solid #aaa;
+ margin: 20px;
+ padding: 10px;
+ overflow: scroll;
+}
View
60 css/syntax.css
@@ -0,0 +1,60 @@
+.highlight { background: #ffffff; }
+.highlight .c { color: #999988; font-style: italic } /* Comment */
+.highlight .err { color: #a61717; background-color: #e3d2d2 } /* Error */
+.highlight .k { font-weight: bold } /* Keyword */
+.highlight .o { font-weight: bold } /* Operator */
+.highlight .cm { color: #999988; font-style: italic } /* Comment.Multiline */
+.highlight .cp { color: #999999; font-weight: bold } /* Comment.Preproc */
+.highlight .c1 { color: #999988; font-style: italic } /* Comment.Single */
+.highlight .cs { color: #999999; font-weight: bold; font-style: italic } /* Comment.Special */
+.highlight .gd { color: #000000; background-color: #ffdddd } /* Generic.Deleted */
+.highlight .gd .x { color: #000000; background-color: #ffaaaa } /* Generic.Deleted.Specific */
+.highlight .ge { font-style: italic } /* Generic.Emph */
+.highlight .gr { color: #aa0000 } /* Generic.Error */
+.highlight .gh { color: #999999 } /* Generic.Heading */
+.highlight .gi { color: #000000; background-color: #ddffdd } /* Generic.Inserted */
+.highlight .gi .x { color: #000000; background-color: #aaffaa } /* Generic.Inserted.Specific */
+.highlight .go { color: #888888 } /* Generic.Output */
+.highlight .gp { color: #555555 } /* Generic.Prompt */
+.highlight .gs { font-weight: bold } /* Generic.Strong */
+.highlight .gu { color: #aaaaaa } /* Generic.Subheading */
+.highlight .gt { color: #aa0000 } /* Generic.Traceback */
+.highlight .kc { font-weight: bold } /* Keyword.Constant */
+.highlight .kd { font-weight: bold } /* Keyword.Declaration */
+.highlight .kp { font-weight: bold } /* Keyword.Pseudo */
+.highlight .kr { font-weight: bold } /* Keyword.Reserved */
+.highlight .kt { color: #445588; font-weight: bold } /* Keyword.Type */
+.highlight .m { color: #009999 } /* Literal.Number */
+.highlight .s { color: #d14 } /* Literal.String */
+.highlight .na { color: #008080 } /* Name.Attribute */
+.highlight .nb { color: #0086B3 } /* Name.Builtin */
+.highlight .nc { color: #445588; font-weight: bold } /* Name.Class */
+.highlight .no { color: #008080 } /* Name.Constant */
+.highlight .ni { color: #800080 } /* Name.Entity */
+.highlight .ne { color: #990000; font-weight: bold } /* Name.Exception */
+.highlight .nf { color: #990000; font-weight: bold } /* Name.Function */
+.highlight .nn { color: #555555 } /* Name.Namespace */
+.highlight .nt { color: #000080 } /* Name.Tag */
+.highlight .nv { color: #008080 } /* Name.Variable */
+.highlight .ow { font-weight: bold } /* Operator.Word */
+.highlight .w { color: #bbbbbb } /* Text.Whitespace */
+.highlight .mf { color: #009999 } /* Literal.Number.Float */
+.highlight .mh { color: #009999 } /* Literal.Number.Hex */
+.highlight .mi { color: #009999 } /* Literal.Number.Integer */
+.highlight .mo { color: #009999 } /* Literal.Number.Oct */
+.highlight .sb { color: #d14 } /* Literal.String.Backtick */
+.highlight .sc { color: #d14 } /* Literal.String.Char */
+.highlight .sd { color: #d14 } /* Literal.String.Doc */
+.highlight .s2 { color: #d14 } /* Literal.String.Double */
+.highlight .se { color: #d14 } /* Literal.String.Escape */
+.highlight .sh { color: #d14 } /* Literal.String.Heredoc */
+.highlight .si { color: #d14 } /* Literal.String.Interpol */
+.highlight .sx { color: #d14 } /* Literal.String.Other */
+.highlight .sr { color: #009926 } /* Literal.String.Regex */
+.highlight .s1 { color: #d14 } /* Literal.String.Single */
+.highlight .ss { color: #990073 } /* Literal.String.Symbol */
+.highlight .bp { color: #999999 } /* Name.Builtin.Pseudo */
+.highlight .vc { color: #008080 } /* Name.Variable.Class */
+.highlight .vg { color: #008080 } /* Name.Variable.Global */
+.highlight .vi { color: #008080 } /* Name.Variable.Instance */
+.highlight .il { color: #009999 } /* Literal.Number.Integer.Long */

0 comments on commit 9352d2d

Please sign in to comment.