How to write a reporter

How To Write a Reporter

PLEASE NOTE: Tap Out is still in early stages of development and the reporter API is likely to change some before all is said and done. Not really a big deal as the overall structure will remain the same. Just head up that you may need to spare a few minutes to update your custom reporter as we move toward a 1.0 release. Also note that this tutorial itself isn't quite complete yet either.

Writing a reporter is fairly straight forward. You are basically writing a listener class, base on an abstract base class that acts a kind of interface class to help you construct a reporter with some basic defaults and helper methods. It's a good idea to look at that code to get an idea of what you will be creating on top of it.

http://github.com/rubyworks/tapout/blob/master/lib/tapout/reporters/abstract.rb

Okay, so now lets start out by creating an empty subclass.

  require 'tapout/reporters/abstract'

  module TapOut

    module Reporters

      # My cool new TapOut reporter.
      class MyReporter < Abstract
        ...
      end

    end

  end

When the TapOut::Reporters::Abstract is inherited the subclass gets added to a list of reporter classes. This is how Tap Out tracks available reporters.

Now with out skeleton setup, lets add some methods to get it to actually do something. Let's start with the very first things that happens when a TAP-Y/J document stream is being consumed, we should receive a suite entry. This entry is passed to the #start_suite method. The methods is simply passed a single argument, a Hash object of the entry data. By looking at the specification for TAP-Y/J we see that the suite entry offers a few different fields. For the fun of it lest be very explicit about all of it.

      # Handle header.
      def start_suite(entry)
        puts "There are #{entry['count']} tests to be tested!"
        puts "The randomization seed is #{entry['seed']}." if entry['seed']
        puts "Testing began at exactly #{entry['start']}."
        puts "This document stream is using revision ##{entry['rev']} of the TAP-Y/J specification."
      end

That wasn't so hard was it? Let's knock out the next type, the case entry.

      # At the start of a new test case.
      def start_case(entry)
        puts "This is a #{entry['subtype']} kind of test case."
        puts "The case is called, '#{entry['label']}'."
        puts "And the case is at level #{entry['level']}."
      end

We might also get a note entry. We'll should output that as given.

      # Handle an arbitray note.
      def note(entry)
        puts "NOTE: #{entry['text']}"
      end

Just before a test is run, the #start_test method is called.

      # This is run before the status handlers.
      def start_test(entry)
      end

After a test is run, one of five status handlers will be called depending on the result of running the test. If the the test passed than the #pass method is called.

      # Handle test with pass status.
      def pass(entry)
        puts "Yeah! The #{entry['label']} test passed!"
        super(entry)
      end

Notice we called super here. The abstract class will keep a running list of each test for each status which we can use later. So that can ave us the trouble of doing it ourselves. It also makes the #tally_message helper method useful when we finally get to the end of the stream.

      # Handle test with fail status.
      def fail(entry)
        super(entry)
      end

      # Handle test with error status.
      def error(entry)
        super(entry)
      end

The next two status handlers are #omit and #skip. Now, you might wonder what the hell is the difference. It's fairly basic. A test is omitted by design. That is to say, there is a test but it was not actually run for some specific reason that the actual test runner determined. A good example might be if the test only applies to a particular platform.

      # Handle test with omit status.
      def omit(entry)
        puts "The #{entry['label']} test was omitted."
        super(entry)
      end

Skip on the other hand is a sort of place holder. There are pending tests, that for one reason or another weren't fully executed because they are not finished being implemented, but ultimately they need to be.

      # Handle test with skip or pending status.
      def skip(entry)
        puts "The #{entry['label']} test is pending completion."
        super(entry)
      end

After the status method is called, the #finish_test method is called.

      # This is run before the status handlers.
      def finish_test(entry)
      end

When a test case is completed the the #finish_case method gets called. The entry argument here is the same as it was for #start_case.

      # When a test case is complete.
      def finish_case(entry)
        puts "The #{entry['label']} case is complete!"
      end

Now to finish up completely when the stream runs out.

      # Handle footer.
      def finish_suite(entry)
        puts "Finished running tests in #{entry['time']} seconds."
        puts tally_message(entry)
      end

Notice that we use the #tally_message helper method to print out the total counts for each status. You can of course handle this yourself, but the helper is convenient for common usage.