Output formatters

[modocache]

Do not merge this yet. I'm issuing this pull request as an invitation for feedback.

One of the goals for issue #306 is to allow test custom test output. This pull request implements this feature in a similar way to how RSpec does it.

Feature Summary

Users may add a KiwiConfiguration.plist file to their test bundle. This plist may contain a formatters key, which refers to a dictionary of key-value pairs in the form formatter_name: output_file_path. For example:

<dict>
    <key>formatters</key>
    <dict>
        <!-- Output JSON to ~/Desktop/test-output.json -->
        <key>KWJSONFormatter</key>
        <string>/Users/modocache/Desktop/test-output.json</string>
        <!-- Output text (same format as Kiwi 2.0) to stderr (same as NSLog) -->
        <key>KWTextFormatter</key>
        <string>stderr</string>
    </dict>
</dict>

All formatter classes specified in the plist are registered to receive notifications regarding test execution via a new KWReporter singleton class. Users may specify any class, including ones they write themselves, so long as that class:

1. Is included in the test bundle
2. Conforms to the KWListener protocol

For example, specifying KWJSONFormatter: /Users/modocache/Desktop/test-output.json writes the following content to a file named test-output.json on the Desktop:

{
  "summary" : {
    "failure_count" : 2,
    "duration" : 0,
    "summary_line" : "5 examples, 2 failures, 1 pending",
    "example_count" : 5,
    "pending_count" : 1
  },
  "examples" : [
    {
      "result" : "passed",
      "description" : "something, does something"
    },
    {
      "result" : "passed",
      "description" : "something, does another thing"
    },
    {
      "result" : "failed",
      "failure_message" : "expected subject to equal (NSString) \"goodbye\", got (NSString) \"hello\"",
      "description" : "something, doesn't know how to say goodbye"
    },
    {
      "failure_message" : "NSInternalInconsistencyException \"oh wow such exception so halting\" raised",
      "exception" : {
        "call_stack_symbols" : [
          "0   CoreFoundation                      0x017505e4 __exceptionPreprocess + 180",
          "1   libobjc.A.dylib                     0x014bf8b6 objc_exception_throw + 44",
          "2   CoreFoundation                      0x017503bb +[NSException raise:format:] + 139",
          "3   KWTestAppTests                      0x08c822a4 __37+[KiwiTestAppSpec buildExampleGroups]_block_invoke50 + 68",
          "4   KWTestAppTests                      0x08c8f1e2 __25-[KWExample visitItNode:]_block_invoke + 114",
          // ...
        ],
        "name" : "NSInternalInconsistencyException",
        "reason" : "oh wow such exception so halting"
      },
      "result" : "failed",
      "description" : "something, raises an exception"
    },
    {
      "result" : "pending",
      "description" : "something, will soon do something else(PENDING)"
    }
  ],
  "messages" : [
    "This is an arbitrary message test writers may send to the reporter."
  ]
}

Implementation Summary

Here are the key players:

• KWReporter: A singleton which maintains an array of listeners. KWExample informs the reporter whenever it passes, fails, etc. The reporter then informs each of its listeners of these events.
• KWListener: A protocol defining how the reporter notifies listeners.
• KWFormatterLoader: A class responsible for loading the formatters from the plist. If the plist does not exist, KWTextFormatter: stderr is used.
• KWTextFormatter: A formatter class which outputs the exact same text as the NSLog statements in Kiwi 2.0 (minus the timestamps).
• KWJSONFormatter: A formatter class which outputs JSON.

The rest is in the diff. Sorry it's so huge. 😦

Specific Requests for Feedback

The API for Specifying Formatters

I don't like plists, but I chose to use one because I couldn't figure out a way to get access to arguments passed on the command line to xcodebuild. It would be much nicer to users if they could execute the following:

xcodebuild -project MyProject.xcodeproj -scheme MyProject -sdk iphonesimulator test \
           -formatters="KWTextFormatter:stderr KWJSONFormatter:/Users/modocache/Desktop/test-output.json"

Any thoughts on how to do this?

Printing to stderr

As far as I know, there is no way to "silence" XCTest output. Therefore, in order to ensure end users can actually parse the Kiwi formatter output, I've decided to redirect their output to a file. I'm not sure if this is the best way to do things. Ideally, I'd like to "silence" XCTest output and instead print the formatter output to stderr.

[modocache]

@mneorr I think the changes to the output might have broken XCPretty's parser! 😆

The tests pass when run with xcodebuild using the command in the Makeflle.

[modocache]

This should probably be considered in conjunction with issue #222.

[supermarin]

There are 2 prints for each test when Kiwi runs:

1. the one from XCTest
2. the one that Kiwi prints

I don't think you were able to affect the first one - have you tried piping it to xcpretty and testing results?
IIRC, we are parsing the standard xctest output.

[modocache]

The only difference is that Kiwi output no longer uses NSLog, and so does not include a timestamp.

Current Kiwi output:

2014-02-14 03:28:47.053 otest[92858:303] + 'nil matchers, when object is non-nil, passes a test for [[x shouldNot] beNil]' [PASSED]

Output as of this pull request:

+ 'nil matchers, when object is non-nil, passes a test for [[x shouldNot] beNil]' [PASSED]

I can investigate why this is the case by looking at XCPretty, but that might be more trouble than it's worth, since I'm anticipating that the output will change based on your and other's feedback anyway. If it's not too much trouble I'd like to get some feedback on the feature as a whole. You can rest assured that before it's merged it'll be playing along with XCPretty.

[modocache]

Oops, sorry, ignore that last post. Turns out it was an encoding issue. I switched the output to ASCII encoding and everything works for now.

[supermarin]

I'm pretty sure xcpretty would support UTF8 encoding as well (but your server needs to have the right locale set - xcpretty/xcpretty#48).

xcpretty doesn't read these NSLogs at all, it reads the output you don't see in Xcode console.
Try running the raw xcodebuild in your terminal and you'll see what I'm talking about :)

[modocache]

Ah, that'll do it then--the formatters in this PR print to stderr unless told otherwise.

Any thoughts on the PR?

[sharplet]

@modocache:

I don't like plists, but I chose to use one because I couldn't figure out a way to get access to arguments passed on the command line to xcodebuild. [...] Any thoughts on how to do this?

We should be able to do it with the shared NSUserDefaults:

// When Kiwi starts up, register the base defaults
[[NSUserDefaults standardUserDefaults] registerDefaults:@{
    @"formatters": @"KWTextFormatter:stderr"
}];

// If -formatters is passed from the command line, that argument will take
// precedence. If no other default is found (for example, from the user's
// global defaults), the registered value will be used.
[[NSUserDefaults standardUserDefaults] stringForKey:@"formatters"];

Check out the documentation on -[NSUserDefaults standardUserDefaults].

A couple of ideas:

• We could provide a KWUserDefaults class that encapsulates knowledge about Kiwi-specific defaults keys.
• In the vein of the -xxxForKey: methods on NSUserDefaults that do some parsing as well, we could provide a helper method on KWUserDefaults that does the work of parsing the -formatters argument out into a real data structure.

[modocache]

@sharplet Thanks for the suggestion, but unfortunately I've tried using -[NSUserDefaults standardUserDefaults] and -[NSProcessInfo environment], and neither seems to find any arguments I pass on the command line when running xcodebuild test.

I've seen people say command-line arguments are accessible via these two, but it's not working for me, so I'm not sure if perhaps I'm doing something wrong...

When running, I get the following output from xcodebuild:

User defaults from command line:
    formatters = KWJSONFormatter:stdout, KWTextFormatter:stdout

But the formatters key is (null) in both user defaults and process info. So I'm kind of stuck--you mentioned a similar approach in issue #222, but are you sure it works?

[modocache]

This was a neat idea, but I won't be working on it anytime soon for Kiwi. Still, building it was a ton of fun!! And I'm hoping to bake it into some sort of testing solution I make one day; perhaps Quick. Quick/Quick#9