Add support for writing the tests output in JSON format

[NSCoder]

Add support for writing the result of the tests to a json file, similar to the XUnit feature

Motivation:

I would like to be able to parse the result of the tests in other commands/programs without to have an elaborated regular expression parsing. For my needs specifically, I would love to be able to read the results using elisp and present them in a nice buffer with options such as retry.

Modifications:

Added a very similar implementation as how XUnit file generation is implemented. I didn't want to do additional modifications in order to keep the changes as minimum as possible but in the future we could abstract this functionality to its own struct/file/module and generalize it to support mode encodings.

One particularity is that I didn't used codable for the implementation given that I noticed that we have our own custom JSON support. I imagined that this has a bigger reason (maybe not wanting to import foundation), so I decided to follow suit.

That said, if I were able to made additional changes I would have loved to implement a --output option so we could simplify the call site read like: `swift test --parallel -output json | jq"

Result:

The swift test command has a new option --json-output that receives an absolute path where the result of the tests will be written encoded in JSON format.

I'm new to the source code and the repository and I'm very flexible to move things around, or change the implementation details. Please let me know! I would love to be able to merge this at some point.

[tomerd]

this flag should be mutually exclusive with with xUnitOutput?

[MaxDesiatov]

would a better alternative be to introduce a single option through which users can specify test output format instead of introducing flags for every single format?

[NSCoder]

Good questions, I don't see a reason why they have to be mutually exclusive besides modifying the options from the command to only allow one output.

@MaxDesiatov I was thinking about this as well, but giving that I'm not familiar with the project I decided to keep this patch it as simple as possible. Would it be all right if we try to add the output option refactor in a following up patch?

[tomerd]

yes, I think it is fine for now

[tomerd]

thanks @NSCoder this is a good addition. couple of ideas / comments:

1. JSONSerializable is an older convention before Foundation was stable. At this point we should not use any more, so lets create little Encodable helpers here
2. Is there a some standard for test results in the JSON format, ie something like XUnit? if there was one, it would probably be better than to roll our own format

[NSCoder]

Thank you for taking a look @tomerd! The feedback makes sense, I will change the JSON Encoding to use codable. About the json output format. I did some quick research and it seems that there is not a JSON based standard, perhaps what we could do is just return the same XUnit structure but in JSON:

{
  "name": "SwiftTest",
  "tests": 2,
  "failures": 1,
  "time": 2,
  "testsuite": [
    {
      "name": "UnitTests",
      "errors": 0,
      "failures": 0,
      "skipped": 0,
      "timestamp": "<A Timestamp>",
      "time": 1,
      "tests": 1,
      "testcase": [
        {
          "classname": "SimpleTest",
          "name": "testASimpleFunction",
          "time": 0.1
        }
      ]
    },
    {
      "name": "FailingUnitTests",
      "errors": 0,
      "failures": 1,
      "skipped": 0,
      "timestamp": "<A Timestamp>",
      "time": 1,
      "tests": 1,
      "testcase": [
        {
          "classname": "FailingTests",
          "name": "testAFailingFunction",
          "time": 0.1,
          "failure": [
            {
              "XCAssert": "1 is not equal to 2"
            }
          ]
        }
      ]
    }
  ]
}

(reference: https://github.com/Kesin11/ts-junit2json)

I'm not sure that we have access to this level of detail in the testing data to fully populate this format, but in the meanwhile we could return what we have.

What do you think?

[tomerd]

@NSCoder using an XUnit JSON representation sounds very good to me.

[tomerd]

if we go for an XUnit like format, we should probably create utility structs which will be the ones that are encoded. there will be a small cost of crating a second representation in memory, but I think it is worth it since it will give us a degree of freedom between the "real" model and the one we want to write to disk as JSON. its a technique I often use when build server JSON APIs so I know it works well.

[tomerd]

fwiw, we can then also modify the XUnit generator to use these XUnit specific representations which will make that consistent across the two on-disk representations