Use --json parameter to write JSON to stdout

[tbolon]

I am planning to use hurl for integration testing and I want to parse the hurl output to generate CI/CD test results.

I have developed a custom program which is responsible for running hurl for multiple files and generate a test result compatible with our CI/CD infrastructure (azure devops).

As my custom program can read the standard output from hurl, and because the json format is perfect to read the results, I expected I could read the json directly from the program output.

But is seems the json result must be written in a file.

Perhaps if the file argument is omitted when adding the --json option, the result could be simply written to the output ?

PS: I know that the json option already append result to files (in case when you want to run hurl multiple times), but I find the standard output parsing easier (because it will not pollute the output directory with a result file).

PS2: Maybe this behavior could also be applied to the --html option ?

[fabricereix]

Indeed, it might be good to be able to select a JSON output (instead of the default body).
We could include:

• the last response body
• the cookies store
• the captures

What data do you need for your CI?

[tbolon]

The actual content of the --json <file> output would be perfect for now, just some points which could be enhanced (based on my use case):

• I have no use for the last response body in my case, but as the current output has it, maybe it should be preserved in the json.
• The captures are already in the json. Perhaps a final capture with all ambiant values at the end of the file?
• I do not use cookies for now.
• The asserts array is quite strange, seems to contains an array with fixed values "asserts": [{"actual": "1.1","expected": "1.1"},{},{}] instead of the actual asserts. If the real asserts are included, maybe it could be useful to know which one failed in the case where there are multiple asserts.

[fabricereix]

ok, we are going to think about it.
It probably makes sense to have only one type of JSON output.

Relating to the asserts in JSON, we haven't used it yet in practise. For sure, that needs to be updated. We will improve it for the next release.

[tbolon]

No problem, take your time. For now I will use the json file on disk to generate the test result summary.

[fabricereix]

Relating to asserts in JSON output/report

We would like to have the same data that we produce in the terminal.

The question is more about the format:
either give the same error message printed to the terminal,
or produce something more structured with an 'actual' and 'expected' field.

what do you think @tbolon ?

example with the status query

Terminal Output:

error: Assert Status
  --> /tmp/test.hurl:8:8
   |
 8 | HTTP/* 203
   |        ^^^ actual value is <200>

json#1

{ 
  "success": false,
  "message": "Assert Status\n--> /tmp/test.hurl:8:8\n   |\n 8 | HTTP/* 203\n   |        ^^^ actual value is <200>",
  "line": 8
}

or
json#2

{ 
  "success": false,
  "actual": 200,
  "expected": 203,
  "line": 8
}

example with xpath

Terminal output:

error: Assert Failure
  --> /tmp/test.hurl:10:0
   |
10 | xpath "string(//title)" startsWith "{{name}}"
   |   actual:   string <Example Domain>
   |   expected: starts with string <toto>
   |

json#1

{ 
  "success": false,
  "message": "Assert Failure\n  --> /tmp/test.hurl:10:0\n\   |\n10 | xpath "string(//title)" startsWith "toto"\n   |   actual:   string <Example Domain>\n   |   expected: starts with string <toto>\n   |",
  "line": 10
}

or
json#2

{ 
  "success": false,
  "actual": "<Example Domain>",
  "expected": "starts with string <toto>",
  "line": 10
}

[tbolon]

My goal is to populate a test report such as the one exposed by azure devops, see this example and this detail of one failed test, or this detail with an actual and expected value

The "error message" pane displays the text returned by the failed test as-is, respecting new-line, so is seems the raw output could be enough.

The only gain of having the structured actual/expected value could if we want to create a custom test failure message to highlight the expected/actual values.

So, to conclude, I am not sure the structured actual/expected properties is really useful.

My main grief was that the default raw output was emitting the entire response body, and I was expecting the json output to give me a more concise result that I could parse and return as a test failure message, which could be useful for the person reading the test result.

[fabricereix]

ok thanks for your explanations.
We will start with the fields status and message.

We will trunk the actual value to a maximum size (both in the terminal ouput and json).

[fabricereix]

Asserts have been added in JSON in #301.

We will keep this issue open until we find something satisfactory for the JSON output usage.
We also tend to think now that JSON should simply be another type of output, rather than an additional report.

[fabricereix]

As described in #330 , --json will now change the default output to JSON.
It won't take a filename anymore.

[lepapareil]

As described in #330 , --json will:

outputs to stdout (instead of a file)
replace default body output
conflicts with --summary