Support GitHub Actions Workflow Commands

[nikeee]

Problem to solve

Enhancing the experience in CI by showing errors in line with the code.

Proposal

GitHub actions supports putting annotations in source files during a CI run:
https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions

Additional context and resources

You can see this in action when using eslint (although it is implemented in a different way for eslint). It looks like this (for no-unused-vars):

This could improve the DX in a similar way to #1067. The current output:

% hurl --test tests.hurlfile
tests.hurlfile: Running [1/1]
error: Assert status code
  --> tests.hurlfile:73:10
   |
73 | HTTP/1.1 200
   |          ^^^ actual value is <404>
   |

Looks similar to what GitHub would render in the web UI.

Tasks to complete

I've opened a similar discussion at a different project, which also outlines some implementation ideas: biomejs/biome#441

The simplest way to implement that would be to print something like this when GITHUB_ACTIONS is true:

::error file={name},line={line},endLine={endLine},title={title}::{message}
# e.g.:
::error file=tests.hurlfile,line=73,col=10,endCol=13::actual value is <404>

This could however, be difficult when Hurl takes the test input via stdin.

[jcamiel]

Hi @nikeee sorry for the late replay!

Do you think we can make this output using the --json? By processing the JSON output of the run, I think we can get the exact output needed for GitHub. Of course, it's simpler if this output is "natively" supported by Hurl, but we would like the JSON output to be useful for this kind of usage.

[nikeee]

I understand that adding a vendor-specific extension may not align with project goals. Processing the JSON output may work (didn't have the chance to look at it at the moment).

biome (formerly rome) is probably going to check whether the command is running in GH actions (via the GITHUB_ACTIONS env var). And if the ci command is used, the output will include these directives. Doing it that way it will "just work" for 90% of the users.

A separate binary for transforming the JSON would probably work, too.

As I'm writing this, the entire feature, whether using JSON or not, would probably not work for most users. In our current setups, we generate the Hurl files in a shell script and save them to a temporary directory. So the used hurl files aren't part of the repository, which makes us unable to highlight assertion errors in source files.
I think this feature can only be used in a meaningful way once #115/#1723/#317 are solved in some way.