Add test reports

[rcasia]

Thank you for the excellent effort!

It would be great if bashunit incorporated a functionality to generate test report files, facilitating seamless integration with IDEs or diverse software platforms.

I'm currently working on neotest-bash, and I've encountered certain challenges (rcasia/neotest-bash#13).

[Chemaclass]

Great idea, @rcasia! I assume it would be a json file such us:

[
  {
    "type": "test",
    "file": "/home/User/project/tests/unit/foo_test.sh",
    "function": "foo_1",
    "line": 12,
    "time": 0.0123
  },
  {
    "type": "test",
    "file": "/home/User/project/tests/unit/foo_test.sh",
    "function": "foo_2",
    "line": 34,
    "time": 0.0045
  },
  {
    "type": "test",
    "file": "/home/User/project/tests/unit/bar_test.sh",
    "function": "bar",
    "line": 8,
    "time": 0.067
  }
]

Could you, please, verify it and/or answer here an example of the expected report file that you would like to have?

[rcasia]

That's a good start. Exploring further into it, to have a smooth integration with other software we would need something more like this example:

[
  {
    "name": "foo1",
    "function": "foo_1",
    "state": "green",
    "time": 0.0123
  },
  {
    "name": "foo2",
    "function": "foo_2",
    "state": "yellow",
    "time": 0.0897,
    "feedback": [
      {
        "line": 15,
        "comment": "We need to test this",
        "type": "todo"
      }
    ]
  },
  {
    "name": "bar",
    "function": "bar",
    "state": "red",
    "time": 0.0054,
    "feedback": [
      {
        "line": 19,
        "comment": "Expected '/tmp/hello-world.txt' to exist but 'does not exist'",
        "type": "failure"
      },
      {
        "line": 20,
        "comment": "Detailed description of what needs to be done",
        "type": "todo"
      },
      {
        "line": 21,
        "comment": "Not running under Commodore",
        "type": "skip"
      }
    ]
  }
]

Some aclarations:

• Having a separate test report for each test file would be handy, especially for bigger projects with basunit. I designed the example with that idea in mind.
• The state is marked as red if there is any failure, yellow if there is any todo or skip, and green if the test has passed.
• name is just the prettier printed version of function name.
• line represents the line of code associated with the reported "comment." If a specific line isn't available, it defaults to the line of the function declaration.

I think using a report that follows these specifications will make it easy for any software to integrate with bashunit.

[khru]

Sounds good to me, on our roadmap we have in mind to use https://github.com/infertux/bashcov wich it's what other libraries are using, we could develop this support, but we have to keep in mind that this would somehow make us give support to this new format, that is only use on your tool, I'm not sure if it makes too much sense or if it makes more sense to create a connector to that report that to this custome support

[rcasia]

Although neotest-bash would currently only require "function" and "state" (along with the referenced test file), I crafted this proposal with a broader perspective, considering potential integrations with IDEs, GitHub Actions, and other CI tools. My goal was to proactively prevent potential breaking changes, anticipating future needs that might necessitate that information in a test report.

[khru]

Although neotest-bash would currently only require "function" and "state" (along with the referenced test file), I crafted this proposal with a broader perspective, considering potential integrations with IDEs, GitHub Actions, and other CI tools. My goal was to proactively prevent potential breaking changes, anticipating future needs that might necessitate that information in a test report.

I understand your point of view, could you elaborate more on how you see this report as part of the integrations?

Not because I don't see value in what you say, but rather because I would like to have a clearer vision of what you see, that is, how could this JSON help us integrate with an IDE, for example? do you have documentation that helps us see where you want to go?

[rcasia]

Certainly not every field holds the same importance. "function" and "state" are fundamental. The remaining could be treated as nice-to-haves and could be done gradually, or perhaps not at all if there's no immediate integration in sight.

IDEs use the reports to offer the feedback information to the final user:

• name and time: simply for a user-friendly display
• feedback: for any improvement to enhance the user experience. examples:
  • to help users notice and keep track of skipped tests or "todo" items, along with reasons.
  • to highlight lines that are related to failures
  • overall user experience

As for GitHub Actions, they have something called 'line annotations' in their status checks. Although it's not well-documented, you can find a brief mention here. Among other things, they enable users to view a summary of failures or to identify the reason for a failure/warning along the line of code in the PR files view. I've started seeing them in every repository at work recently.

Ultimately, you'll determine whether it makes sense or not.