Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

deno test: Consider adding "tap" format #14390

Closed
bartlomieju opened this issue Nov 12, 2021 · 8 comments
Closed

deno test: Consider adding "tap" format #14390

bartlomieju opened this issue Nov 12, 2021 · 8 comments
Labels
help wanted community help requested suggestion suggestions for new features (yet to be agreed) testing related to deno test and coverage

Comments

@bartlomieju
Copy link
Member

Currently deno test has only default and json format available. Tap is a very popular format (especially in Node world) that would allow users to pipe the output into different reporters of their liking.

We should research what it would entail to implement it.
@bartlomieju bartlomieju added suggestion suggestions for new features (yet to be agreed) testing related to deno test and coverage help wanted community help requested labels Nov 12, 2021
@bartlomieju
Copy link
Member Author

CC @brianleroux you might be interested in this one?

@brianleroux
Copy link

def! we make heavy use of TAP

@lucasfcosta
Copy link

Hi @bartlomieju, I'd be interested in contributing with a tap reporter for deno test (if @brianleroux isn't working on it already).

However, as I was reading the code, I couldn't find the json reporter you mention is already available. My plan was to have a look at how the json reporter is implemented so that I could propose a plan to implement the tap reporter.

I did try to look around the entirety of the code for where this reporter could be specified but I could only find the PrettyTestReporter to be quite hardwired into the current testing logic. I also couldn't find any guidance in deno test --help after a cargo install from master.

I've only recently started writing rust, so I apologize in advance if this is an obvious question.

Also, as someone who's core-maintained chai, sinon, and previously contributed to jest (and also did quite a few things around testing, including a book) I'd be quite keen to also contribute with anything else test related going forward (in case you'd like).

@bartlomieju
Copy link
Member Author

Hi @lucasfcosta, thanks for picking interest in this topic!

However, as I was reading the code, I couldn't find the json reporter you mention is already available. My plan was to have a look at how the json reporter is implemented so that I could propose a plan to implement the tap reporter.

I did try to look around the entirety of the code for where this reporter could be specified but I could only find the PrettyTestReporter to be quite hardwired into the current testing logic. I also couldn't find any guidance in deno test --help after a cargo install from master.

Yeah, my mistake, we don't have JSON reporter for deno test, but rather for deno lint. Indeed the logic right now is pretty much hardwired and would require some refactors to allow different reporter kinds. AFAIK no one is working on this feature right now, let me discuss this with the rest of the team and get back to you in a couple days.

Also, as someone who's core-maintained chai, sinon, and previously contributed to jest (and also did quite a few things around testing, including a book) I'd be quite keen to also contribute with anything else test related going forward (in case you'd like).

Great, I'd love to hear your feedback on the topic.

@bartlomieju bartlomieju transferred this issue from denoland/deno Apr 24, 2022
@bartlomieju
Copy link
Member Author

bartlomieju commented Apr 24, 2022 •
edited
Loading

Transfered to deno_std as this should be implemented on top of Deno.test() API similar to testing/bdd.ts

Scratch that, got confused. This needs to be implemented in Rust.

@bartlomieju bartlomieju transferred this issue from denoland/std Apr 25, 2022
@bartlomieju
Copy link
Member Author

We could use testanything crate to do the heavy lifting for us. With #19786 landed it should become quite accessible to integrate that crate - see #19747 for some pointers.

@bartlomieju bartlomieju mentioned this issue Jul 14, 2023
Merged
@bartlomieju bartlomieju mentioned this issue Aug 5, 2023
Merged
bartlomieju pushed a commit that referenced this issue Aug 25, 2023
@SyrupThinker
feat(cli/tools): add TAP test reporter (#14390) (#20073)
a526cff 
This PR adds a test reporter for the [Test Anything
Protocol](https://testanything.org).

It makes the following implementation decisions:
- No TODO pragma, as there is no such marker in `Deno.test`
- SKIP pragma for `ignore`d tests
- Test steps are treated as TAP14 subtests
  - Support for this in consumers seems spotty
- Some consumers will incorrectly interpret these markers, resulting in
unexpected output
- Considering the lack of support, and to avoid implementation
complexity,
subtests are at most one level deep (all test steps are in the same
subtest)
- To accommodate consumers that use comments to indicate test-suites
(unspecced)
  - The test module path is output as a comment
  - This is disabled for `--parallel` testing
- Failure diagnostics are output as JSON, which is also valid YAML
- The structure is not specified, so the format roughly follows the spec
example:
  ```
  ---
  message: "Failed with error 'hostname peebles.example.com not found'"
  severity: fail
  found:
    hostname: 'peebles.example.com'
    address: ~
  wanted:
    hostname: 'peebles.example.com'
    address: '85.193.201.85'
  at:
    file: test/dns-resolve.c
    line: 142
  ...
  ```
@bartlomieju
Copy link
Member Author

Closed by #20073. deno test --reporter=tap will be available in Deno v1.37 in September.

@brianleroux
Copy link

❤️ nice!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
help wanted community help requested suggestion suggestions for new features (yet to be agreed) testing related to deno test and coverage
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@brianleroux @lucasfcosta @bartlomieju