Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve --progress and --report output #161

Merged
merged 7 commits into from
Oct 13, 2023
Merged

Conversation

cgay
Copy link
Member

@cgay cgay commented Oct 1, 2023

For example, the failures report now looks like this:

$ _build/bin/testworks-test-suite-app --progress none
FAILED: testworks-test-suite-app
  FAILED: test-assert-equal-output
    FAILED: b
      1 (from expression "1") is not an instance of {class <string>}.
    FAILED: f
      no condition signaled
    CRASHED: g
      Error evaluating assertion expression: foo
    FAILED: list, same size, different elements
      want: #("a", "b", "c", "d")
      got:  #("a", "b", "x", "d")
      detail: element 2 is the first mismatch
      ...
Ran 1289 assertions
Ran 54 tests and 2 benchmarks: 1 failed and 4 expected failures
FAILED in 0.312805 seconds

@cgay cgay force-pushed the got-want branch 2 times, most recently from d05ba78 to e402e93 Compare October 12, 2023 16:10
@cgay cgay changed the title Improve output of assert-equal failures Improve --progress and --report output Oct 12, 2023
* Use multi-line output with "want" and "got" values, and detailed output if
  present, on separate lines.
* Don't sort detail results since they aren't guaranteed to be sortable.
* Separate do-check-equal into same plus do-check-not-equal; it's less
  complex.

dylan-lang#139
These are needless complexity as far as I can tell and `with-test-unit` has
been a no-op for a long time.

This is an incompatible changes so I'm bumping the major version to 3.
This addresses an old TODO in the code and a related problem reflected in the
CLI.

```
Flag        Old options             New options
--progress  {none,default,verbose}  {none,minimal,all}
--debug     {none,crashes,failures} {none,crashes,all}
```

Note that previously `--debug failures` already had the effect of debugging
both crashes and failures, so `--debug all` is a more accurate name. (And I
think it unlikely anyone would need to debug failures but not crashes.)
Example:
```
Ran 1276 assertions
Ran 53 tests and 2 benchmarks: 4 expected failures
PASSED in 0.023699 seconds
```

Notably, no attempt is made to report the number of passing/failing assertions;
only tests and benchmarks. The total number of assertions is still displayed
since it can (sometimes) give a signal as to the thoroughness of the tests.
* Remove unnecessary "Running "
* Always show result status (PASSED/FAILED/etc) on left margin
* Always put failure detail on a line by itself.
* Make them look roughly like the progress report, with pass/fail status up
  front and failure reason on a separate line.
* Greatly simplify the print-result-info generic.
* Use the same indentation mechanism as the progress report.
@cgay cgay merged commit 5aa98e6 into dylan-lang:master Oct 13, 2023
2 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant