New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[TW-1936] Tweak tests to have fuller TAP compliance #1948

Closed
taskwarrior opened this Issue Feb 14, 2018 · 5 comments

Comments

Projects
None yet
2 participants
@taskwarrior
Contributor

taskwarrior commented Feb 14, 2018

Paul Fenwick on 2017-11-26T23:50:35Z says:

I almost cried with joy when I discovered that TaskWarrior uses TAP for testing, but after running [prove|https://metacpan.org/pod/prove] I discovered there's a couple of places where it departs from the [TAP specification|https://testanything.org/tap-specification.html].

In particular, for the python code:

  • Expected failures were reported as as 'skip' rather than not ok # TODO
  • Output was being sent to STDERR by default, rather than STDOUT.

In both python and C++:

  • Skipped tests were reported as 'skip' rather than ok # skipped

The soon-to-be-added patch¹ adds more TAP compliance to the TaskWarrior test-suite, meaning one can do cool things like prove --state=failed to only re-run failed tests.

¹ I have a working patch, but I'm checking my python changes for best-practice, since that's the language I'm least skilled in.

@taskwarrior taskwarrior added this to the 2.6.0 milestone Feb 14, 2018

@taskwarrior

This comment has been minimized.

Contributor

taskwarrior commented Feb 14, 2018

Migrated metadata:

Created: 2017-11-26T23:50:35Z
Modified: 2017-12-02T17:35:48Z
@taskwarrior

This comment has been minimized.

Contributor

taskwarrior commented Feb 14, 2018

Paul Fenwick on 2017-11-27T01:41:28Z says:

Patches added, targeting 2.6.0. Things to note:

  • filter.t has three failing tests on my system, but they were failing before these patches.
  • I'm still not convinced that I've made the changes to having the python library use STDOUT in the best way, but I've submitted my changes as-is so they can be reviewed and improved.
  • I'd love to have a way for template.t to skip everything (possibly based upon environment variables), so a simple prove ./*.t can be used to run everything, but that will be another test for another time.

Feedback very welcome! I'm a daily user of TAP, but not a daily user of C++ and python. :)

@taskwarrior

This comment has been minimized.

Contributor

taskwarrior commented Feb 14, 2018

Paul Beckingham on 2017-12-02T17:21:45Z says:

While I like the patch (0001), it changes the summary reporting from this:
{code:java}
Passed: 3976
Failed: 3
Unexpected successes: 0
Skipped: 11
Expected failures: 5{code}
To this:
{code:java}
Passed: 3986
Failed: 9
Unexpected successes: 0
Skipped: -5
Expected failures: 5{code}
I need to understand this before I push.  Investigating now.  Once resolved, I'd like to apply this patch to multiple projects that use the same framework.

Edit: Perhaps I should have tested 0002 before commenting.

@taskwarrior

This comment has been minimized.

Contributor

taskwarrior commented Feb 14, 2018

Paul Beckingham on 2017-12-02T17:35:48Z says:

Thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment