Support different categories of test outcomes #83
Comments
jernst
pushed a commit
to jernst/feditest
that referenced
this issue
May 15, 2024
…eport on New classes XXXTranscript for JSON-based serialization and deserialization of full TestRun transcripts per fediverse-devnet#103 Distinguish between hard failures, soft failures, degradations, errors: use new feditest.xxx_assert_that instead of Hamcrest assert_that to raise different kinds of problems per fediverse-devnet#83 A TestRunController decides what to do next after a session, test, or step has been executed; automatic and interactive implementations per fediverse-devnet#79 Simplified result reporting implementations, can report in several formats, also addresses fediverse-devnet#105 New sub-command: convert_transcript Shrunk what needs to be in feditest.__init__.py More consistent object naming when printing to the console FEDITEST_VERSION as a constant Would be surprised if I hadn't broken some stuff, sorry
jernst
added a commit
that referenced
this issue
May 16, 2024
* Remove duplicate code Make acct: regex more correct based on reading the RFC use http_https_acct_uri_parse_validate in prompt_user validation for webfinger handles Add run --template flag to support alternate HTML templates Add jinja template that generates an HTML fragment suitable for embedding not an entire page Add more info to HTML reports use .find not .index for substring search * Missing return value * Missing r'' for this regex * Move the sandbox protocol to use floats instead of ints, so we can demonstrate a soft AssertionError * Make linter happy * Extension of the WebFingerNode API * Record start and end times of the test run * Now recording everything in a TestRun regardless of what we want to report on New classes XXXTranscript for JSON-based serialization and deserialization of full TestRun transcripts per #103 Distinguish between hard failures, soft failures, degradations, errors: use new feditest.xxx_assert_that instead of Hamcrest assert_that to raise different kinds of problems per #83 A TestRunController decides what to do next after a session, test, or step has been executed; automatic and interactive implementations per #79 Simplified result reporting implementations, can report in several formats, also addresses #105 New sub-command: convert_transcript Shrunk what needs to be in feditest.__init__.py More consistent object naming when printing to the console FEDITEST_VERSION as a constant Would be surprised if I hadn't broken some stuff, sorry --------- Co-authored-by: Johannes Ernst <git@j12t.org>
|
I think we are done with this. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
It might be advantageous if the outcome of a test, or a step in a test, could have more categories than just pass/fail, such as:
Maybe the way to implement this is to:
The Exceptions come in several flavors:
AssertionErrorand subclasses indicate Hard fail.NotImplementedByNodeErrorfor "not supported".It might be advantageous to make raising those really concise, along the lines of Hamcrest's
assert_that, e.g.hardfail_assert_thatsoftfail_assert_thatdegrade_assert_thatImplemented like Hamcrest does, which is just a handful of lines: https://github.com/hamcrest/PyHamcrest/blob/main/src/hamcrest/core/assert_that.py
The text was updated successfully, but these errors were encountered: