Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[enhancement] Add support for "stand-alone assertions". #114

Closed
aghast opened this issue Jan 15, 2018 · 6 comments

Comments

@aghast
Copy link

commented Jan 15, 2018

Description

The features that make doctest attractive for unit tests also make it attractive for other uses: it is cheap to include, lightweight, and low-cost in terms of compilation speed and performance impact.

I'd like to see a version of doctest that supports using the existing macros and code to run "stand alone" assertions. That is, I want to be able to use all the various assertion macros in my mainline code:

double sqrt(double x, double epsilon)
{
    FAST_CHECK_GE(x, 0.0);
    FAST_CHECK_LE(epsilon, 0.01);
    // ...
}

In order to do this, the reporting has to change - I'm not quite clear how. I made one attempt at just "faking" a test case in my own main() and then calling my program() function. This unfortunately mixes the output of my program with the output from doctest, so it's not quite a solution. (I wanted to make sure that all the regular data structures that doctest maintains internally were active while it was running.)

Keeping in mind that the idea is to use doctest for both in-source unit testing and stand-alone assertions, I suspect the solution will either involve a new set of macros, or will involve checking the "running in test" status prior to dispatching an error report. The existing macro-based solution for logging will obviously only support all-or-nothing changes, rather than "test mode" and "run mode".

@onqtam

This comment has been minimized.

Copy link
Owner

commented Feb 7, 2018

That's interesting - I'll think about it, but it probably won't happen for version 1.3...
And sorry for the late reply :)

@onqtam

This comment has been minimized.

Copy link
Owner

commented May 23, 2018

Hi! I'm actually thinking about this for the upcoming 1.3 release (hopefully within a month)...

But such inline asserts can come in many forms:

what set of features would you like to see implemented..? for example: user defined callback functions...?

what would make sense for doctest?

@aghast

This comment has been minimized.

Copy link
Author

commented Jun 2, 2018

First, my "vision" for this request is simply that the doctest unit test assertions represent a constraint DSL. Given that I want to impose constraints elsewhere in my code (preconditions, invariants, etc.) it makes sense to use the same language so as to reduce confusion. So the priority then is for the "test" statements and the "assert" statements to have exactly the same look and to have exactly the same meaning.

With that said, what makes sense for doctest is what currently makes sense for doctest: simple, lightweight, fast. Anything that violates one of those, is wrong.

Ideally, the same macros would be used, because that's simple. They would just magically figure out whether they were in a test or not. Otherwise, if somehow that is not possible, the minimum different macros should be available: FAST_ASSERT_LE(x, y) or some such.

If something more complex is needed, it should be an optional extra configuration (like implementing parameter parsing in my own main() function is "optional" and "extra"). So if I want the "assertion failed" to call a user defined function, it should be either a separate (optional) #define or an extra configuration statement or something.

The essence of assertions is that they (1) can be configured on or off; and (2) they terminate the program on failure.

Point (1) seems like a no-brainer. There should be some option -DDOCTEST_NO_ASSERT or whatever, that compiles support for assertions right out.

Point (2) is less certain. I have seen some assert libraries that support "levels" much the way the logging does, with "FATAL" assertions and "DEBUG" assertions. I would suggest leaving that for a "version 2", since I'm sure there are some tricky bits (such as the inevitable "how can I replace your log library with my own log library") that would need to be dealt with.

I don't have any secret desire for controlling how the assertions are handled, because in my own uses cases an assertion failures means the program is borked and I don't care how long it takes any more, just give me a good report. Obviously, other people will have different views on that. But I favor the attributes mentioned above over some complex beast that can report failures using TCP over barbed wire.

onqtam added a commit that referenced this issue Jul 2, 2018
- added the ability to call asserts outside of a testing context with…
…out recompiling! currently the fast asserts are optimized for runtime speed to the maximum but the other asserts are not. Sadly logging macros - like INFO() - aren't supported... yet! - closes #114 - the way to do it is to call setAsDefaultForAssertsOutOfTestCases() on a doctest::Context and optionally to register a custom handler with doctest::Context::setAssertHandler()

- doctest::isRunningInTest() changed to doctest::is_running_in_test - a bool... for performance reasons - relates #56
@onqtam

This comment has been minimized.

Copy link
Owner

commented Jul 2, 2018

just implemented this in the dev branch! The fast asserts are optimized for speed - all other asserts will work as well but I would use them sparingly in production code. So now you can compile all your production code once and have doctest asserts in it - either for production builds or test builds!

custom handlers should look something like this:

void handler(const doctest::AssertData& ad) {
    ((void)ad.m_file); // access data about the assert - this is called only on failure
    ((void)ad.m_line);
    ((void)ad.m_expr);
}

do this on a doctest::Context somewhere:

context.setAsDefaultForAssertsOutOfTestCases();
context.setAssertHandler(handler);

if no handler is set (but a context is set as a default one - otherwise will crash) - then std::abort() is called when an assert fails.

documentation and an example will follow soon! Should also do it for the new reporter system...

I may also change the names before release. Let me know what you think!

EDIT:

https://github.com/onqtam/doctest/blob/dev/doc/markdown/assertions.md#using-asserts-out-of-a-testing-context

onqtam added a commit that referenced this issue Jul 3, 2018
polished the feature for allowing the use of asserts outside of a tes…
…ting context (also example and docs) - relates #114
onqtam added a commit that referenced this issue Jul 3, 2018

@onqtam onqtam closed this in 1d68a13 Aug 23, 2018

onqtam added a commit that referenced this issue Aug 23, 2018
polished the feature for allowing the use of asserts outside of a tes…
…ting context (also example and docs) - relates #114
onqtam added a commit that referenced this issue Aug 23, 2018
@nuzrub

This comment has been minimized.

Copy link

commented Nov 10, 2018

This was very helpful to me, thanks :) I hope this will be added to the documentation at somepoint (or maybe it already has but I just couldn't find it).

@onqtam

This comment has been minimized.

Copy link
Owner

commented Nov 12, 2018

@nuzrub here it is in the documentation along with an example :)

Perhaps the documentation isn't structured very well - people often don't find something...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.