Approval Test Integration Bug

[lp55]

Hi,

I really like the [Boost].UT framework. It's clean and easy to use and does away with macros. Thanks for developing it!

Anyways, I want to use approval tests (https://github.com/approvals/ApprovalTests.cpp) in a project I'm working on, and I'm implementing the integration between UT <> ApprovalTests.cpp. I followed the guidelines (https://github.com/approvals/ApprovalTests.cpp/blob/master/doc/SupportingNewTestFramework.md#top) and the UT reporter example (https://godbolt.org/z/gsAPKg). I made this simple example:

#include <string>

#include "ut.hpp"
#include "ApprovalTests.hpp"

namespace ut = boost::ut;

namespace ApprovalTests
{
    namespace cfg {
        class reporter : public ut::reporter<ut::printer> {
            private:
                TestName currentTest;

            public:
                auto on(ut::events::test_begin test_begin) -> void {
                    std::string name = test_begin.name;
                    currentTest.sections.emplace_back(name);
                    ut::reporter<ut::printer>::on(test_begin);
                }
                auto on(ut::events::test_run test_run) -> void { 
                    ut::reporter<ut::printer>::on(test_run);
                }
                auto on(ut::events::test_skip test_skip) -> void { ut::reporter<ut::printer>::on(test_skip); }
                auto on(ut::events::test_end test_end) -> void {
                    while (!currentTest.sections.empty()) {
                        currentTest.sections.pop_back();
                    }
                    ut::reporter<ut::printer>::on(test_end);
                }
                template <class TMsg>
                auto on(ut::events::log<TMsg>) -> void {}
                template <class TLocation, class TExpr>
                auto on(ut::events::assertion_pass<TLocation, TExpr> location) -> void {
                    currentTest.setFileName(location.location.file_name());
                    ApprovalTestNamer::currentTest(&currentTest);
                    ut::reporter<ut::printer>::on(location);
                }
                template <class TLocation, class TExpr>
                auto on(ut::events::assertion_fail<TLocation, TExpr> fail) -> void { ut::reporter<ut::printer>::on(fail); }
                auto on(ut::events::fatal_assertion fatal) -> void { ut::reporter<ut::printer>::on(fatal); }
                auto on(ut::events::exception exception) -> void { ut::reporter<ut::printer>::on(exception); }
                auto on(ut::events::summary summary) -> void { ut::reporter<ut::printer>::on(summary); }
        };
    }  // namespace cfg
}

template <>
auto ut::cfg<ut::override> = ut::runner<ApprovalTests::cfg::reporter>{};

int main()
{
    using namespace ut;
    using namespace ApprovalTests;

	"Approval"_test = []() {
        //expect(true);
        expect(nothrow([] { Approvals::verify("Approval Tests can verify text via the golder master method"); }));
	};

}

The issue is unless the line containing expect(true); is uncommented it doesn't work. It complains that currentTest was not configured. This seems more like a UT issue than an ApprovalTests issue.

Can anyone help? I'm not familiar enough with the UT code logic to track this issue. Or am I wrong and this is an ApprovalTests.cpp issue?

I appreciate any help.

[claremacrae]

Hi, I happen to be one of the maintainers of ApprovalTests.cpp... Am happy to try and help with this, or answer questions, if I can...

[claremacrae]

Oh, and thank you for trying to make this integration!

[lp55]

Hi @claremacrae,

It would be great if you could check things on the ApprovalTests.cpp side. As I said I don't think it's an issue there, but I haven't had time to do a in depth debug.
And thanks for ApprovalTests.cpp! It's a very cool idea and it really fills a need I have in my current project. I have a small patch for TortoiseGitMerge tool support that I'll submit soon.

[claremacrae]

Hi @lp55

You're welcome...

I can reproduce the behaviour you describe, but I can't get a debugger to break in the ApprovalTests code to work out what's going on. It's almost as though it's all being executed at compile time???

[claremacrae]

Are you available to screens-share and pair on this?

[lp55]

Hi @claremacrae

I'm available for screen-share depending on the tool. What do you use?

[claremacrae]

Great - How about you email me to github@cfmacrae.fastmail.co.uk and we take it from there?

[claremacrae]

We were able to get this working - the key turned out to be that we needed the code that handled ut::events::test_begin to have access to the filename the test was in... which we were able to get by changing the test_begin method to this:

                auto on(ut::events::test_begin test_begin) -> void {
                    const std::string fileName =
                            ut::reflection::source_location::current().file_name();
                    currentTest.setFileName(fileName);

                    std::string name = test_begin.name;
                    currentTest.sections.emplace_back(name);

                    ApprovalTestNamer::currentTest(&currentTest);

                    ut::reporter<ut::printer>::on(test_begin);
                }

[claremacrae]

In case anyone else wants to look at this in the short term, I've attached a slightly-modified version of what we ended up with - renamed to .txt so I could upload it here.

approvals.txt

@lp55 had kindly offered to work it up to something to add to ApprovalTests.cpp, so that it supports ut directly - thank you!

[krzysztof-jusiak]

Thank you, @lp55 and @claremacrae for your work, very exciting stuff 👍 ApprovalTests looks very interesting and I would love to put a link to the integration between it and it on the website, please keep me in the know when you have something ready so that I can update the website, thanks again 🙏 nice one 👍

[krzysztof-jusiak]

Closing it for now as there is nothing to do from ut perspective ATM, feel free to reopen the issue and/or ping me with any updates/problems, thanks.