Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to filter test by name (-t, --run_test) if template type contains multiple parameters #223

TorstenRobitzki opened this issue May 10, 2019 · 7 comments


Copy link

@TorstenRobitzki TorstenRobitzki commented May 10, 2019

I'm used to using the -t option to separate a failing test during TDD and to inspect the cause be using printf() debugging. Usually, I just copy the test name just from the error message. So, if I have an error message like this:

error: in "filter_me<test_parameters<100, 100>>": check sizeof( int ) == sizeof( Param ) has failed [4 != 1]

I'm used to filter the test with: -t "filter_me<test_parameters<100, 100>>". That works fine as long as there is no comma in the test name, as above.

Following example:

#define BOOST_TEST_MODULE "reproducer"
#include <boost/test/included/unit_test.hpp>

#include <tuple>

template < int, int >
struct test_parameters {};

using test_cases = std::tuple<
    test_parameters< 100, 100 >

BOOST_AUTO_TEST_CASE_TEMPLATE( filter_me, Param, test_cases )
    BOOST_CHECK_EQUAL( sizeof( int ), sizeof( Param ) );

Will generate the error message above and will not allow you to filter the test by the name printed. However, the test can be filter with a wildcard behind the first comma: -t "filter_me<test_parameters<100,*".

I would expect that I can use the reported name of a failing test to filter for exactly that failing test.

Edit: Observed in Boost 1.68 with Apple LLVM version 10.0.0

Copy link

@raffienficiaud raffienficiaud commented May 13, 2019

Thanks for the report. Right now, two possible solutions that come to my mind:

  1. Either replace every symbols (<, >, ,...) etc by _
  2. Or properly handle the name of the test test_parameters< 100, 100 > in the filtering

As a user, I find test_parameters< 100, 100 > nicer, but it also gives the sense that the framework is able to parse the syntax properly (means that if I have test_parameters<100,100. > it should also work).
Since 1. is much easier than 2. and 2. can also be misleading, and also considering the fact that we already do that for user provided names, I'll go for 1.

What do you think?

Copy link

@TorstenRobitzki TorstenRobitzki commented May 13, 2019

Hi, I would just expect the framework to take the exact same name, that it prints. My expectation would be, that when the framework says test "filter_me<test_parameters<100, 100>>" discovered a bug, that I can execute exactly that test by exactly that name ("filter_me<test_parameters<100, 100>>"). (Which by the way, works most of the time).

Copy link

@raffienficiaud raffienficiaud commented May 13, 2019

We both agree to qualify the current behaviour that you described as buggy, and that was not my question.

Thanks for the report, I'll go for 1.

Copy link

@TorstenRobitzki TorstenRobitzki commented May 14, 2019

sorry for not understanding your question! :-) I'm ok with the names of the symbols as they are now, but I would be ok if they would change.

Thank you for looking into this!

Copy link

@raffienficiaud raffienficiaud commented Oct 30, 2019

I implemented a fix on the branch topic/GH-223-cannot-filter-template-test-cases. Would it be possible for you to try?

raffienficiaud added a commit that referenced this issue Oct 31, 2019

* topic/GH-223-cannot-filter-template-test-cases:
  Change log
  Documentation updates
  Normalizing test names accross several compilers
  Sanitizing template test cases that contain ',' in their name
Copy link

@TorstenRobitzki TorstenRobitzki commented Nov 2, 2019

Works like a charm. Thank you very much!

Copy link

@raffienficiaud raffienficiaud commented Nov 2, 2019

Thanks for the quick feedback! (I'll close the issue when this gets merged to master)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
None yet

No branches or pull requests

2 participants