Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to filter test by name (-t, --run_test) if template type contains multiple parameters #223

Closed
TorstenRobitzki opened this issue May 10, 2019 · 7 comments
Assignees
Milestone

Comments

@TorstenRobitzki
Copy link

@TorstenRobitzki TorstenRobitzki commented May 10, 2019

Hello,
I'm used to using the -t option to separate a failing test during TDD and to inspect the cause be using printf() debugging. Usually, I just copy the test name just from the error message. So, if I have an error message like this:

error: in "filter_me<test_parameters<100, 100>>": check sizeof( int ) == sizeof( Param ) has failed [4 != 1]

I'm used to filter the test with: -t "filter_me<test_parameters<100, 100>>". That works fine as long as there is no comma in the test name, as above.

Following example:

#define BOOST_TEST_MODULE "reproducer"
#include <boost/test/included/unit_test.hpp>

#include <tuple>

template < int, int >
struct test_parameters {};

using test_cases = std::tuple<
    test_parameters< 100, 100 >
>;

BOOST_AUTO_TEST_CASE_TEMPLATE( filter_me, Param, test_cases )
{
    BOOST_CHECK_EQUAL( sizeof( int ), sizeof( Param ) );
}

Will generate the error message above and will not allow you to filter the test by the name printed. However, the test can be filter with a wildcard behind the first comma: -t "filter_me<test_parameters<100,*".

I would expect that I can use the reported name of a failing test to filter for exactly that failing test.

Edit: Observed in Boost 1.68 with Apple LLVM version 10.0.0

@raffienficiaud

This comment has been minimized.

Copy link
Member

@raffienficiaud raffienficiaud commented May 13, 2019

Thanks for the report. Right now, two possible solutions that come to my mind:

  1. Either replace every symbols (<, >, ,...) etc by _
  2. Or properly handle the name of the test test_parameters< 100, 100 > in the filtering

As a user, I find test_parameters< 100, 100 > nicer, but it also gives the sense that the framework is able to parse the syntax properly (means that if I have test_parameters<100,100. > it should also work).
Since 1. is much easier than 2. and 2. can also be misleading, and also considering the fact that we already do that for user provided names, I'll go for 1.

What do you think?

@raffienficiaud raffienficiaud added this to the 1.71 milestone May 13, 2019
@TorstenRobitzki

This comment has been minimized.

Copy link
Author

@TorstenRobitzki TorstenRobitzki commented May 13, 2019

Hi, I would just expect the framework to take the exact same name, that it prints. My expectation would be, that when the framework says test "filter_me<test_parameters<100, 100>>" discovered a bug, that I can execute exactly that test by exactly that name ("filter_me<test_parameters<100, 100>>"). (Which by the way, works most of the time).

@raffienficiaud

This comment has been minimized.

Copy link
Member

@raffienficiaud raffienficiaud commented May 13, 2019

We both agree to qualify the current behaviour that you described as buggy, and that was not my question.

Thanks for the report, I'll go for 1.

@TorstenRobitzki

This comment has been minimized.

Copy link
Author

@TorstenRobitzki TorstenRobitzki commented May 14, 2019

Hi,
sorry for not understanding your question! :-) I'm ok with the names of the symbols as they are now, but I would be ok if they would change.

Thank you for looking into this!

@raffienficiaud

This comment has been minimized.

Copy link
Member

@raffienficiaud raffienficiaud commented Oct 30, 2019

I implemented a fix on the branch topic/GH-223-cannot-filter-template-test-cases. Would it be possible for you to try?

raffienficiaud added a commit that referenced this issue Oct 31, 2019
…xt-internal

* topic/GH-223-cannot-filter-template-test-cases:
  Change log
  Documentation updates
  Normalizing test names accross several compilers
  Sanitizing template test cases that contain ',' in their name
@TorstenRobitzki

This comment has been minimized.

Copy link
Author

@TorstenRobitzki TorstenRobitzki commented Nov 2, 2019

Works like a charm. Thank you very much!

@raffienficiaud

This comment has been minimized.

Copy link
Member

@raffienficiaud raffienficiaud commented Nov 2, 2019

Thanks for the quick feedback! (I'll close the issue when this gets merged to master)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.