Provide runtime options for positively or negatively selecting tests or suites

[austinbes]

I recently found myself in a position where I wanted to run only a subset of my unit tests (while I was actively iterating on them) because certain tests ran very slowly, and slowed me down. To do this, I ended up changing the suites passed to munit_suite_main(), but it sure would be nice to be able to pass a command-line flag to do this instead.

I'm willing to implement this feature myself, so this issue is primarily to gauge interest level (and get suggestions regarding an API).

I'm thinking something like this:

Support inclusion/exclusion by a psuedo-glob on the fully-qualified test name. Something like --include /suite1/* would include all tests or sub-suites of /suite1. --exclude /suite2/long_test would exclude a single test. --include * would be the equivalent of the current default behavior.

Include and exclude can be chained, with the most recent flag taking precedence. --exclude /suite1/* --include /suite1/important_test would only run /important_test from /suite1.

Any thoughts? Does this seem like it could be a worthwhile feature?

[austinbes]

Now that I think about this a little more, using * has the potential for confusing and undesirable behavior, since it'll be subject to expansion by the shell. I'm not sure what the right syntactical solution is, then...

[freeekanayaka]

I don't care much about which syntax would be used, but this feature would definitely benefit me too (re gauging the interest level).

[61131]

An easier approach might be simply to do a start-of-string anchored string match between the test name provided on the command line and the existing test names. This would provide for some specificity in (the multiple of) tests to be executed without potentially be caught by shell expansion.

[nemequ]

An easier approach might be simply to do a start-of-string anchored string match between the test name provided on the command line and the existing test names.

µnit already does that. If you're careful about how you structure your tests it's pretty handy, but sometimes a way to specify exclusions would be pretty useful too.

[austinbes]

µnit already does that. If you're careful about how you structure your tests it's pretty handy, but sometimes a way to specify exclusions would be pretty useful too.

I just realized this the other day. Closing this issue since the existing feature meets 90% of my use-cases, but if there's serious interest in an exclusion pattern throw some comments down and I'll think about reopening and implementing that.