Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change behavior of --test and --include so that they are cumulative #4721

Closed
pekkaklarck opened this issue Apr 5, 2023 · 1 comment
Closed

Comments

@pekkaklarck
Copy link
Member

pekkaklarck commented Apr 5, 2023

Currently when you use --suite x --test y --include z, a test is selected only if it is in suite x, has name y and contains a tag z. This is rarely if ever useful. It would typically be enough to just use --test y to select that test.

On the other hand, it would more useful to be able to use --suite x --test y so that all tests in suite x are selected in addition to test y. Similarly, --include x --test y should mean selecting all tests containing tag x in addition to test y.

This change is backwards incompatible, but because the current behavior is pretty strange it's unlikely there are lot of problems. Changing the behavior in Robot Framework 7.0 ought to thus be fine. The plan is to change the behavior with --suite also otherwise and making all these changes in the same release would be good.

Although these options should have a cumulative effect, we probably should interpret --exclude so that tests containing the specified tag aren't selected. That would preserve its behavior when used with other options, most importantly with --include.


UPDATE: We decided to keep the old --suite behavior. See the comment below for more details. Notice also that --suite not affecting which files are parsed was implemented already in RF 6.1 (#4688).

@pekkaklarck pekkaklarck added this to the v7.0 milestone Apr 5, 2023
pekkaklarck added a commit that referenced this issue Apr 5, 2023
- Explain their usage and how they work together better.
- Explain that new Name is not compatible with --suite (#4583)
- Explain how --suite is going to be changed (incl. fixing of the
  above issue) in RF 7.0 (#4720, #4721, #4688)
@pekkaklarck pekkaklarck self-assigned this Sep 11, 2023
@pekkaklarck pekkaklarck changed the title Change behavior of using --suite, --test and --include together so that they are cumulative Change behavior of --test and --include so that they are cumulative Sep 12, 2023
@pekkaklarck
Copy link
Member Author

After discussion on Slack, we decided to keep the old --suite behavior. The main motivation is that robot --suite x root is often used instead of robot root/x.robot to get suite setup and teardown on the root level executed. In such usage robot --suite x --include y root is handy to further filter tests to be run under the x suite. Changing this behavior would be pretty badly backwards incompatible and, even more importantly, would remove a useful functionality.

We will change --test and --include so that they are cumulative, though. In practice --test x --include y will in the future select tests that have name x or contain a tag y. This is useful if you want to run one or few extra named tests in addition to ones selected based on tags.

Earlier --test x --include y meant that only tests with name x and a tag y were selected. This behavior didn't make too much sense, because typically just --test x is enough. If you have multiple tests with name x, it's easier to use the full name like --test root.intermediate.x to pinpoint which one to select instead of using tags. Also this is a backwards incompatible change, but the benefits are bigger than possible problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant