Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow to run utPLSQL without certain tags (ut.run(a_tags=>'-notThisTag')) #983

Closed
pesse opened this issue Jul 25, 2019 · 12 comments · Fixed by #1008
Labels

Comments

@pesse
Copy link
Member

@pesse pesse commented Jul 25, 2019

Is your feature request related to a problem? Please describe.
Playing around with tags, for me it is often much easier to tag several problematic (or long-running) tests which I don't want to run in several situtations.

Describe the solution you'd like
Would be great to enable ut-runner Tags-parameter to allow -tag:

begin
  ut.run(a_tags=>'-long_running');
end;

This would run all tests which don't have the long_running tag.

begin
  ut.run(a_tags=>'wip,-long_running');
end;

This would run all tests that have the wip tag but not the long_running tag

Additional context
We discussed an alternative approach with logic AND/OR operators (a_tags => 'tag1&tag2|tag3) but the - seems much more useful.

@pesse pesse added the enhancement label Jul 25, 2019
@lwasylow

This comment has been minimized.

Copy link
Member

@lwasylow lwasylow commented Aug 20, 2019

Hi @pesse
So we would have to do some validation to ensure that tag is not starting with '-' as such in annotation cache and treat the ^-.% as reserved.
We would basically execute all tests ( tagged or not ) that do not have a tag -tag or have no tag at all.
Is that correct ?

@jgebal

This comment has been minimized.

Copy link
Member

@jgebal jgebal commented Aug 20, 2019

@lwasylow
I think we should only allow single_word tag names.
It is to be considered if they should be case sensitive or not.
to be honest I'm not sure if they are considered case-sensitive today.

Also, it looks like we silently remove/ignore tags that are not single_words with this line of code:
a_list := ut_utils.convert_collection( ut_utils.filter_list(l_tag_list,ut_utils.gc_word_no_space) );

We should probably put warnings on tag names that are not valid (before we filter them out).
We should also put warning and ignore tag names that start with a "-" symbol.

@alexeyhimself

This comment has been minimized.

Copy link

@alexeyhimself alexeyhimself commented Oct 20, 2019

Also, it looks like we silently remove/ignore tags that are not single_words with this line of code:
a_list := ut_utils.convert_collection( ut_utils.filter_list(l_tag_list,ut_utils.gc_word_no_space) );

This is due to:

Tags should adhere to following rules:

* tags are case sensitive
* tags cannot be an empty string
* tags cannot contain spaces e.g. to create a multi-word tag please use underscores,dashes, dots etc. e.g. test_of_batch
* tags with empty spaces will be ignored during execution
* tags can contain special characters

(given from: http://utplsql.org/utPLSQL/v3.1.7/userguide/annotations.html)

@lwasylow

This comment has been minimized.

Copy link
Member

@lwasylow lwasylow commented Oct 20, 2019

Yes, will fix that and add a warning about it.

@alexeyhimself

This comment has been minimized.

Copy link

@alexeyhimself alexeyhimself commented Oct 20, 2019

I support @pesse 's approach of adding "minus" as a prefix to the tags for the tests that must be excluded:

ut.run(a_tags=>'wip,-long_running');

We would basically execute all tests ( tagged or not ) that do not have a tag -tag or have no tag at all.
Is that correct ?

Yes, as to me this is correct. Consider a use case scenario: test case is marked as

--%tag(long_running)

and other test cases have other tags or have no tags at all.
When I call:

ut.run(a_tags=>'-long_running');

then I expect that will be executed ALL test cases EXCEPT those marked with "long_running" tag.
When I call:

ut.run(a_tags=>'wip,-long_running');

then I expect that will be executed ALL test cases EXCEPT those marked with "long_running" tag PLUS those that marked with "wip".

In a conflict situation, when we have intersecting sets:

--%tag(wip)
procedure test_wip;

--%tag(wip,long_running)
procedure test_wip_long;

--%tag(long_running)
procedure test_long;

and when we call:

ut.run(a_tags=>'wip,-long_running');

or

ut.run(a_tags=>'-long_running,wip');

then I expect that will be ALL test cases EXCEPT those marked with "long_running" tag PLUS those that marked with "wip" EXCEPT those marked with "long_running" again. I.e. in example above will be run only

--%tag(wip)
procedure test_wip;

But now this MY expectation goes against the "OR" rule of annotations (http://utplsql.org/utPLSQL/v3.1.7/userguide/annotations.html):

Framework applies OR logic when resolving a tags so any tests / suites that match at least 
one tag will be included in the test run.

That's why I'd considered a change to the "OR" rule for annotations and made it this way:

Framework applies OR logic when resolving "positive" tags (without "minus" prefix) and 
at the end applies AND NOT logic when resolving "negative" tags (with "minus" prefix). 
So processing for the scope for the test run is in the following order:
1.  ("OR" rule, legacy): any tests / suites that match at least one "positive" tag will be 
included in the test run.
2. ("AND NOT" rule, new feature): any tests / suites that match at least one "negative" tag 
will be excluded from the test run. 
3. (special case caused by new feature): all "positive" vs. "negative" conflicts will be 
resolved to satisfy "negative" tags because the rule "AND NOT" is applied at the end 
of the scope formation.

What would you say?

@jgebal

This comment has been minimized.

Copy link
Member

@jgebal jgebal commented Oct 20, 2019

I would say:
Exclusion is stronger than inclusion and so in your scenario:

--%tag(wip)
--%test
procedure test_wip;

--%tag(wip,long_running)
--%test
procedure test_wip_long;

--%tag(long_running)
--%test
procedure test_long;

and when we call ut.run(a_tags=>'wip,-long_running'); or ut.run(a_tags=>'-long_running,wip');, only the test_wip test gets executed.

This is because of the exclusion logic.

So we say:

  • include only wip items tagged as wip and exclude all long_running

The or logic is applied to inclusion/exclusion list only.
When applying inclusions/exclusions we would use set-based logic that is similar to this operation

select item from items where tag = 'wip'
minus
select item from items where tag = 'long_running'

When you combine filtering by suitepath and filtering by tag and operator is used.
So when you run a suite, only suites that are on the specified suitepath AND contain any of specified tags will be executed.

@jgebal

This comment has been minimized.

Copy link
Member

@jgebal jgebal commented Oct 20, 2019

@lwasylow - I'm already making changes for tags

@alexeyhimself

This comment has been minimized.

Copy link

@alexeyhimself alexeyhimself commented Oct 20, 2019

@jgebal, I totally agree with you saying:

I would say:
Exclusion is stronger than inclusion and so in your scenario:
...

And thank you very much for making changes for tags already! Tags are very important for automation tools usages.

jgebal added a commit that referenced this issue Oct 20, 2019
Resolves #983
jgebal added a commit that referenced this issue Oct 20, 2019
Resolves #983
jgebal added a commit that referenced this issue Oct 21, 2019
@jgebal jgebal closed this in #1008 Oct 23, 2019
@jgebal

This comment has been minimized.

Copy link
Member

@jgebal jgebal commented Oct 23, 2019

@alexeyhimself
Changes are now merged into 3.1.9 develop
Will be leased soon.

@alexeyhimself

This comment has been minimized.

Copy link

@alexeyhimself alexeyhimself commented Oct 23, 2019

@jgebal, thank you very much!

@pesse

This comment has been minimized.

Copy link
Member Author

@pesse pesse commented Oct 23, 2019

Also, cli already supports it. You can just go

utplsql run user/pw@connection --tags=-long_running

with 3.1.8 cli against 3.1.9 core

@alexeyhimself

This comment has been minimized.

Copy link

@alexeyhimself alexeyhimself commented Oct 23, 2019

@pesse, yes, I've seen that in utPLSQL-cli repo! And I really appreciate that! Thank you!
We will try the feature as soon as utPLSQL v3.1.9 appear!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.