Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tags for tests and suites #66

Closed
Pazus opened this issue Oct 2, 2016 · 13 comments

Comments

4 participants
@Pazus
Copy link
Member

commented Oct 2, 2016

Introduce annotations to tag tests and suites.
Implement filters on execution procedures to include/exclude suites/tests by tags.

We should discuss if "tag" should be assigned to a suite or "suite_type" with predetermined values fits bettet.

@Pazus Pazus added this to the version3 milestone Oct 2, 2016

@Pazus Pazus modified the milestones: version3.1, version3 Dec 7, 2016

@jgebal jgebal added enhancement and removed discussion labels May 15, 2017

@jgebal jgebal removed the task label Aug 31, 2017

@jgebal jgebal added this to To Do in suite management Oct 25, 2017

@jgebal jgebal removed this from the v3.1.0 milestone Feb 8, 2018

@lwasylow lwasylow self-assigned this Mar 21, 2019

@pesse

This comment has been minimized.

Copy link
Member

commented Mar 22, 2019

If we introduce tags we should probably also give a way to say "run all tests except tag 'x'".
Use-case: "run all tests except tag 'very long-running integration test'"

What do you think?

@jgebal

This comment has been minimized.

Copy link
Member

commented Mar 22, 2019

Absolutely.
With current implementation we can do negative for all:
Run scema except path
Run path except subpath
That should be a seperate feature request probably

@lwasylow lwasylow added the in progress label Apr 4, 2019

@lwasylow

This comment has been minimized.

Copy link
Member

commented Apr 4, 2019

How we would like to invoke it? New parameter? That's simplest option probably. As a path? But then we would need to include a keyword to differeniate between real database object etc.
Maybe something like tag:batch

@jgebal

This comment has been minimized.

Copy link
Member

commented Apr 4, 2019

We currently use:
:path or name how about *tag or some other start character.
I wouldn't want to have separate parameter for it.

@jgebal

This comment has been minimized.

Copy link
Member

commented Apr 4, 2019

Tags would be just amotjlher way of expressing suitepaths to include.
We should have a separate issue for exclusions like:
-name,-:path,-*tag

@Pazus

This comment has been minimized.

Copy link
Member Author

commented Apr 5, 2019

I vote for providing tags as a separate parameter. I think it’s ortogonal to the scope of tests to search and execute. With path we set a scope of tests, with tags we filter it further. So I vote for another overload with tags parameter accepting a single/list of values that are applied on provided path.

@lwasylow

This comment has been minimized.

Copy link
Member

commented Apr 5, 2019

It's a valid point that address a name uniqueness domain as well.

@jgebal

This comment has been minimized.

Copy link
Member

commented Apr 5, 2019

I was thonking a bit differently.
Tags and suites are both used for inclusion of tests just lile we use paths and package/procedure names interchangeably today.

Exclusion would come as a separate feature and would be applicable to all three patterns for specifying elements to run.

Exclusuin could be applied by adding - to element that needs to be excluded.

Tags could be identified in input list by special leading character % just like suitepaths are today :
That way I can specify:

  • ut.run('%payment , -%slow');
  • ut.run(':org.utplsql.stuff, -org.utplsql.stuff.not_this, %utils, %clob');

Maybe we should start by writing potential input param combinations and describe how it should behave before we make final decission on combining it into path parameter or making it separate parameter.

@lwasylow

This comment has been minimized.

Copy link
Member

commented Apr 5, 2019

It's a great idea but I'm worried about complexity of syntax. New parameter is it extra option indeed but is easier to remember than a special keyword.

@lwasylow

This comment has been minimized.

Copy link
Member

commented Apr 13, 2019

Happy with a special character I think the common one is hash so I'm thinking

ut.run('#nightlybatch,#customer_services') run all tests with tag nighltybatch and customer_services

ut.run(':core.batch,#nightlybatch') run all tests on suitepath core.batch with tag nightlybatch

This however can lead to ambiguity when package start with hash.

@lwasylow

This comment has been minimized.

Copy link
Member

commented Apr 16, 2019

Possible options to callout the tags are below.

1.

  • ut.run('ut3$user#.test_expectations_cursor ,#contain')
  • ut.run('ut3$user#,#cursor')
  • ut.run('ut3$user#:utplsql.test_user.expectations,#cursor')

2.

  • ut.run(a_path => 'ut3$user#.test_expectations_cursor',a_tags => 'contain')
  • ut.run(a_path =>'ut3$user#',a_tags =>'cursor')
  • ut.run(a_path =>'ut3$user#:utplsql.test_user.expectations',a_tags =>'cursor')

Option 1

Pros

  • No new parameter, easier call

Cons

  • Complicated syntax, user require more education
  • Possible false positive when we using a schema or package that starts with # and interpretation as tag

Option 2

Pros

  • Simpler syntax and clear identification of option which means user with minimum knowledge can run it
  • Clear differentiate between tag and other objects

Cons

  • Extra parameter , makes a call a bit more complex and require call by name

Given that tags are treated as a way to filter down a execution path to certain area I'm leaning towards the option 2 as better one.

@Pazus

This comment has been minimized.

Copy link
Member Author

commented Apr 16, 2019

I also prefer the second one. It’s simpler and clearer for reading and we read code much more then we write it.
Are there any issues with backwards compatibility for existing scripts?

Just an idea: we have an increasing number of parameter combinations for ut.run procedure. Maybe we can creation ut.run_builder function which returns Object Type implementing a builder pattern with all the defaults set? I wonder how modern IDEs provide code completion for nested object type calls.

@jgebal

This comment has been minimized.

Copy link
Member

commented Apr 16, 2019

After some thinking...
Option 2 is best from my perspective too.

Builder could be an option to go with but I would separate it from this issue.
Change of invocation pattern will require change in every possible consumer. That is a major breaking change.

We could embed the builder in ut_run type directly.
Builder pattern will be however very unnatural to use for most db engineers.

@jgebal jgebal moved this from To Do to In Progress in suite management Jul 13, 2019

@jgebal jgebal moved this from In Progress to Done in suite management Jul 13, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.