New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for tags #16

Closed
zoechi opened this Issue Feb 20, 2015 · 6 comments

Comments

Projects
None yet
2 participants
@zoechi

zoechi commented Feb 20, 2015

During development I won't care about a lot of tests, especially integration tests (which are often slow(er)). By using tags I could declare sets of unit tests I care about and are likely to break because of what I'm working on.
This would make it more convenient to run tests frequently.
Running all tests is more appropriate for checkin hooks and CI.

This feature could be combined with #6, #7, #14

Examples:

  • run only server tests (tag: server)
  • run only fast server tests (tags: server,unittest) (as don't run integration tests)
  • run all firefox tests (tags: firefox)
  • could be combined with #14 (tags: -skip) (exclude skip)
  • run tests for a specific feature (tags: authentication)

It should be supported to add tags at group and test level, where tags on a group apply also to tests within that group.

I just found an example where this is already done http://olivinelabs.com/busted/.

@nex3 nex3 added the enhancement label Feb 20, 2015

@nex3 nex3 modified the milestone: 1.0.0 Mar 31, 2015

@nex3

This comment has been minimized.

Show comment
Hide comment
@nex3

nex3 Apr 22, 2015

Member

I did some thinking about this, and here's how I envision it working:

Tags are defined in roughly the same manner as other metadata. For example,

@Tag("awesome")

void main() {
  group("a neat group", () {
    test("a fun test", () {
      // ...
    }, tag: ["fun", "cool"]);
  }, tag: "neat");
}

Alternately, we could say that tag: always takes a string and that the string is comma-separated for multiple tags. Tags can be arbitrary strings, but to catch typos, they must be declared in the config file (#46):

tags:
  awesome:
  fun:
  cool:
  neat:

Tags can be used from the command-line with the --exclude and --include (--only?) flags, which specify which tags not to run or to run exclusively, respectively.

Tags can also have metadata associated with them in the configuration file. This metadata applies to all tests with that tag. For example:

tags:
  slow: {timeout: 2x}
  ie: {test_on: windows}
  git:
    on_platform:
      windows: {timeout: 2x}

Tags can also be configured differently for different presets (#83). For example:

presets:
  travis:
    tags: {ie: {skip: true}}
Member

nex3 commented Apr 22, 2015

I did some thinking about this, and here's how I envision it working:

Tags are defined in roughly the same manner as other metadata. For example,

@Tag("awesome")

void main() {
  group("a neat group", () {
    test("a fun test", () {
      // ...
    }, tag: ["fun", "cool"]);
  }, tag: "neat");
}

Alternately, we could say that tag: always takes a string and that the string is comma-separated for multiple tags. Tags can be arbitrary strings, but to catch typos, they must be declared in the config file (#46):

tags:
  awesome:
  fun:
  cool:
  neat:

Tags can be used from the command-line with the --exclude and --include (--only?) flags, which specify which tags not to run or to run exclusively, respectively.

Tags can also have metadata associated with them in the configuration file. This metadata applies to all tests with that tag. For example:

tags:
  slow: {timeout: 2x}
  ie: {test_on: windows}
  git:
    on_platform:
      windows: {timeout: 2x}

Tags can also be configured differently for different presets (#83). For example:

presets:
  travis:
    tags: {ie: {skip: true}}
@zoechi

This comment has been minimized.

Show comment
Hide comment
@zoechi

zoechi Apr 23, 2015

Looks great. I would use --tags and --exclude-tags.

zoechi commented Apr 23, 2015

Looks great. I would use --tags and --exclude-tags.

@nex3

This comment has been minimized.

Show comment
Hide comment
@nex3

nex3 Apr 23, 2015

Member

That sounds fine.

Member

nex3 commented Apr 23, 2015

That sounds fine.

@nex3

This comment has been minimized.

Show comment
Hide comment
@nex3

nex3 Oct 13, 2015

Member

Tags can be arbitrary strings, but to catch typos, they must be declared in the config file

Failing to do this should produce a warning. We should also allow tags that are specified on the command-line even if they're not in the config file, to support short-term tags like "solo".

Member

nex3 commented Oct 13, 2015

Tags can be arbitrary strings, but to catch typos, they must be declared in the config file

Failing to do this should produce a warning. We should also allow tags that are specified on the command-line even if they're not in the config file, to support short-term tags like "solo".

@nex3

This comment has been minimized.

Show comment
Hide comment
@nex3

nex3 Dec 1, 2015

Member

Another thought: the --tags parameter should support the full platform-selector syntax so that users can express complex constraints like "run all fast isolate tests that don't use IO" (as --tags "fast && browser && !io"). This has the side effect of requiring that all tags be representable as Dart identifiers, but I don't think that's onerous.

--tags a,b,c would still work—it would be equivalent to --tags "(a) && (b) && (c)". Similarly, --exclude-tags a,b,c would be equivalent to !(a) && !(b) && !(c).

Member

nex3 commented Dec 1, 2015

Another thought: the --tags parameter should support the full platform-selector syntax so that users can express complex constraints like "run all fast isolate tests that don't use IO" (as --tags "fast && browser && !io"). This has the side effect of requiring that all tags be representable as Dart identifiers, but I don't think that's onerous.

--tags a,b,c would still work—it would be equivalent to --tags "(a) && (b) && (c)". Similarly, --exclude-tags a,b,c would be equivalent to !(a) && !(b) && !(c).

nex3 added a commit that referenced this issue Dec 4, 2015

Disallow non-hyphenated-identifier tag names.
This provides forward-compatibility for when we eventually add support
for full expressions when selecting tags.

See #16

R=kevmoo@google.com

Review URL: https://codereview.chromium.org//1491383003 .

@nex3 nex3 changed the title from It should be possible to add tags to tests and include/exclude tests with specified tags to Support for tags Jan 12, 2016

@nex3

This comment has been minimized.

Show comment
Hide comment
@nex3

nex3 Feb 5, 2016

Member

Initial configuration file support has landed, so this is no longer blocked.

Member

nex3 commented Feb 5, 2016

Initial configuration file support has landed, so this is no longer blocked.

@nex3 nex3 removed the blocked label Feb 5, 2016

nex3 added a commit that referenced this issue Feb 16, 2016

nex3 added a commit that referenced this issue Feb 17, 2016

@nex3 nex3 closed this in 832d104 Feb 19, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment