Support for tags

[zoechi]

During development I won't care about a lot of tests, especially integration tests (which are often slow(er)). By using tags I could declare sets of unit tests I care about and are likely to break because of what I'm working on.
This would make it more convenient to run tests frequently.
Running all tests is more appropriate for checkin hooks and CI.

This feature could be combined with #6, #7, #14

Examples:

• run only server tests (tag: server)
• run only fast server tests (tags: server,unittest) (as don't run integration tests)
• run all firefox tests (tags: firefox)
• could be combined with The testrunner should report the number of skipped tests and tests within skipped groups #14 (tags: -skip) (exclude skip)
• run tests for a specific feature (tags: authentication)

It should be supported to add tags at group and test level, where tags on a group apply also to tests within that group.

I just found an example where this is already done http://olivinelabs.com/busted/.

[nex3]

I did some thinking about this, and here's how I envision it working:

Tags are defined in roughly the same manner as other metadata. For example,

@Tag("awesome")

void main() {
  group("a neat group", () {
    test("a fun test", () {
      // ...
    }, tag: ["fun", "cool"]);
  }, tag: "neat");
}

Alternately, we could say that tag: always takes a string and that the string is comma-separated for multiple tags. Tags can be arbitrary strings, but to catch typos, they must be declared in the config file (#46):

tags:
  awesome:
  fun:
  cool:
  neat:

Tags can be used from the command-line with the --exclude and --include (--only?) flags, which specify which tags not to run or to run exclusively, respectively.

Tags can also have metadata associated with them in the configuration file. This metadata applies to all tests with that tag. For example:

tags:
  slow: {timeout: 2x}
  ie: {test_on: windows}
  git:
    on_platform:
      windows: {timeout: 2x}

Tags can also be configured differently for different presets (#83). For example:

presets:
  travis:
    tags: {ie: {skip: true}}

[zoechi]

Looks great. I would use --tags and --exclude-tags.

[nex3]

That sounds fine.

[nex3]

Tags can be arbitrary strings, but to catch typos, they must be declared in the config file

Failing to do this should produce a warning. We should also allow tags that are specified on the command-line even if they're not in the config file, to support short-term tags like "solo".

[nex3]

Another thought: the --tags parameter should support the full platform-selector syntax so that users can express complex constraints like "run all fast isolate tests that don't use IO" (as --tags "fast && browser && !io"). This has the side effect of requiring that all tags be representable as Dart identifiers, but I don't think that's onerous.

--tags a,b,c would still work—it would be equivalent to --tags "(a) && (b) && (c)". Similarly, --exclude-tags a,b,c would be equivalent to !(a) && !(b) && !(c).