Be able to run webdev test on travis

[alorenzen]

Right now, travis has support for directly running pub run test with extra flags as part of your travis.yml config.
Once we have the ability to run tests directly via webdev, we should add this support to travis.

Just adding the issue here for tracking.

[kevmoo]

FYI: this request may be handled w/ new features in pkg:test. Stay tuned...

[kevmoo]

We've changed our plans on the test side – not sure how we're going to handle it with webdev – yet.

[natebosch]

Since webdev is already opinionated about using build_runner I can't think of a big downside to implementing webdev test forwarding to pub run build_runner test. Figuring out the UX with flags may be a bit tricky.

Should we start on this? I think our previous hesitation was around uncertainty about whether pub run build_runner test would exist in the long term, but I think with modular kernel the odds of us having a satisfying solution with only pub run test are dropping. @jakemac53 - do you know what are the current downside to pub run build_runner test against pub run test?

Off the top of my head:

• Is there any weirdness around cwd and reading test resources using dart:io?
• How does the failure to build case compare? With pub run test a failing dart2js looks like a test failure right?
• Is being unable to compile only a specific test a blocker?

[jakemac53]

Is there any weirdness around cwd and reading test resources using dart:io?

I believe there is weirdness reading generated (to cache) test files but not others? Would have to check to verify.

How does the failure to build case compare? With pub run test a failing dart2js looks like a test failure right?

You get instead a build failure and it will not attempt to even run the tests.

Is being unable to compile only a specific test a blocker?

I don't think so, with DDC the time to compile a test itself is negligible, and dart2js is likely only being ran for integration tests.

[natebosch]

You get instead a build failure and it will not attempt to even run the tests.

Do we currently not attempt tests for any failed build? I think we'd need to change that. We should allow some steps to fail. I shouldn't be blocked from testing test/foo_test.dart because there is a dart2js compile failure in test/bar_test.dart.

[jakemac53]

Do we currently not attempt tests for any failed build? I think we'd need to change that. We should allow some steps to fail. I shouldn't be blocked from testing test/foo_test.dart because there is a dart2js compile failure in test/bar_test.dart.

We could just invoke the test runner as normal but make sure to still set a non-zero exit code? There was a period of time where the test runner would time out on a test that failed to compile which is I think why we made that decision, but I don't think that is the case any longer?

[kevmoo]

@grouma @jakemac53 – we're running travis tests fine now, right?

[grouma]

webdev has it's own set of tests that run on Travis but I believe this bug is related to webdev test which isn't currently supported.

[grouma]

We don't intend to support webdev test in the near future so closing.