Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multitest generates tests into single folder, even if tests have same names #27139

Closed
alsemenov opened this issue Aug 24, 2016 · 1 comment
Closed
Assignees
Labels
area-infrastructure Use area-infrastructure for SDK infrastructure issues, like continuous integration bot changes. P1 A high priority bug; for example, a single project is unusable or has many test failures type-bug Incorrect behavior (everything from a crash to more subtle misbehavior)

Comments

@alsemenov
Copy link
Contributor

Please, make sure you are familiar with https://github.com/dart-lang/sdk/blob/master/tools/testing/dart/multitest.dart

The problem is dicovered with co19 test suite.
Let's imagine we have two tests in the suite:
col9\Language\Expressions\Logical_Boolean_Expressions\syntax_t10.dart

class A {}

main() {
  A || true; /// 01: static type warning, dynamic type error
}

and col9\Language\Expressions\Shift\syntax_t10.dart

class A {}

main() {
  1 >> A; /// 01: static type warning, runtime error
}

The tests have the same file name, but they reside in different folders.
The tests have different expected results: dynamic type error and runtime error

So, when test suite is executed, there is wrong test result:

co19/Language/Expressions/Logical_Boolean_Expressions/syntax_t10/01: RuntimeError

Executing the test manually shows that test result is wrong.
The investigation shows, that test files are generated to the same folder out/platform/generated_tests/co19. So there is only one set of generated tests syntax_t10_none.dart syntax_t10_01.dart with the content, that corresponds to col9\Language\Expressions\Shift\syntax_t10.dart test. If I make the test case keys unique,
then three files are generated syntax_t10_none.dart syntax_t10_key1.dart syntax_t10_key2.dart, which is not good either. (Although this fixes the wrong test result problem).

This problem blocks fixing dart-lang/co19#69 because we have modified a lot of tests and are not able to execute the test suite before committing.

@whesse whesse self-assigned this Aug 24, 2016
@whesse whesse added P1 A high priority bug; for example, a single project is unusable or has many test failures area-infrastructure Use area-infrastructure for SDK infrastructure issues, like continuous integration bot changes. type-bug Incorrect behavior (everything from a crash to more subtle misbehavior) labels Aug 24, 2016
@whesse
Copy link
Contributor

whesse commented Aug 24, 2016

Fixed by https://codereview.chromium.org/2270413002/ .
Let me know if this doesn't work for you.

@whesse whesse closed this as completed Aug 24, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area-infrastructure Use area-infrastructure for SDK infrastructure issues, like continuous integration bot changes. P1 A high priority bug; for example, a single project is unusable or has many test failures type-bug Incorrect behavior (everything from a crash to more subtle misbehavior)
Projects
None yet
Development

No branches or pull requests

2 participants