[Pytest] Sort unit tests before running. #9188
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Lunderberg
requested review from
comaniac,
jroesch,
junrushao,
tqchen,
yzhliu and
zhiics
as code owners
areusch
approved these changes
Oct 5, 2021
Lunderberg
force-pushed
the
pytest_sort
branch
3 times, most recently
from
41f8a48
to
85c3abe
Compare
By default, pytest will sort tests to maximize the re-use of fixtures. However, this assumes that all fixtures have an equal cost to generate, and no caches outside of those managed by pytest. A fixture for a `tvm.testing.parameter` is effectively free, while a fixture maintaining a cache of reference data `tvm.testing.utils._fixture_cache` be quite large. Since most of the TVM fixtures are specific to a python function, sort the test ordering by python function, so that tvm.testing.utils._fixture_cache can be cleared sooner rather than later.
When sorting the tests, the order of parametrizations may change. Therefore, the tests checking for automatic target parametrization shouldn't depend on order.
Lunderberg
force-pushed
the
pytest_sort
branch
from
85c3abe
to
ebac851
Compare
masahi
pushed a commit
to Laurawly/tvm-1
that referenced
this pull request
Oct 14, 2021
* [Pytest] Sort unit tests before running. By default, pytest will sort tests to maximize the re-use of fixtures. However, this assumes that all fixtures have an equal cost to generate, and no caches outside of those managed by pytest. A fixture for a `tvm.testing.parameter` is effectively free, while a fixture maintaining a cache of reference data `tvm.testing.utils._fixture_cache` be quite large. Since most of the TVM fixtures are specific to a python function, sort the test ordering by python function, so that tvm.testing.utils._fixture_cache can be cleared sooner rather than later. * Updated TestTargetAutoParametrization When sorting the tests, the order of parametrizations may change. Therefore, the tests checking for automatic target parametrization shouldn't depend on order.
Lunderberg
added a commit
to Lunderberg/tvm
that referenced
this pull request
Oct 22, 2021
A follow-up from apache#9188. The `item.location` tuple contains `(filename, line_number, test_name)`, where the `test_name` includes a string representation of all parameters. This change preserves pytest's sorting of parametrized values within a parametrized test, rather than sorting by strings.
Lunderberg
added a commit
to Lunderberg/tvm
that referenced
this pull request
Oct 22, 2021
A follow-up from apache#9188. The `item.location` tuple contains `(filename, line_number, test_name)`, where the `test_name` includes a string representation of all parameters. This change preserves pytest's sorting of parametrized values within a parametrized test, rather than sorting by strings.
masahi
pushed a commit
that referenced
this pull request
Oct 26, 2021
A follow-up from #9188. The `item.location` tuple contains `(filename, line_number, test_name)`, where the `test_name` includes a string representation of all parameters. This change preserves pytest's sorting of parametrized values within a parametrized test, rather than sorting by strings.
ylc
pushed a commit
to ylc/tvm
that referenced
this pull request
Jan 7, 2022
* [Pytest] Sort unit tests before running. By default, pytest will sort tests to maximize the re-use of fixtures. However, this assumes that all fixtures have an equal cost to generate, and no caches outside of those managed by pytest. A fixture for a `tvm.testing.parameter` is effectively free, while a fixture maintaining a cache of reference data `tvm.testing.utils._fixture_cache` be quite large. Since most of the TVM fixtures are specific to a python function, sort the test ordering by python function, so that tvm.testing.utils._fixture_cache can be cleared sooner rather than later. * Updated TestTargetAutoParametrization When sorting the tests, the order of parametrizations may change. Therefore, the tests checking for automatic target parametrization shouldn't depend on order.
ylc
pushed a commit
to ylc/tvm
that referenced
this pull request
Jan 7, 2022
A follow-up from apache#9188. The `item.location` tuple contains `(filename, line_number, test_name)`, where the `test_name` includes a string representation of all parameters. This change preserves pytest's sorting of parametrized values within a parametrized test, rather than sorting by strings.
ylc
pushed a commit
to ylc/tvm
that referenced
this pull request
Jan 13, 2022
* [Pytest] Sort unit tests before running. By default, pytest will sort tests to maximize the re-use of fixtures. However, this assumes that all fixtures have an equal cost to generate, and no caches outside of those managed by pytest. A fixture for a `tvm.testing.parameter` is effectively free, while a fixture maintaining a cache of reference data `tvm.testing.utils._fixture_cache` be quite large. Since most of the TVM fixtures are specific to a python function, sort the test ordering by python function, so that tvm.testing.utils._fixture_cache can be cleared sooner rather than later. * Updated TestTargetAutoParametrization When sorting the tests, the order of parametrizations may change. Therefore, the tests checking for automatic target parametrization shouldn't depend on order.
ylc
pushed a commit
to ylc/tvm
that referenced
this pull request
Jan 13, 2022
A follow-up from apache#9188. The `item.location` tuple contains `(filename, line_number, test_name)`, where the `test_name` includes a string representation of all parameters. This change preserves pytest's sorting of parametrized values within a parametrized test, rather than sorting by strings.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
By default, pytest will sort tests to maximize the re-use of fixtures. However, this assumes that all fixtures have an equal cost to
generate, and no caches outside of those managed by pytest. A fixture for a
tvm.testing.parameteris effectively free, while a fixture maintaining a cache of reference datatvm.testing.utils._fixture_cachebe quite large.Since most of the TVM fixtures are specific to a python function, sort the test ordering by python function, so that
tvm.testing.utils._fixture_cachecan be cleared sooner rather than later.