-
Notifications
You must be signed in to change notification settings - Fork 221
GTest development
This is a list of recommendations for GTests development (either fixing current tests or writing new tests and porting from the old test infrastructure).
-
TEST_P([TESTSUITE_NAME], [TEST_INFO]) INSTANTIATE_TEST_SUITE_P([TEST_TYPE], [TESTSUITE_NAME], ...);
TEST_TYPE = {Smoke|Full|Perf}
-
Smoke
is for small set of short running tests quickly verifying basic functionality -
Full
is for the main set of tests -
Perf
is for dedicated performance tests
TESTSUITE_NAME = [HW_TYPE]_[NAME]_[DATATYPE]
HW_TYPE = {CPU|GPU}-
CPU
is for the tests which do not require GPU and do not verify kernels
DATATYPE = {FP8..64|BFP16|BFP8|I64|I32|I16|I8|NONE}
-
NONE
is for the tests which are datatype agnostic, they must haveCPU
prefix, like db tests or similar
NAME = {ALGORITHM_DIRECTION|AnyOtherTestNamePattern}
TEST_INFO = {Test|Regression|AnyOtherAdditionalTestInfo}Example:
TEST_P(GPU_Mha_fwd_FP32, Test) INSTANTIATE_TEST_SUITE_P(Smoke, GPU_Mha_fwd_FP32, testing::ValuesIn(GenSmokeTestCases())); INSTANTIATE_TEST_SUITE_P(Full, GPU_Mha_fwd_FP32, testing::ValuesIn(GenFullTestCases()));
Will result in the following testname
Smoke/GPU_Mha_fwd_FP32.Test/0 ... Smoke/GPU_Mha_fwd_FP32.Test/100 Full/GPU_Mha_fwd_FP32.Test/0 ... Full/GPU_Mha_fwd_FP32.Test/10
-
-
All the test parameters must have an appropriate test name generator:
Either by the
operator<<
friend std::ostream& operator<<(std::ostream& os, const TestCaseType& tc) { return os << "Param1: " << tc.param1; }
or by the explicit INSTANTIATE_TEST_SUITE_P test name generator:
INSTANTIATE_TEST_SUITE_P(..., ..., testing::ValuesIn(GenFullTestCases()), [](const auto& case){ return "Param1: " + std::to_string(case.param.param1); });
-
All the tests must not use env variables to adjust it's state. Separated test must be created with appropriate naming instead of using env variables.
-
TestSuite/TestCase must detect compatible HW and automatically skip non-compatible. See #2675 and #2605
-
Use
EXPECT_
macro for non-fatal checks. For the cases when we want to continue the test even if the check failed (for example, various numeric checks). -
Use
ASSERT_
macro for fatal check. For the cases when we can't continue the test execution or the test execution does not make sense (for example, failed handle creation). -
Exit from the test as early as possible. Any test skipping functionality must reside in
void SetUp() override
. For example, right now 198 AddLayerNorm tests take around 20s to skip, but if we move skip routine at the beginning of theSetUp
, it takes 1ms. -
Do not use headers if there is no intention to include it into multiple cpp files. Use anonymous namespace for all the helper functions in cpp file.
-
Appropriately use
testing::Values
,testing::ValuesIn
, etc. See https://google.github.io/googletest/reference/testing.html#param-generators -
Use EXPECT_PRED* instead of EXPECT_TRUE if possible. See https://google.github.io/googletest/reference/assertions.html#predicates
-
Use testsuite shared resources as much as possible. See https://google.github.io/googletest/advanced.html#sharing-resources-between-tests-in-the-same-test-suite
NOTE: If your test fixture defines SetUpTestSuite() or TearDownTestSuite() they must be declared public rather than protected in order to use TEST_P. -
Reset PRNG everywhere to make the tests order-agnostic:
void SetUp() override{ prng::reset_seed(); ... }