Create an optional EXPENSIVE label for tests

[jackm321]

Description:
Create a set of tests with the EXPENSIVE label (add_glow_test(EXPENSIVE ...)) and only run those tests with ninja check_expensive. Make all existing OutputCheck tests use the EXPENSIVE label. Build all tests all the time, only run OutputCheck tests when all build requirements were satisfied.

Testing:
ninja check
ninja check_expensive

Documentation:
Comments, Updated docs

[jackm321]

If we like this approach, maybe we should think about adding some more tests to the EXPENSIVE list like:

MLTest: 149.27 sec
BackendCorrectnessTest: 49.35 sec
OperatorTest: 29.83 sec
QuantizationTest: 21.86 sec
ConvTest: 16.13 sec
GradCheckTest: 7.63 sec
BackendTest: 7.49 sec

[nickgg]

Which target does the CI run?

[bertmaher]

For things like MLTest, the biggest issue is that it's not properly parallelized, so that one test is a long pole that blocks everything else. If you run our test suite with proper parallelization (like using gtest-parallel), then everything runs pretty quickly.

I wonder if there's a way to get cmake to use gtest-parallel?

[jackm321]

@nickgg CI would run check and check_expensive
@bertmaher that's interesting about gtest-parallel, we should look into that, I can create a task for it.

[nickgg]

Cool!