Change parameterized testing system to be compatible with unittest

[mthrok]

Summary: The previous implementation of parameterized testing worked by modifying test.common_utils inplace. This doesn't work in general because unittest's contract with test modules is such that it must be able to load the module and run the test itself. Because the previous implementation needed to load the module and modify it, it is incompatible.

Reviewed By: mthrok

Differential Revision: D21964676

[codecov]

Codecov Report

Merging #712 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #712   +/-   ##
=======================================
  Coverage   89.23%   89.23%           
=======================================
  Files          26       26           
  Lines        2350     2350           
=======================================
  Hits         2097     2097           
  Misses        253      253           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c82a7f9...54a2029. Read the comment docs.

[vincentqb]

Summary: The previous implementation of parameterized testing worked by modifying test.common_utils inplace. This doesn't work in general because unittest's contract with test modules is such that it must be able to load the module and run the test itself. Because the previous implementation needed to load the module and modify it, it is incompatible.

Can you give a snippet of the error we were getting on the internal side?

[vincentqb]

Based on offline discussion, this LGTM. I agree modules should not be modified by other modules.

[vincentqb]

Are there other cases behaving like this that use define_test_suites and globals?

[mthrok]

Are there other cases behaving like this that use define_test_suites and globals?

Ya, I am working to remove them now.

[mthrok]

Thanks