Testing: run Coverage report only on CircleCI

[humitos]

I've been hitting an issue while debugging test cases that it doesn't stop exactly where I put the breakpoint. This has been pretty annoying since I'm in the middle of an inner function and I have to do multiple steps forward to find the exact place where I put my breakpoint.

Today I found this problem is due to an incompatibility of pytest-cov and the debuggers: https://pytest-cov.readthedocs.io/en/latest/debuggers.html.

[humitos]

Take a look at this example:

diff --git a/readthedocs/rtd_tests/tests/test_profile_views.py b/readthedocs/rtd_tests/tests/test_profile_views.py
index 42298cde2..fb99c597f 100644
--- a/readthedocs/rtd_tests/tests/test_profile_views.py
+++ b/readthedocs/rtd_tests/tests/test_profile_views.py
@@ -37,6 +37,7 @@ class ProfileViewsTest(TestCase):
         )
         self.assertTrue(resp.status_code, 200)
 
+        import pdb; pdb.set_trace()
         self.user.refresh_from_db()
         self.user.profile.refresh_from_db()
         self.assertEqual(self.user.first_name, 'Read')

Running the tests using coverage

When it stops, I'm inside refresh_from_db 🤷🏼

root@ce473d3604cd:/usr/src/app/checkouts/readthedocs.org# tox -e py310 -- --nomigrations -k test_edit_profile
py310: commands[0]> sh -c 'export DJANGO_SETTINGS_MODULE=readthedocs.settings.test; pytest --cov-report=xml --cov-config .coveragerc --cov=. --pyargs readthedocs --suppress-no-test-exit-code -m "not proxito and not embed_api" --nomigrations -k test_edit_profile'

...

readthedocs/rtd_tests/tests/test_profile_views.py 

...

--Call--
> /usr/src/app/checkouts/readthedocs.org/.tox/py310/lib/python3.10/site-packages/django/db/models/base.py(607)refresh_from_db()
-> def refresh_from_db(self, using=None, fields=None):
(Pdb) l
602  	       return {
603  	           f.attname for f in self._meta.concrete_fields
604  	           if f.attname not in self.__dict__
605  	       }
606  	
607  ->	   def refresh_from_db(self, using=None, fields=None):
608  	       """
609  	       Reload field values from the database.
610  	
611  	       By default, the reloading happens from the database this instance was
612  	       loaded from, or by the read router if this instance wasn't loaded from
(Pdb)

Running tests without using coverage (this PR)

I'm stopped exactly where I put the pdb 💯

root@ce473d3604cd:/usr/src/app/checkouts/readthedocs.org# tox -e py310 -- --nomigrations -k test_edit_profile
py310: commands[0]> sh -c 'export DJANGO_SETTINGS_MODULE=readthedocs.settings.test; pytest --pyargs readthedocs --suppress-no-test-exit-code -m "not proxito and not embed_api" --nomigrations -k test_edit_profile'

....

readthedocs/rtd_tests/tests/test_profile_views.py 

...

> /usr/src/app/checkouts/readthedocs.org/readthedocs/rtd_tests/tests/test_profile_views.py(41)test_edit_profile()
-> self.user.refresh_from_db()
(Pdb) l
 36  	           },
 37  	       )
 38  	       self.assertTrue(resp.status_code, 200)
 39  	
 40  	       import pdb; pdb.set_trace()
 41  ->	       self.user.refresh_from_db()
 42  	       self.user.profile.refresh_from_db()
 43  	       self.assertEqual(self.user.first_name, 'Read')
 44  	       self.assertEqual(self.user.last_name, 'Docs')
 45  	       self.assertEqual(self.user.profile.homepage, 'readthedocs.org')
 46  	
(Pdb)

[stsewd]

I'm -1 on removing our coverage. We should use it more instead of removing it.
Conditionally running coverage could be supported with an env var or similar {env:COVERAGEOFF:--cov-report=xml...}.

[humitos]

I like the idea of using an env variable to decide whether or not run it. I will update this PR to run it by default only on CircleCI