Coverage file "branch-check" gets truncated multiple times during coverage testing

[Zalathar]

As observed in #1712, the branch-check file that is written by coverage.py gets truncated multiple times during the coverage test run, instead of just once at the beginning.

(This happens on master too, not just in the v4 PR branch.)

Presumably this means that many of the entries are being lost, so the branch coverage checks aren't as thorough as they appear.

[Zalathar]

The truncation happens here:

hypothesis/hypothesis-python/src/hypothesis/internal/coverage.py

Lines 62 to 65 in 8726b80

	if IN_COVERAGE_TESTS:
	with open("branch-check", "w"):
	pass
	written = set() # type: Set[Tuple[str, bool]]

This occurs at the top-level of the module, and is presumably only supposed to happen once. But in reality it runs multiple times.

[Zalathar]

I created a global flag that gets set when the truncation occurs.

Future truncations see this flag as unset, which suggests that the module is being imported multiple times from different contexts that don't share an import cache.

[Zalathar]

I also tried removing the automatic truncation entirely, and truncated branch-check manually instead.

This resulted in ~14 “always True” complaints, which suggests that this issue has been concealing holes in branch-check coverage.

[Zalathar]

I modified coverage.py to append sys.argv[0] to a separate file whenever it truncates branch-check. These are the results:

$HOME/hypothesis/hypothesis-python/.tox/coverage/lib/python3.6/site-packages/pytest.py
/private/var/folders/54/vs3dcnhx7rlddhj58w5mzlz40000gn/T/pytest-of-stuart/pytest-13/test_puts_the_database_in_the_0/assertlocation.py
/private/var/folders/54/vs3dcnhx7rlddhj58w5mzlz40000gn/T/pytest-of-stuart/pytest-13/test_picks_up_verbosity_from_e0/printdebug.py

Looks like the problem is triggered by subprocess tests that import coverage.py and don't understand that they shouldn't truncate branch-check.

[Zalathar]

I think the solution here is to remove the truncation from coverage.py, and put an rm -f branch-check command in the tox configuration instead.

(Subprocess invocations still won't share a cache of previously-written lines, but that shouldn't affect correctness. The checker script can handle duplication.)