Run code coverage on v2 unit tests.

[asherf]

Problem

V2 pytest code coverage doesn't have enough tests so it is fragile.

Solution

Long term we should have tests, but for now, just enabling it and dogfooding it will expose issues

Result

Prevent regressions like: #9914

[asherf]

it repos the bug I reported here: #9914

Aborted CI, will rebase once a fix is merged.

[asherf]

02:42:31 [INFO] Starting: Run Pytest for tests/python/pants_test/backend/graph_info/tasks:sort_targets
02:42:31 [INFO] Starting: Run Pytest for tests/python/pants_test/cache:cache_setup
02:42:31 [INFO] Starting: Run Pytest for tests/python/pants_test/base:exception_sink
02:42:31 [INFO] Starting: Run Pytest for tests/python/pants_test/base:validation
02:42:31 [INFO] Starting: Run Pytest for src/python/pants/backend/pants_info:tests
02:42:31 [INFO] Starting: Run Pytest for src/python/pants/backend/python:tests
02:42:43 [INFO] Completed: Run Pytest for tests/python/pants_test/util:fileutil
02:42:43 [INFO] Starting: Run Pytest for src/python/pants/python:tests
02:42:43 [ERROR] 1 Exception encountered:
Engine traceback:
  in `test` goal
  in Run Pytest for tests/python/pants_test/util:fileutil
Traceback (no traceback):
  <pants native internals>
Exception: Failed to execute process: 9-FAILED_PRECONDITION: "Output type mismatch"
Traceback (most recent call last):
  File "/home/travis/build/pantsbuild/pants/pants.pex/pants/bin/local_pants_runner.py", line 356, in run
    engine_result = self._maybe_run_v2(v2)
  File "/home/travis/build/pantsbuild/pants/pants.pex/pants/bin/local_pants_runner.py", line 236, in _maybe_run_v2
    return self._maybe_run_v2_body(goals, poll=False)
  File "/home/travis/build/pantsbuild/pants/pants.pex/pants/bin/local_pants_runner.py", line 260, in _maybe_run_v2_body
    poll_delay=(0.1 if poll else None),
  File "/home/travis/build/pantsbuild/pants/pants.pex/pants/init/engine_initializer.py", line 256, in run_goal_rules
    goal_product, params, poll=poll, poll_delay=poll_delay
  File "/home/travis/build/pantsbuild/pants/pants.pex/pants/engine/internals/scheduler.py", line 525, in run_goal_rule
    self._raise_on_error([t for _, t in throws])
  File "/home/travis/build/pantsbuild/pants/pants.pex/pants/engine/internals/scheduler.py", line 489, in _raise_on_error
    wrapped_exceptions=tuple(t.exc for t in throws),
pants.engine.internals.scheduler.ExecutionError: 1 Exception encountered:
Engine traceback:
  in `test` goal
  in Run Pytest for tests/python/pants_test/util:fileutil
Traceback (no traceback):
  <pants native internals>
Exception: Failed to execute process: 9-FAILED_PRECONDITION: "Output type mismatch"
02:42:44 [INFO] Completed: Run Pytest for tests/python/pants_test/util:xml_parser
02:42:44 [INFO] Completed: Run Pytest for tests/python/pants_test/util:rwbuf
[=== 00:00 CI BEGINS ===]

@Eric-Arellano @stuhood any idea why this fails ?

[stuhood]

"Output type mismatch" sounds like perhaps declaring and output as a file that is actually a directory: perhaps this:

pants/src/python/pants/backend/python/rules/pytest_coverage.py

Line 291 in 32a72a8

output_files=(".coverage",),

?

[asherf]

"Output type mismatch" sounds like perhaps declaring and output as a file that is actually a directory: perhaps this:

pants/src/python/pants/backend/python/rules/pytest_coverage.py

Line 291 in 32a72a8

output_files=(".coverage",),

?

not sure what you mean... this works fine in other repos.. .coverage is a file (with the raw coverage results)

[Eric-Arellano]

I think the unit tests are timing out with coverage enabled because coverage makes the tests much slower. Run ./v2 test --use-coverage tests/python/pants_test/util:objects, then ./v2 test tests/python/pants_test/util:objects, then ./v2 test --use-coverage tests/python/pants_test/util:objects --coverage-py-filter=pants.util.objects.

It’s very plausible that our plugin is responsible for most of the slowdown. Benjy’s changes mean that we will soon be able to remove the Pants plugin

[asherf]

ok. It is still super weird that it take that long... but I can try to re-push this PR once those changes are merged.

[stuhood]

I think the unit tests are timing out with coverage enabled because coverage makes the tests much slower.

If we have a decent sense for how much overhead coverage adds, we could automatically adjust/multiply timeouts under coverage...

[asherf]

It seems that src/python/pants/engine:tests is the one timing out, I'll run it locally with and without coverage to try to see what is the overhead.

[asherf]

this specific test hangs when running locally/

[asherf]

@stuhood @Eric-Arellano this passes now.
I think this is worth merging.

[stuhood]

I think that I agree. It would be nice to send it somewhere, if possible? But ok to follow up there perhaps? @Eric-Arellano : thoughts?

[asherf]

I can follow up on sending it to coveralls.

[Eric-Arellano]

I think it'd be great to have Coverage data overall. I'm wondering if we should first a) unify the integration and unit test shards as there is no reason to separate them with remote execution, and then b) turn on coverage for the whole project?

It may be misleading to only have coverage data from unit tests.

[stuhood]

I think it'd be great to have Coverage data overall. I'm wondering if we should first a) unify the integration and unit test shards as there is no reason to separate them with remote execution, and then b) turn on coverage for the whole project?

It may be misleading to only have coverage data from unit tests.

+1 to merging the unit/integration shards. But I'm not sure it blocks... depends whether coveralls/cover.io support receiving multiple files per SHA?

[asherf]

I think that I agree. It would be nice to send it somewhere, if possible? But ok to follow up there perhaps? @Eric-Arellano : thoughts?

#10299