The python backend has surprising goal (non) dependencies.

[jsirois]

In particular, test does not depend on compile. For example, Aurora just installs the python backend:

...
[GLOBAL]
pants_version: 1.1.0-rc7

plugins: [
    'pantsbuild.pants.contrib.python.checks==%(pants_version)s',
  ]

# This is a deprecated option that needs to be set to the empty list here
# to avoid warnings.
# TODO(John Sirois): Remove once we upgrade to 1.3.0+.
default_backend_packages: []

backend_packages: [
    # Pants built-ins:
    'pants.backend.codegen',
    'pants.backend.graph_info',
    'pants.backend.python',
  ]
...

And sees:

$ ./pants --explain test
Goal Execution Order:

test

Goal [TaskRegistrar->Task] Order:

test [test-prep-command->RunTestPrepCommand_test_test_prep_command, test->NoopTest_test, pytest->PytestRun_test_pytest]

As a result, you can run tests green over code that violates compile checks (which includes style checks).

[jsirois]

This issue presents a good excuse to finally (re)implement @wickman's work from long ago introducing a product pipeline centered on PEX chroot products. Roughly:

resolve.python-chroot():PythonChroot -> compile.python-eval(PythonChroot):None
resolve.python-chroot():PythonChroot -> test.pytest(PythonChroot):None
resolve.python-chroot():PythonChroot -> repl.py(PythonChroot):None
resolve.python-chroot():PythonChroot -> run.py(PythonChroot):None
resolve.python-chroot():PythonChroot ->  binary.python-binary-create(PythonChroot):None
resolve.python-chroot():PythonChroot ->  setup-py(PythonChroot):None

Where the representation is <task>(<product deps>*):<product>.

This though still leaves test not depending on compile, so either the new python-chroot task would need to be installed in the compile goal or else a compile-product would need to be introduced for {test.pytest, reply.py, run.py, binary.python-binary-create, setup-py} to depend on - say CompiledPythonChroot. In fact though the current PythonChroot ~product offered up via PythonTask superclass methods is compiled (.pyc are generated) since all chroots, cached and temporary, freeze their PexBuilder.

So I'm thinking introducing compile.python-chroot():PythonChroot instead of resolve.python-chroot():PythonChroot is the way to go here. Since pex currently mixes resolution and assembly / compilation this doesn't even stray from the current underlying model.

So more like:

[compile.python-chroot():PythonChroot -> compile.python-eval(PythonChroot):None] -> test.pytest(PythonChroot):None
...

[kwlzn]

@jsirois we've been thinking about a good candidate for the first E2E v2 engine product pipeline implementation and had mused about tackling the python bits first as lower hanging fruit. Any chance you'd be interested in laying this new pipeline down ~entirely in the v2 engine?

[jsirois]

If you can tolerate my intermittent connectivity, yes. I've been climbing alot lately and that looks to continue. I think I still want to lay down the new python task pipeline in the old engine 1st though using the ad-hoc parallelism constructs available there. This may provide a nice perf baseline for the new engine impl to crush or catch up to as the case may be.

[kwlzn]

sgtm!

[Eric-Arellano]

Closed because the V2 backend has much better scope and no longer registers these things.