Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove stale "pants.pex for integration tests" mechanism #10279

Conversation

stuhood
Copy link
Sponsor Member

@stuhood stuhood commented Jul 7, 2020

Problem

Since #8265, we have been running PantsRunIntegrationTests from loose sources in the repository, which are included via src/python/pants/testutil:int-test's dependency on src/python/pants/bin:pants_local_binary. That change thus removed the need for the pants.pex from our wrapper scripts.

Solution

Remove the "pants.pex for integration tests" mechanism. Post #8625, pants is run from the PYTHONPATH of the test target, which will automatically include either loose sources or a pants_requirement target, depending on whether the test is run in or out of the pantsbuild/pants repo.

Additionally, remove one set of tests that was attempting to test that nested runs of pants do not deadlock. #7654 will eventually allow that issue to be resolved by allowing the concurrent run rather than via any sort of automatic disabling of pantsd, and that is much easier to test via concurrent runs without nesting.

[ci skip-rust-tests]

…un from the PYTHONPATH of the test target, which will automatically include either loose sources or requirements, depending on whether the test is run in or out of the pantsbuild/pants repo.

# Rust tests will be skipped. Delete if not intended.
[ci skip-rust-tests]
Copy link
Member

@jsirois jsirois left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's more cleanup that can happen surrounding not needing this in most spots:

AWS_GET_PANTS_PEX_COMMAND = (

I'm not sure if you're intending that for a follow-up.

@stuhood
Copy link
Sponsor Member Author

stuhood commented Jul 7, 2020

There's more cleanup that can happen surrounding not needing this in most spots:

CI will still use the pants.pex produced by the bootstrap shards in order to avoid needing to compile the engine: so those codepaths are still live, I think (although clearly we have been wrong before!)

@jsirois
Copy link
Member

jsirois commented Jul 7, 2020

CI will still use the pants.pex produced by the bootstrap shards in order to avoid needing to compile the engine: so those codepaths are still live, I think (although clearly we have been wrong before!)

OK. It looks like that's true. It's also unfortunately obscure and a good seed for confusion, but just directly uploading and downloading the .so definitely is out of scope.

Comment on lines -126 to -130
no_regen_pex="${NO_REGEN_PEX:-${TRAVIS}}"
if [[ "${test_goal_used}" == 'true' && "${no_regen_pex}" != 'true' ]]; then
"$HERE/build-support/bin/bootstrap_pants_pex.sh"
echo -e "Finished bootstrapping pants.pex for integration tests.\n" >&2
fi
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yay!

Copy link
Sponsor Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yay indeed!

…k in future versions (via pantsbuild#7654) that implementation will not require nesting to validate: only concurrent runs.
@stuhood stuhood force-pushed the stuhood/remove-pants.pex-for-integration-tests branch from 3703fed to b8169c1 Compare July 7, 2020 19:21
Copy link
Contributor

@Eric-Arellano Eric-Arellano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Glad to see nested runs go.

@stuhood stuhood merged commit 9dd7328 into pantsbuild:master Jul 7, 2020
@stuhood stuhood deleted the stuhood/remove-pants.pex-for-integration-tests branch July 7, 2020 20:38
stuhood added a commit that referenced this pull request Jul 31, 2020
### Problem

Integration tests depend transitively on the `native_engine.so`, which is continuously validated because it must be present to run, and which is bootstrapped before pants starts up. But integration tests were also including the rust sources unnecessarily.

### Solution

Remove the dependency on the rust sources, and the associated target. Also, align the comment about our use of `pants_local_binary` to what we learned in #10279.

[ci skip-rust]
[ci skip-build-wheels]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants