Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

using a PEX with pytest-xdist #538

Open
tostasmistas opened this issue Aug 16, 2018 · 0 comments
Open

using a PEX with pytest-xdist #538

tostasmistas opened this issue Aug 16, 2018 · 0 comments

Comments

@tostasmistas
Copy link

Hi,

I am trying to use a PEX to run tests from the pytest framework while using the pytest-xdist plugin for distributed testing.

However I believe I am facing an issue similar to this one, as my worker processes are not recognizing packages that are packaged with pex.

My pex build is configured to have pytest as a module entry point (-m pytest) and from this main process I launch the workers in python subprocesses that are instantiated with the following command pytest -d --tx popen//python=python3.

When the workers begin executing I get the error ModuleNotFoundError: No module named '_pytest' which leads me to believe that the pytest package (contained in my pex file) is not being recognized by the subprocesses.

I've tried to bootstrap the pex environment by configuring the workers before their execution in the pytest_configure_node xdist hook with a call to pex_bootstrapper.bootstrap_pex_env('my.pex') but the problem remains.

Any ideas? Thanks
benjyw pushed a commit to benjyw/pex that referenced this issue Nov 30, 2018
@borgstrom @kwlzn
Allow passing a preamble file to the CLI (pex-tool#400)
5b173e3 
This PR adds a new CLI option, -p or --preamble-file, that is used to provide a preamble to the pex builder object.

We use pex via its API in the internal build tooling at NerdWallet (👋, we're neighbors!). We've been leveraging it for multi-platform support, but now that pex-tool#394 / 1.2.9 has landed the only thing missing from the CLI that we leverage is the preamble in the builder, which was pretty easy to hook up. Once this PR has landed we can remove all of the API integration and just rely on the pex CLI!

The CLI option and the PEXBuilder preamble functionality are covered by tests.

Local functional testing:

(pex-foo) 
evanborgstrom@evanborgstrom /tmp/preamble — u:34 j:1 (21:49:54 08.16)
pex-tool#536 ❯❯❯ echo 'print "foo!"' > preamble
(pex-foo) 
evanborgstrom@evanborgstrom /tmp/preamble — u:34 j:1 (21:50:11 08.16)
pex-tool#537 ❯❯❯ pex -p preamble 
foo!
Python 2.7.13 (default, Jun  7 2017, 11:02:53) 
[GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.38)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> 
(pex-foo) 
evanborgstrom@evanborgstrom /tmp/preamble — u:34 j:1 (21:50:18 08.16)
pex-tool#538 ❯❯❯ pex 
Python 2.7.13 (default, Jun  7 2017, 11:02:53) 
[GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.38)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@tostasmistas