Make --platform/--interpreter multi-options

[wickman]

This is to make it easier to build multi-platform pex files a la pants.

[lclarkmichalek]

Is pants the only way to do this at the moment? I can't seem to find any documentation on it

[wickman]

sadly yes but this should be a pretty straightforward PR

[lclarkmichalek]

If I understand it correctly, it would be a case of iterating through the combinations and creating resolvers for each, installing packages from those resolvers and then only after that, creating/freezing the pex file? I can't see where Pants is handling its multiplatform stuff

[jjjake]

@wickman is anybody working on this? I'm not very familiar with the pex codebase, but I would like to take a try at implementing this if nobody else is.

We're using pex at the Internet Archive to generate binaries for our ia command-line tool (https://github.com/jjjake/internetarchive). I would like to add some dependencies that would require platform specific distributions, and it would be great if we could package everything into a single binary that could be used on multiple platforms.

Thanks for all the work on pex, it's been a big win for us at the Archive being able to easily package the ia command-line tool as a single binary that works on many platforms. : )

[kwlzn]

@jjjake I've not heard of anyone planning to tackle this in pex anytime soon, so I'd say feel free to take a stab at it.

btw, per the earlier conversation on this thread here's the relevant code from pants: https://github.com/pantsbuild/pants/blob/97c0bdba3eb39ee8dd4836a6547bed7e6cf937d6/src/python/pants/backend/python/python_chroot.py#L241

it'd be ideal if we could subsume this functionality directly into pex at the library level (e.g. a pluralized platforms= kwarg to pex.resolver.resolve etc), such that pants could utilize baked-in multi-platform support via the API and drop its own multi-platform logic.

[jjjake]

Awesome. Thank you for the details @kwlzn, those are helpful. I'll go ahead and take a stab at it.

[jjjake]

@kwlzn here's my first pass at making --platform/--interpreter multi-options: jjjake@d948fa9

Please let me know if I'm on the right track, and I'll work on cleaning things up, docstrings and tests for this change. Thanks!

[kwlzn]

@jjjake assuming this is functional, the approach looks reasonable to me at a high level. I'll hold off on smaller/style comments until you have a pull request ready to go.

I'd recommend cutting a branch for your changes before you get much further which should lead to an easier initial PR - then once you have a PR up you can utilize our TravisCI integration for test results (and keep pushing updates from your branch to re-run CI as needed during iteration).

[mzdaniel]

Commit 572f7f1 including jjjake's is now passing travis tests (https://travis-ci.org/mzdanieltest/pex/builds/132036887), though I noticed cache_dir has a different meaning (it is a list instead of a string). In particular, my original pex call doesn't run:
pex -v --disable-cache --no-index -f dist -c sphinxserve -o pex/scripts/sphinxserve dist/*
and raises:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.4/bin/pex", line 11, in <module>
    sys.exit(main())
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/pex/bin/pex.py", line 537, in main
    options.cache_dir = make_relative_to_root(options.cache_dir)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/pex/bin/pex.py", line 518, in make_relative_to_root
    return os.path.normpath(path.format(pex_root=ENV.PEX_ROOT))
AttributeError: 'list' object has no attribute 'format'

What would be the correct call to create a multi interpreter and platform?
Do the the different binary wheels need to be in different directories?

[kwlzn]

@mzdaniel this was actually a bug, fixed in ec63631 today. rebasing master should include this fix so you can continue testing.

btw, once you think your continuation of jjjake's change is ready to go, feel free to craft a new PR and tag that with the ready-for-review label and I'll take a look.

[kramasamy]

We have integrated pex with bazel for Heron compilation and we want to support multi-platform pex files for distribution. We are augmenting existing pex binary rules to take in platform option for generating these pex files. As I understand, the functionality of multi platform is with pants. Wondering if you could make this functionality available with pex and it will help us seamless integrate with bazel as well.

[kwlzn]

@kramasamy I've landed support for this in master - should be able to cut a release tomorrow for you guys to begin using.

[kwlzn]

@kramasamy release is out: https://pypi.python.org/pypi/pex/1.2.9

[kramasamy]

Awesome. Thanks Kris. Let me integrate this into pex rules.

…

On Jul 24, 2017, at 9:46 PM, Kris Wilson ***@***.***> wrote: @kramasamy <https://github.com/kramasamy> release is out. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#68 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAWcRErt0gyzGftKSEMXq2XTCnoo5Fu5ks5sRXMPgaJpZM4EAN9t>.