Can't build cross-platform pex containing doit due to file-watching dependencies

[kevincon]

I'm trying to use pex to create a cross-platform (Mac/Linux) executable containing doit, but unfortunately this is failing due to the extras_require dictionary in doit's setup.py file that guards against installing either macfsevents or pyinotify based on the value of sys.platform. It seems that pex is not capable of processing the sys.platform environmental marker with respect to the various target platforms for the PEX being created (see the issue I opened on pex for more info: pex-tool/pex#455).

I think ideally pex would be capable of processing those environmental markers, but according to that issue it seems to be a difficult thing to support. As an alternative, do you have any thoughts about how we could change doit to support this use case?

One idea I had - I don't actually need doit's file watching functionality for my particular use case, and from #101 I understand this isn't available on Windows despite doit otherwise supporting Windows. Could file watching be turned into an optional dependency for doit in setup.py?

Upvote & Fund

• We're using Polar.sh so you can upvote and help fund this issue.
• We receive the funding once the issue is completed & confirmed by you.
• Thank you in advance for helping prioritize & fund our backlog.

[schettino72]

I have never used pex, not familiar with it. As you said, clearly a bug on their side.

Could file watching be turned into an optional dependency for doit in setup.py?

No, sorry.

As an alternative, do you have any thoughts about how we could change doit to support this use case?

Sure 😊

First, can you explain me how you run pex. Can't you just manually modify setup.py when you generate the executable? Or you run this as part of continuous integration system that includes other code?

Please share in more details what are you trying to do and also the error message. I am not really going to study pex but if show me one command or 2 I might give it a try...

[vlcinsky]

We run into similar problem when creating frozen set of python package requirements (e.g. $ pip freeze > requirement.txt, in our case we used pip-compile command from pip-tools) on Linux and then trying to install it on Windows, where pyinotify is simply not available.

One solution would be to have pyinotify/macfsevents as doit extras, so it would not install by default. For use cases, where it is not really used (as is in our case), it would allow/simplify creating platform independent distribution (either with pex, requirements.txt with frozen dependencies etc.).

Options we have are:

1. do nothing, consider this requirement ouf ot doit scope
2. use single doit[watch] extras, which would install file watcher for given platform (if available).
3. use extras doit[linuxwatch] and doit[macwatch] installing file watching library needed (and failing for inapropriate platform).

[schettino72]

I guess it will need some code changes too. because the code assumes a watch library is available or not depending on the platform.

[kevincon]

First, can you explain me how you run pex.

Sure, first install it via pip:

pip install pex 

Then you could run it like so to create a PEX specific for your host platform. pex will pull down dependencies from pypi as needed. Notice that it pulls in MacFSEvents because I'm running this on a Mac:

➜ pex -v doit --console-script doit --output-file doit.pex
  doit 0.31.1 pex :: Resolving distributions
  MacFSEvents 0.8.1
  cloudpickle 0.5.2
pex: Building pex: 666.1ms
pex:   Resolving distributions: 661.4ms
Saving PEX file to doit.pex

I specify --console-script doit as the console script that should be run when the resulting PEX file is executed. See pex -h and https://pex.readthedocs.io/en/stable/ for more info about the available CLI options.

The resulting doit.pex file can be run like so:

➜ ./doit.pex help
doit -- automation tool
http://pydoit.org

Commands
  doit.pex auto              automatically execute tasks when a dependency changes
  doit.pex clean             clean action / remove targets
  doit.pex dumpdb            dump dependency DB
  doit.pex forget            clear successful run status from internal DB
  doit.pex help              show help
  doit.pex ignore            ignore task (skip) on subsequent runs
  doit.pex info              show info about a task
  doit.pex list              list tasks from dodo file
  doit.pex reset-dep         recompute and save the state of file dependencies without executing actions
  doit.pex run               run tasks
  doit.pex strace            use strace to list file_deps and targets
  doit.pex tabcompletion     generate script for tab-completion

  doit.pex help              show help / reference
  doit.pex help task         show help on task dictionary fields
  doit.pex help <command>    show command usage
  doit.pex help <task-name>  show task usage

In theory the doit.pex file can be cp/scp'd to another user's machine and they can use it immediately as-is as long as they have Python installed (i.e. nothing needs to be installed via pip, pip doesn't need to be installed, etc.).

But this is only true if the dependencies bundled into the PEX file aren't platform-specific. It is possible to make a cross-platform PEX that includes platform-specific dependencies, but you have to provide pre-built wheels for each platform and provide to pex 1) the path to the wheels via --find-links and 2) the multiple --platform arguments to have it bundle in the platform-specific dependency wheels.

Building my own wheels for doit and its platform-specific dependencies is sort of a last resort for me because it means those wheels will have to be re-built (on machines/VMs running the specific platforms) anytime a new version of doit is released.

Along those lines, I'm not sure how difficult it would be to start providing platform-specific wheels for doit on pypi, but I think that would also help solve my issue. I figured that might be asking too much, though.

Can't you just manually modify setup.py when you generate the executable?

Yes that's another alternative that may be simpler than building my own platform-specific wheels for doit, but similar to the platform-specific wheel approach I would have to re-do something each time a new verison of doit is released. That could just mean reapplying a patch that removes those lines from setup.py or pulling the new release from upstream into a fork I maintain, but I wanted to explore changing doit directly instead so that I don't have to do something like that.

Or you run this as part of continuous integration system that includes other code?

Please share in more details what are you trying to do

Similar to this article I am using doit as a library to build my own CLI tool that takes advantage of doit's dependency management while providing some additional functionality. I want to provide this CLI tool to my teammates (some of which are Mac users while others use Linux) as a PEX to minimize the work they have to do to start using it.

and also the error message.

Copied from pex-tool/pex#455:

➜  pex -v --disable-cache --python=python3 doit --platform macosx_10.12-x86_64 --platform linux-x86_64 -o doit.pex
  doit 0.31.1 pex :: Resolving distributions :: Packaging MacFSEvents
  cloudpickle 0.5.2
  MacFSEvents 0.8.1
pex: Target package WheelPackage('file:///private/var/folders/b7/qgjhz8kx5fvc6kz_fnx55b600000gq/T/tmpt_08qmf1/MacFSEvents-0.8.1-cp36-cp36m-macosx_10_12_x86_64.whl') is not compatible with CPython-3.6.4 / linux-x86_64
Traceback (most recent call last):
  File "/usr/local/bin/pex", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/site-packages/pex/bin/pex.py", line 663, in main
    pex_builder = build_pex(reqs, options, resolver_options_builder)
  File "/usr/local/lib/python3.6/site-packages/pex/bin/pex.py", line 601, in build_pex
    for dist in resolveds:
  File "/usr/local/lib/python3.6/site-packages/pex/resolver.py", line 438, in resolve_multi
    allow_prereleases):
  File "/usr/local/lib/python3.6/site-packages/pex/resolver.py", line 376, in resolve
    return resolver.resolve(resolvables_from_iterable(requirements, builder))
  File "/usr/local/lib/python3.6/site-packages/pex/resolver.py", line 209, in resolve
    dist = self.build(package, resolvable.options)
  File "/usr/local/lib/python3.6/site-packages/pex/resolver.py", line 177, in build
    raise Untranslateable('Package %s is not translateable by %s' % (package, translator))
pex.resolver.Untranslateable: Package SourcePackage('https://pypi.python.org/packages/28/2e/1ff399cfd2a6a8ebb65152203c920643c2169aa507b2e96559d19baeb7c3/MacFSEvents-0.8.1.tar.gz#md5=45759553d58bab6f7c8d6187fb0fe12f') is not translateable by ChainedTranslator(WheelTranslator, EggTranslator, SourceTranslator)

The actual part to look at in that error is:

Target package WheelPackage('file:///private/var/folders/b7/qgjhz8kx5fvc6kz_fnx55b600000gq/T/tmpt_08qmf1/MacFSEvents-0.8.1-cp36-cp36m-macosx_10_12_x86_64.whl') is not compatible with CPython-3.6.4 / linux-x86_64

One solution would be to have pyinotify/macfsevents as doit extras, so it would not install by default.

Yeah exactly, that's what I meant when I asked "Could file watching be turned into an optional dependency for doit in setup.py?".

Options we have are:
2. use single doit[watch] extras, which would install file watcher for given platform (if available).

That's the approach I would take (assuming it's possible) to keep things simple for users of doit.

I guess it will need some code changes too. because the code assumes a watch library is available or not depending on the platform.

I'm happy to make these changes and submit a PR. Does that sound okay? Is there any particular approach you would want me to take with that (or not take)?

Also thanks very much for the quick responses and helpful discussion.

[schettino72]

wheels

I never bothered about wheels because doit is a python only project. Its dependencies already use wheel so I thought it would not be of great value to justify the effort. And no one ever asked for wheel support. Said that, if anyone volunteer to maintain wheel packages I would gladly accept.

There is no service for creating wheels on all platforms automatically?

setup.py changes

From your command line sample, you use doit code directly from pypi. Can you create a pex from a local repo too?

Since not supporting auto is ok for you. I guess the easiest way for us to have a single setup.py that works in all cases is to rely on some ENV variable. If pex does not set any ENV we could just create one, like DOIT_INSTALL_FILE_WATCH.

On setup.py check if this variable is present or not, If present make extra args an empty dict.
I did not try this... if it works I guess it is a good compromise... what do you think? Can you give this a try?

I guess it will need some code changes too. because the code assumes a watch library is available or not depending on the platform.

I'm happy to make these changes and submit a PR. Does that sound okay? Is there any particular > approach you would want me to take with that (or not take)?

Just to make it work! I guess just need to change from checking from platform, to check platform and import does not fail. Should be trivial. I just do NOT want to see a traceback, a clean error message.

[Kwpolska]

Since doit is a Python-only project, you should be able to run ./setup.py bdist_wheel (after one-time pip install wheel) and get a multi-platform wheel file that you can upload to PyPI (as you do with the source distribution).

[schettino72]

@Kwpolska thanks. that was easy. now doit has wheel packages :)

But I notice that pyinotify has no wheels. @kevincon doit wheel helps in any way?

[kevincon]

doit wheel helps in any way?

The command I was trying to use before (described above and in pex-tool/pex#455) still fails with the same error.

I also tried downloading the .whl file you uploaded to pypi and using the pex --find-links argument to have it use the wheel locally, and that still fails with the same error:

➜  ls
doit-0.31.1-py3-none-any.whl
➜  pex -v --disable-cache --python=python3 --find-links . doit --platform macosx_10.12-x86_64 --platform linux-x86_64 -o doit.pex
  doit 0.31.1 pex :: Resolving distributions :: Packaging MacFSEvents
  MacFSEvents 0.8.1
  cloudpickle 0.5.2
pex: Target package WheelPackage('file:///private/var/folders/b7/qgjhz8kx5fvc6kz_fnx55b600000gq/T/tmpEoniPP/MacFSEvents-0.8.1-cp36-cp36m-macosx_10_12_x86_64.whl') is not compatible with CPython-3.6.4 / linux-x86_64
Traceback (most recent call last):
  File "/usr/local/bin/pex", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/site-packages/pex/bin/pex.py", line 663, in main
    pex_builder = build_pex(reqs, options, resolver_options_builder)
  File "/usr/local/lib/python2.7/site-packages/pex/bin/pex.py", line 601, in build_pex
    for dist in resolveds:
  File "/usr/local/lib/python2.7/site-packages/pex/resolver.py", line 438, in resolve_multi
    allow_prereleases):
  File "/usr/local/lib/python2.7/site-packages/pex/resolver.py", line 376, in resolve
    return resolver.resolve(resolvables_from_iterable(requirements, builder))
  File "/usr/local/lib/python2.7/site-packages/pex/resolver.py", line 209, in resolve
    dist = self.build(package, resolvable.options)
  File "/usr/local/lib/python2.7/site-packages/pex/resolver.py", line 177, in build
    raise Untranslateable('Package %s is not translateable by %s' % (package, translator))
pex.resolver.Untranslateable: Package SourcePackage(u'https://pypi.python.org/packages/28/2e/1ff399cfd2a6a8ebb65152203c920643c2169aa507b2e96559d19baeb7c3/MacFSEvents-0.8.1.tar.gz#md5=45759553d58bab6f7c8d6187fb0fe12f') is not translateable by ChainedTranslator(WheelTranslator, EggTranslator, SourceTranslator)

It seems that the extras_require platform-specific dependencies are still encoded in the wheel even though the project is otherwise pure Python.

Anyway, today I learned that there is a new requirement for my use case that the resulting PEX file works under Python 2, so I guess even if we fixed this in the latest version of doit, that wouldn't help me anymore since doit dropped support for Python 2.

On the bright side, this means I can satisfy my use case by just maintaining my own local copy of doit 0.29.0 (the last version that supported Python 2) that is manually patched to remove the platform-specific file-watching dependencies.

I guess I don't need this issue to be addressed after all, so feel free to close this issue unless others would prefer to keep it open for addressing the pip freeze issue that @vlcinsky described: #251 (comment)

[schettino72]

ok. I didn't think the wheel would help...

[schettino72]

@kevincon btw, it would be nice if you could contribute a 'success story' giving more details of what are you using doit for... and why you chose 'doit'?

[kevincon]

Sure thing, I'll put up a PR to contribute my story when I get a little closer to completing the project.