Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

this wheel's on fire #61

Closed
o-smirnov opened this issue Feb 28, 2020 · 9 comments · Fixed by #63
Closed

this wheel's on fire #61

o-smirnov opened this issue Feb 28, 2020 · 9 comments · Fixed by #63

Comments

@o-smirnov
Copy link
Contributor

(For those too young to get the reference...)

So this currently happens when I pip install it, see below.

In summary, it tries to build a wheel for radiopadre, falls over because it can't find jupyter, then goes on to run setup.py for radiopadre normally, and succeeds in the end.

I don't fully (or even partially) understand WTF a wheel is, but it's clearly (a) entirely optional, (b) currently un-buildable, and so (c) should be disabled?

(Actually the same thing happens for pyregions or whatsit, but that's not our baby...)

radiopadre.client [29.00s]: young: Building wheels for collected packages: radiopadre
radiopadre.client [29.01s]: young:   Building wheel for radiopadre (setup.py): started
radiopadre.client [29.81s]: young:   Building wheel for radiopadre (setup.py): finished with status 'error'
radiopadre.client [29.82s]: young:   Running setup.py clean for radiopadre
radiopadre.client [29.82s]: young stderr:   ERROR: Command errored out with exit status 1:
radiopadre.client [29.82s]: young stderr:    command: /home/oms/.radiopadre/venv/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-kyxmrynr/radiopadre/setup.py'"'"'; __file__='"'"'/tmp/pip-install-kyxmrynr/radiopadre/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-6w5yazqz
radiopadre.client [29.82s]: young stderr:        cwd: /tmp/pip-install-kyxmrynr/radiopadre/
radiopadre.client [29.83s]: young stderr:   Complete output (126 lines):
radiopadre.client [29.83s]: young stderr:   /home/oms/.radiopadre/venv/lib/python3.6/site-packages/setuptools/dist.py:476: UserWarning: Normalizing '1.0-pre4' to '1.0rc4'
radiopadre.client [29.83s]: young stderr:     normalized_version,
radiopadre.client [29.83s]: young stderr:   running bdist_wheel
radiopadre.client [29.83s]: young stderr:   running build
radiopadre.client [29.83s]: young stderr:   running build_py
radiopadre.client [29.83s]: young stderr:   creating build
radiopadre.client [29.83s]: young stderr:   creating build/lib
radiopadre.client [29.83s]: young stderr:   creating build/lib/radiopadre
radiopadre.client [29.83s]: young stderr:   copying radiopadre/render.py -> build/lib/radiopadre
radiopadre.client [29.83s]: young stderr:   copying radiopadre/layouts.py -> build/lib/radiopadre
radiopadre.client [29.83s]: young stderr:   copying radiopadre/table.py -> build/lib/radiopadre
radiopadre.client [29.83s]: young stderr:   copying radiopadre/datadir.py -> build/lib/radiopadre
radiopadre.client [29.83s]: young stderr:   copying radiopadre/imagefile.py -> build/lib/radiopadre
radiopadre.client [29.83s]: young stderr:   copying radiopadre/file.py -> build/lib/radiopadre
radiopadre.client [29.83s]: young stderr:   copying radiopadre/filelist.py -> build/lib/radiopadre
radiopadre.client [29.83s]: young stderr:   copying radiopadre/pdffile.py -> build/lib/radiopadre
radiopadre.client [29.83s]: young stderr:   copying radiopadre/notebook.py -> build/lib/radiopadre
radiopadre.client [29.83s]: young stderr:   copying radiopadre/htmlfile.py -> build/lib/radiopadre
radiopadre.client [29.83s]: young stderr:   copying radiopadre/textfile.py -> build/lib/radiopadre
radiopadre.client [29.83s]: young stderr:   copying radiopadre/settings_manager.py -> build/lib/radiopadre
radiopadre.client [29.83s]: young stderr:   copying radiopadre/__init__.py -> build/lib/radiopadre
radiopadre.client [29.83s]: young stderr:   copying radiopadre/casatable.py -> build/lib/radiopadre
radiopadre.client [29.83s]: young stderr:   copying radiopadre/fitsfile.py -> build/lib/radiopadre
radiopadre.client [29.83s]: young stderr:   creating build/lib/radiopadre_kernel
radiopadre.client [29.83s]: young stderr:   copying radiopadre_kernel/__init__.py -> build/lib/radiopadre_kernel
radiopadre.client [29.83s]: young stderr:   creating build/lib/radiopadre_utils
radiopadre.client [29.83s]: young stderr:   copying radiopadre_utils/notebook_utils.py -> build/lib/radiopadre_utils
radiopadre.client [29.83s]: young stderr:   copying radiopadre_utils/__init__.py -> build/lib/radiopadre_utils
radiopadre.client [29.83s]: young stderr:   running build_scripts
radiopadre.client [29.83s]: young stderr:   creating build/scripts-3.6
radiopadre.client [29.83s]: young stderr:   copying and adjusting bin/deprecated-install-radiopadre -> build/scripts-3.6
radiopadre.client [29.83s]: young stderr:   copying and adjusting bin/install-js9-colormaps -> build/scripts-3.6
radiopadre.client [29.83s]: young stderr:   copying and adjusting bin/setup-radiopadre-virtualenv -> build/scripts-3.6
radiopadre.client [29.83s]: young stderr:   copying and adjusting bin/deprecated-run-remote-padre -> build/scripts-3.6
radiopadre.client [29.83s]: young stderr:   copying and adjusting bin/bootstrap-radiopadre-install -> build/scripts-3.6
radiopadre.client [29.83s]: young stderr:   copying and adjusting bin/radiopadre-http-server.py -> build/scripts-3.6
radiopadre.client [29.83s]: young stderr:   copying and adjusting bin/deprecated-run-radiopadre -> build/scripts-3.6
radiopadre.client [29.83s]: young stderr:   changing mode of build/scripts-3.6/deprecated-install-radiopadre from 664 to 775
radiopadre.client [29.83s]: young stderr:   changing mode of build/scripts-3.6/install-js9-colormaps from 664 to 775
radiopadre.client [29.83s]: young stderr:   changing mode of build/scripts-3.6/setup-radiopadre-virtualenv from 664 to 775
radiopadre.client [29.83s]: young stderr:   changing mode of build/scripts-3.6/deprecated-run-remote-padre from 664 to 775
radiopadre.client [29.83s]: young stderr:   changing mode of build/scripts-3.6/bootstrap-radiopadre-install from 664 to 775
radiopadre.client [29.83s]: young stderr:   changing mode of build/scripts-3.6/radiopadre-http-server.py from 664 to 775
radiopadre.client [29.83s]: young stderr:   changing mode of build/scripts-3.6/deprecated-run-radiopadre from 664 to 775
radiopadre.client [29.83s]: young stderr:   installing to build/bdist.linux-x86_64/wheel
radiopadre.client [29.83s]: young stderr:   running install
radiopadre.client [29.83s]: young stderr:   running install_lib
radiopadre.client [29.83s]: young stderr:   creating build/bdist.linux-x86_64
radiopadre.client [29.83s]: young stderr:   creating build/bdist.linux-x86_64/wheel
radiopadre.client [29.83s]: young stderr:   creating build/bdist.linux-x86_64/wheel/radiopadre_kernel
radiopadre.client [29.83s]: young stderr:   copying build/lib/radiopadre_kernel/__init__.py -> build/bdist.linux-x86_64/wheel/radiopadre_kernel
radiopadre.client [29.83s]: young stderr:   creating build/bdist.linux-x86_64/wheel/radiopadre_utils
radiopadre.client [29.83s]: young stderr:   copying build/lib/radiopadre_utils/notebook_utils.py -> build/bdist.linux-x86_64/wheel/radiopadre_utils
radiopadre.client [29.83s]: young stderr:   copying build/lib/radiopadre_utils/__init__.py -> build/bdist.linux-x86_64/wheel/radiopadre_utils
radiopadre.client [29.83s]: young stderr:   creating build/bdist.linux-x86_64/wheel/radiopadre
radiopadre.client [29.83s]: young stderr:   copying build/lib/radiopadre/render.py -> build/bdist.linux-x86_64/wheel/radiopadre
radiopadre.client [29.83s]: young stderr:   copying build/lib/radiopadre/layouts.py -> build/bdist.linux-x86_64/wheel/radiopadre
radiopadre.client [29.83s]: young stderr:   copying build/lib/radiopadre/table.py -> build/bdist.linux-x86_64/wheel/radiopadre
radiopadre.client [29.83s]: young stderr:   copying build/lib/radiopadre/datadir.py -> build/bdist.linux-x86_64/wheel/radiopadre
radiopadre.client [29.83s]: young stderr:   copying build/lib/radiopadre/imagefile.py -> build/bdist.linux-x86_64/wheel/radiopadre
radiopadre.client [29.83s]: young stderr:   copying build/lib/radiopadre/file.py -> build/bdist.linux-x86_64/wheel/radiopadre
radiopadre.client [29.83s]: young stderr:   copying build/lib/radiopadre/filelist.py -> build/bdist.linux-x86_64/wheel/radiopadre
radiopadre.client [29.83s]: young stderr:   copying build/lib/radiopadre/pdffile.py -> build/bdist.linux-x86_64/wheel/radiopadre
radiopadre.client [29.83s]: young stderr:   copying build/lib/radiopadre/notebook.py -> build/bdist.linux-x86_64/wheel/radiopadre
radiopadre.client [29.83s]: young stderr:   copying build/lib/radiopadre/htmlfile.py -> build/bdist.linux-x86_64/wheel/radiopadre
radiopadre.client [29.83s]: young stderr:   copying build/lib/radiopadre/textfile.py -> build/bdist.linux-x86_64/wheel/radiopadre
radiopadre.client [29.83s]: young stderr:   copying build/lib/radiopadre/settings_manager.py -> build/bdist.linux-x86_64/wheel/radiopadre
radiopadre.client [29.83s]: young stderr:   copying build/lib/radiopadre/__init__.py -> build/bdist.linux-x86_64/wheel/radiopadre
radiopadre.client [29.83s]: young stderr:   copying build/lib/radiopadre/casatable.py -> build/bdist.linux-x86_64/wheel/radiopadre
radiopadre.client [29.83s]: young stderr:   copying build/lib/radiopadre/fitsfile.py -> build/bdist.linux-x86_64/wheel/radiopadre
radiopadre.client [29.83s]: young stderr:   running install_egg_info
radiopadre.client [29.83s]: young stderr:   running egg_info
radiopadre.client [29.83s]: young stderr:   writing radiopadre.egg-info/PKG-INFO
radiopadre.client [29.83s]: young stderr:   writing dependency_links to radiopadre.egg-info/dependency_links.txt
radiopadre.client [29.83s]: young stderr:   writing requirements to radiopadre.egg-info/requires.txt
radiopadre.client [29.83s]: young stderr:   writing top-level names to radiopadre.egg-info/top_level.txt
radiopadre.client [29.83s]: young stderr:   reading manifest file 'radiopadre.egg-info/SOURCES.txt'
radiopadre.client [29.83s]: young stderr:   reading manifest template 'MANIFEST.in'
radiopadre.client [29.83s]: young stderr:   writing manifest file 'radiopadre.egg-info/SOURCES.txt'
radiopadre.client [29.83s]: young stderr:   Copying radiopadre.egg-info to build/bdist.linux-x86_64/wheel/radiopadre-1.0rc4-py3.6.egg-info
radiopadre.client [29.83s]: young stderr:   running install_scripts
radiopadre.client [29.83s]: young stderr:   creating build/bdist.linux-x86_64/wheel/radiopadre-1.0rc4.data
radiopadre.client [29.84s]: young stderr:   creating build/bdist.linux-x86_64/wheel/radiopadre-1.0rc4.data/scripts
radiopadre.client [29.84s]: young stderr:   copying build/scripts-3.6/deprecated-install-radiopadre -> build/bdist.linux-x86_64/wheel/radiopadre-1.0rc4.data/scripts
radiopadre.client [29.84s]: young stderr:   copying build/scripts-3.6/install-js9-colormaps -> build/bdist.linux-x86_64/wheel/radiopadre-1.0rc4.data/scripts
radiopadre.client [29.84s]: young stderr:   copying build/scripts-3.6/setup-radiopadre-virtualenv -> build/bdist.linux-x86_64/wheel/radiopadre-1.0rc4.data/scripts
radiopadre.client [29.84s]: young stderr:   copying build/scripts-3.6/deprecated-run-remote-padre -> build/bdist.linux-x86_64/wheel/radiopadre-1.0rc4.data/scripts
radiopadre.client [29.84s]: young stderr:   copying build/scripts-3.6/bootstrap-radiopadre-install -> build/bdist.linux-x86_64/wheel/radiopadre-1.0rc4.data/scripts
radiopadre.client [29.84s]: young stderr:   copying build/scripts-3.6/radiopadre-http-server.py -> build/bdist.linux-x86_64/wheel/radiopadre-1.0rc4.data/scripts
radiopadre.client [29.84s]: young stderr:   copying build/scripts-3.6/deprecated-run-radiopadre -> build/bdist.linux-x86_64/wheel/radiopadre-1.0rc4.data/scripts
radiopadre.client [29.84s]: young stderr:   changing mode of build/bdist.linux-x86_64/wheel/radiopadre-1.0rc4.data/scripts/deprecated-install-radiopadre to 775
radiopadre.client [29.84s]: young stderr:   changing mode of build/bdist.linux-x86_64/wheel/radiopadre-1.0rc4.data/scripts/install-js9-colormaps to 775
radiopadre.client [29.84s]: young stderr:   changing mode of build/bdist.linux-x86_64/wheel/radiopadre-1.0rc4.data/scripts/setup-radiopadre-virtualenv to 775
radiopadre.client [29.84s]: young stderr:   changing mode of build/bdist.linux-x86_64/wheel/radiopadre-1.0rc4.data/scripts/deprecated-run-remote-padre to 775
radiopadre.client [29.84s]: young stderr:   changing mode of build/bdist.linux-x86_64/wheel/radiopadre-1.0rc4.data/scripts/bootstrap-radiopadre-install to 775
radiopadre.client [29.84s]: young stderr:   changing mode of build/bdist.linux-x86_64/wheel/radiopadre-1.0rc4.data/scripts/radiopadre-http-server.py to 775
radiopadre.client [29.84s]: young stderr:   changing mode of build/bdist.linux-x86_64/wheel/radiopadre-1.0rc4.data/scripts/deprecated-run-radiopadre to 775
radiopadre.client [29.84s]: young stderr:   WARNING: Package(s) not found: jupyter
radiopadre.client [29.84s]: young stderr:   setup-radiopadre-virtualenv: Setting up radiopadre inside virtualenv /home/oms/.radiopadre/venv
radiopadre.client [29.84s]: young stderr:   Traceback (most recent call last):
radiopadre.client [29.84s]: young stderr:     File "./bin/setup-radiopadre-virtualenv", line 98, in <module>
radiopadre.client [29.84s]: young stderr:       raise subprocess.CalledProcessError(-1, "jupyter", "no jupyter found")
radiopadre.client [29.84s]: young stderr:   subprocess.CalledProcessError: Command 'jupyter' died with <Signals.SIGHUP: 1>.
radiopadre.client [29.84s]: young stderr:   Traceback (most recent call last):
radiopadre.client [29.84s]: young stderr:     File "<string>", line 1, in <module>
radiopadre.client [29.84s]: young stderr:     File "/tmp/pip-install-kyxmrynr/radiopadre/setup.py", line 54, in <module>
radiopadre.client [29.84s]: young stderr:       "Programming Language :: Python :: 3.6",
radiopadre.client [29.84s]: young stderr:     File "/home/oms/.radiopadre/venv/lib/python3.6/site-packages/setuptools/__init__.py", line 144, in setup
radiopadre.client [29.84s]: young stderr:       return distutils.core.setup(**attrs)
radiopadre.client [29.84s]: young stderr:     File "/usr/lib/python3.6/distutils/core.py", line 148, in setup
radiopadre.client [30.17s]: young: Failed to build radiopadre
radiopadre.client [30.84s]: young: Installing collected packages: pillow, numpy, python-dateutil, cycler, pyparsing, kiwisolver, matplotlib, astropy, decorator, networkx, PyWavelets, imageio, scipy, scikit-image, astropy-healpix, reproject, Cython, pyregion, pyavm, shapely, APLpy, backcall, ptyprocess, pexpect, wcwidth, prompt-toolkit, pygments, pickleshare, ipython-genutils, traitlets, parso, jedi, ipython, tornado, pyzmq, jupyter-core, jupyter-client, ipykernel, zipp, importlib-metadata, pyrsistent, attrs, jsonschema, nbformat, webencodings, bleach, entrypoints, pandocfilters, MarkupSafe, jinja2, mistune, testpath, defusedxml, nbconvert, terminado, Send2Trash, prometheus-client, notebook, widgetsnbextension, ipywidgets, jupyter-console, qtconsole, jupyter, argparse, future, python-casacore, radiopadre
radiopadre.client [56.30s]: young:     Running setup.py install for radiopadre: started
radiopadre.client [116.85s]: young:     Running setup.py install for radiopadre: still running...
@Athanaseus
Copy link
Member

Wheels are the new standard of Python distribution and are intended to replace eggs.

TL;DR - A wheel is a zip-format archive with a specially formatted file name and the .whl extension. It contains a single distribution nearly as it would be installed according to PEP 376 with a particular installation scheme. Although a specialized installer is recommended, a wheel file may be installed by simply unpacking into site-packages with the standard 'unzip' tool while preserving enough information to spread its contents out onto their final paths at any later time.
https://www.python.org/dev/peps/pep-0427/

When pip doesn't find a wheel for the requirement, it downloads the source dist and tries to build a wheel from it locally. If it succeeds, the wheel is stored in pip's cache for future reinstalls. if it fails, pip switches to the legacy installation from source dist (invoking python setup.py install).

I found my mistake, the distributions key should be one level above. This should be built and deployed by Travis automatically. At the moment only the dist is uploaded.
https://github.com/ratt-ru/radiopadre/blob/master/.travis.yml#L24

@Athanaseus
Copy link
Member

(For those too young to get the reference...)

Nice jam 🔥

PS: Should I create a PR for the change? similar to ratt-ru/radiopadre-client@b2df9a9

@Athanaseus Athanaseus mentioned this issue Feb 28, 2020
@o-smirnov
Copy link
Contributor Author

Ah ok, cool. I assume we'll need to make a pre5 release for this to make its way into PyPI? If so, please make a b1.0-pre5 branch for this change and make a PR, but don't release yet, I may have other minor fixes to make...

@Athanaseus
Copy link
Member

Duly.

@o-smirnov
Copy link
Contributor Author

Elsewhere you say

I'll look into this in-depth, I'm suspecting that the customized setup.py has to do with this.

...and I think you're right. Or to put it another way, I don't think radiopadre can be shipped as a wheel at all, in this incarnation. Because, as you quote above:

a wheel file may be installed by simply unpacking into site-packages with the standard 'unzip' tool

Which radiopadre clearly cannot be -- for instance, the step it's falling over at at the moment is when the setup script invokes jupyter to register a custom kernel -- well registering a kernel is not something that can be accomplished by dropping a bunch of files into site-packages.

The root cause is, radiopadre is more than a set of Python packages and modules -- there are all these awkward add-ons -- so if wheels cannot accommodate, so be it.

So I would say, don't worry about building a wheel -- rather figure out how to tell it not to build a wheel in the first place (of course I can add a --no-binary option to calls of pip install in the client, but it would be cleaner if radiopadre's own setup.py knew not to try to build a wheel to begin with.)

@Athanaseus
Copy link
Member

Which radiopadre clearly cannot be -- for instance, the step it's falling over at at the moment is when the setup script invokes jupyter to register a custom kernel -- well registering a kernel is not something that can be accomplished by dropping a bunch of files into site-packages.

The root cause is, radiopadre is more than a set of Python packages and modules -- there are all these awkward add-ons -- so if wheels cannot accommodate, so be it.

This is exactly what I was thinking. Before I drop down the wheels please try installing the wheels and test so we can confirm that they will be of no use in this instance.

You can run the following:
python3 setup.py bdist_wheel
If it's within a fresh venv it will raise the jupyter error. Just pip install jupyter.
Note this pre-requirement will be installed by pip using the pyproject.toml file.
Finally, pip install dist/radiopadre-1.0rc5-py3-none-any.whl

@o-smirnov
Copy link
Contributor Author

@Athanaseus, just reading up on wheels -- if you install a wheel, at what point is setup-radiopadre-vritualenv executed?

It sounds to me that a custom script can only executed at wheel build time -- and the build may, in principle, be completely decoupled from wheel installation time, right? A wheel can be cached, and can be installed from cache into a completely new virtual environment, correct?

If my interpretation is correct, then we can't use a wheel yet, as the custom setup script is not guaranteed to run in the new environment...

@Athanaseus
Copy link
Member

... just reading up on wheels -- if you install a wheel, at what point is setup-radiopadre-vritualenv executed?

Yes, this won't get executed.

It sounds to me that a custom script can only executed at wheel build time -- and the build may, in principle, be completely decoupled from wheel installation time, right?

Yes, that's correct.

A wheel can be cached, and can be installed from cache into a completely new virtual environment, correct?

100% sir.

PS: I guess source distribution is the way here.

Thanks 👍

@adrianjhpc
Copy link
Collaborator

You sort of want this:

pypa/packaging-problems#308

Where you'd have a second package that would solve the problems of the first package. But it's a mess...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants