ENH:Create universal wheel on PyPi for pyicu
#39
Conversation
…eel. Should pip install of pyicu, and fix the polyglot python 2.7 error on MacOSX.
|
On Dec 11, 2016, at 13:46, Linwood Creekmore ***@***.***> wrote:
If this is too instructive, forgive me. Just wanted to require minimal effort from original author since this is me asking you to add something to your repository.
Summary of Merge Request
@ovalhub, this merge request creates a “Universal Wheel” to resolve an issue on the MacOS where pyicu's inability to fully install via pip install pyicu causes an error with polyglot, which is a multilingual NLP library created by @aboSamoor .
Details of Request
Request @ovalhub accept merge request, and run the following commands to update the PyICU PyPi packaging. This will include a universal wheel in the File section.
I've never heard of such "wheels" before but reading the link above I see this:
Only use the --universal setting, if:
Your project runs on Python 2 and 3 with no changes (i.e. it does not require 2to3).
Your project does not have any C extensions.
PyICU has no python code in it (except tests), it's a pure C++ module that links with ICU and Python. In other words, it's C extension. This would seem to contradict item 2. quoted above, no ?
And 2to3 is needed to run tests on Python 3, contradicting 1. if that applies to tests.
Please explain, instruct me, why all this makes sense and what it achieves. On my end, I'll try pip install to see if I can reproduce the failure you reported.
Thanks !
Andi..
… The problem encountered and fix are discussed in detail here.
What's the Change?
Just created a simple setup.cfg file as described in the Universal Wheels section of the Python Packaging User Guide. The exact content of the simple setup.cfg file is here.
How to "update" your Packaging
After this file is in the master pyicu repository, you run the following commands, as described in the Packaging User Guide:
>>> cd pyicu
>>> python setup.py bdist_wheel --universal # create the universal wheel
>>> python setup.py bdist_wheel # creates platform specific wheel just in case
Now, the last step is just uploading the distribution to PyPi. If you use twine (recommended approach, see here for security discussion):
>>> twine upload dist/*
Or the setuptools alternative (not recommended, see here for security discussion):
>>> python setup.py sdist bdist_wheel upload
This should solve one of the problems discussed in the polyglot issue.
You can view, comment on, or merge this pull request online at:
#39
Commit Summary
Creating a "setup.cfg" file and "2to3note" for the universal pyicu wheel. Should pip install of pyicu, and fix the polyglot python 2.7 error on MacOSX.
File Changes
A setup.cfg (5)
A test/2to3.note (1)
Patch Links:
https://github.com/ovalhub/pyicu/pull/39.patch
https://github.com/ovalhub/pyicu/pull/39.diff
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
|
|
No problem @ovalhub . Thanks for considering this. I'll explain as best as I can (not a Comp Sci person). I figured this out by trouble shooting and just reading tons. What does a "wheel" accomplish?It resolved the install issues I had for a project that uses pyicu as a dependency; will likely resolve other issues, but also, aligns your I created two workarounds for You may ask, but who cares about Python wheels? Why does
|
|
On Dec 11, 2016, at 15:40, Linwood Creekmore ***@***.***> wrote:
No problem. Thanks for considering this. I'll explain as best as I can (not a Comp Sci person). I figured this out by trouble shooting and just reading tons.
What does a "wheel" accomplish?
It resolved the install issues I had for a project that uses pyicu as a dependency; will likely resolve other issues, but also, aligns your pyicu project to the new PEP Distribution Standard (not so new since standard was approved in Feb 2013) , which uses the wheel binary package format because it frees installers from having to know about the build system, saves time by amortizing compile time over many installations, and removes the need to install a build system in the target environment.
Aah, so it's a way to distribute binaries. Well, this can't work with PyICU since it can't be a universal binary. Each binary PyICU is different depending on:
- version of ICU compiled and linked against (a dozen or so)
- version of OS (let's say three: linux, macos, windows)
- version of Python (let's say 3: 2.7.x, 3.x, 3.y,
- version of PyPI (if used)
where 3.x is before the new string implementation and 3.y after)
That's at minimum 12*3*3*2 = 216 different binary versions. Including them all in a wheel would make for huge downloads for everyone and lots's of users would still not be covered by any of these 216 versions. PyICU is not pure Python, it's pure C++, there is no python code in it.
More comments below.
I created two workarounds for pyicu to successfully install on Python 2.7 and Python 3.5, but I think this small merge would help the problem (you would add a universal wheel to your PyPi distribution).
You may ask, but who cares about Python wheels? Why does pyicu need it?
The webpage pythonwheels describes this benefit best, but also directly relates to pyicu as it's a C extension as you said above. This is text pasted from the webpage, but note the bullet I bolded below):
What are wheels?
Wheels are the new standard of python distribution and are intended to replace eggs. Support is offered in pip >= 1.4 and setuptools >= 0.8.
Advantages of wheels
Faster installation for pure python and native C extension packages.
Avoids arbitrary code execution for installation. (Avoids setup.py)
Installation of a C extension does not require a compiler on Windows or OS X.
Sadly, it does require a compiler for PyICU because the binary version of PyICU depends on the version of the ICU library it is compiled against and linked with. Each version of ICU has different APIs, old ones disappear, new ones appear with each release. One does need a working compiler and linker for building PyICU (and ICU before that).
Allows better caching for testing and continuous integration.
Creates .pyc files as part of installation to ensure they match the python interpreter used.
There are no .pyc files in PyICU, it has no python code (except for tests).
More consistent installs across platforms and machines.
I believe adding this file will fix the install problem for the scientific community folks who use anaconda. Moreover, it will help the folks who want to use polyglot, which depends on pyicu.
Yes, but then you need to create a specific distribution of PyICU for Anaconda / Polyglot users where the version of ICU is frozen (to 54.1, right ?), not the latest version of ICU others users expect it to wrap (like 58.1, currently). That would be a binary fork of PyICU specific to this use case. You're welcome to create one.
I created two work to install pyicu on a MacOS machine running Anaconda and using Python 2.7 or Python 3.5
For what version of ICU ?
Recreating my problem
To recreate the problem I experienced (I'm on a Mac), I think you may need to have Anaconda installed. It's a scientific computing distribution.
Here are the steps for me:
pip install polyglot
python
from polyglot.text import Text, Word
That always gives me an error like this:
Library not loaded: libicui18n.54.dylib which comes from something called docs, line 23:
Do you have ICU 54 installed on your machine before you launch that command ?
Andi..
… Here is a screen capture of the traceback:
Hope this explains it!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
|
|
@ovalhub , yes, I did tests with 58.1 and 54.1.0, and 54.1.1. Only 54.1.1 works. Weird, but that's what i found through trial and error. this gets me icu 54.1.1: conda install -c ccordoba12 icu=54.1which is from here: https://anaconda.org/ccordoba12/icu None of these work (which is why it was confusing..lots of trial and error): conda install icu # installs 54.1.0, doesn't work
conda install -c conda-forge icu=58.1 # doesn't work
conda install -c anaconda icu=57.1 # doesn't workEven tried Homebrew, but Homebrew only install 58.1: brew install icu4cAll gave me that But, even with that, when I tried to run |
|
I think the main problem is Another Option (more simple)When I use pip to download your tar file, you have the
Then, you would run these commands, and redistribute to PyPi. >>> cd pyicu
>>> python setup.py bdist_wheel --universal # create the universal wheel
>>> python setup.py bdist_wheel # creates platform specific wheel just in case
>>> twine upload dist/*
#or
>>> python setup.py sdist bdist_wheel uploadI tested, and the Let me know if I can assist anymore. Thanks again for considering. |
|
On Dec 11, 2016, at 16:32, Linwood Creekmore ***@***.***> wrote:
I think the main problem is pip install; something about the easy_install works, while it doesn't work in the pip. To be clear, I don't get an error on the pip install pyicu, but when I try to import something from pyicu, it gives an error. However, when I install pyicu with easy_install, I don't get that error.
Another Option (more simple)
When I use pip to download your tar file, you have the setup.cfg. I believe adding a simple line as you see in this pic would do the trick:
Did you understand my points about why PyICU can't be universal = 1 ?
Then, you would run these commands, and redistribute to PyPi.
I cannot distribute a PyICU binary on PyPI as it wouldn't work for any people that aren't running the same combination compiler, OS, ICU, Python that I used to build it. Did my earlier points about that make sense ?
Andi..
… >>> cd pyicu
>>> python setup.py bdist_wheel --universal # create the universal wheel
>>> python setup.py bdist_wheel # creates platform specific wheel just in case
>>> twine upload dist/*
#or
>>> python setup.py sdist bdist_wheel upload
Let me know if I can assist anymore. Thanks again for considering.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
|
|
I'm Sorry @ovalhub , I missed the earlier comments. Went back, read, and understand your point. The combinatorics make this undoable as the size would blow up and things change too much. This comment from the numba maintainers made it hit home. Yes, I think polyglot is locked to 54.1.1 (not other way to explain all the errors); I'll look into how I create the binary fork you mentioned (have to figure out how to do it). Thanks for considering; Learned a bit from you too!!! I have the workarounds in Gists so hopefully people will come up on those when they run into errors. I'll link in case a Google search gets them to this page:
Feel free to close; or do I close?? |
|
On Dec 11, 2016, at 18:32, Linwood Creekmore ***@***.***> wrote:
I'm Sorry @ovalhub , I missed the earlier comments. Went back, read, and understand your point. The combinatorics make this undoable as the size would blow up and things change too much. This comment from the numba maintainers made it hit home.
Yes, I think polyglot is locked to 54.1.1 (not other way to explain all the errors); I'll look into how I create the binary fork you mentioned (have to figure out how to do it).
Thanks for considering; Learned a bit from you too!!!
I have the workarounds in Gists so hopefully people will come up on those when they run into errors.
I'll link in case a Google search gets them to this page:
Workaround for pyicu error in Anaconda/Python 2.7 config
Workaround for pyicu error in Anaconda/Python 3.5 config
Feel free to close; or do I close??
You're welcome to close the issue if you like.
Andi..
… —
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
|
|
fwiw, a "universal" wheel is inappropriate for packages which have binary dependencies (such as this one) as universal is intended for pure python packages |
I believe it is possible to provide prebuilt binaries for the people who can use them and still provide source distributions for people who can't. That would be one wheel for each target platform and version. There is even a third-party PyPI package that seems to be doing this for linux versions. For example the As far as I know, the main thing required for getting started is just uploading the built distribution like so: This would allow users on compatible platforms to |
|
On Sun, 27 Jan 2019, apnewberry wrote:
> I cannot distribute a PyICU binary on PyPI as it wouldn't work for any
> people that aren't running the same combination compiler, OS, ICU, Python
> that I used to build it.
I believe it is possible to provide prebuilt binaries for the people who
can use them and still provide source distributions for people who can't.
That would be one wheel for each target platform and version. There is
even a [third-party PyPI
package](https://pypi.org/project/PyICU-binary/#files) that seems to be
doing this for linux versions. For example the `coverage.py` project
offers [many different wheels](https://pypi.org/project/coverage/#files),
and `pip install coverage` will automatically select the right one for the
user's platform. It is also common for open-source projects to take
advantage of free continuous-integration server time to build artifacts on
multiple platforms, such as on Travis, Appveyor, and CircleCI.
As far as I know, the main thing required for getting started is just
uploading the built distribution like so: ``` python setup.py sdist
bdist_wheel twine upload dist/* ```
This would allow users on compatible platforms to `pip install pyicu`
without having to build it themselves.
Are you proposing to set this up and maintain it ?
If so, I suggest you fork pyicu into a new pyicu-binaries project and if the
process you implement proves to work as expected and if the failure modes
are better than the current build-from-source situation, we could consider
merging the fork back with pyicu proper - assuming you stick around for
the long term and keep the build configurations current with new binary
combinations as they appear.
…
--
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
#39 (comment)
|
|
May I ask what commands you currently run to produce and release a new version? |
|
On Jan 27, 2019, at 20:13, apnewberry ***@***.***> wrote:
May I ask what commands you currently run to produce and release a new version?
$ python setup.py register sdist
… —
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
|
I'm not prepared to take on that responsibility. I did want to share what I understand about what would be involved. I believe the current recommendation from PyPA boils down to and assuming the setup.py is written per the docs, I think that'll upload the source distribution and the wheel appropriate to the platform executing the command. It's also possible to produce wheels for multiple linux targets using the manylinux project. |
|
On Jan 27, 2019, at 21:11, apnewberry ***@***.***> wrote:
Are you proposing to set this up and maintain it ?
I'm not prepared to take on that responsibility. I did want to share what I understand about what would be involved.
Neither am I.
… I believe the current recommendation from PyPA boils down to
python setup.py sdist bdist_wheel
twine upload dist/*
and assuming the setup.py is written per the docs, I think that'll upload the source distribution and the wheel appropriate to the platform executing the command. It's also possible to produce wheels for multiple linux targets using the manylinux project.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
|
|
A note on manylinux, for libraries which typically would link against a system library (for example, This ~might be undesirable as it removes the benefits of dynamic linking from a security perspective -- that is you no longer get benefit from your system package manager shipping security patches to libraries that are dynamically linked. By extension, this ~probably makes releasing pyicu necessary if a security issue happened in the That said, once you have a script which can build manylinux wheels, the rest is pretty easy (usually via docker) -- here's some examples from my projects (not super sophisticated): sass/libsass-python asottile/setuptools-golang (NOTE: I've only manylinux packaged things without |
|
Just a note that
https://github.com/chubin/cheat.sh/runs/1409411905#step:3:75 |
|
@abitrolly that's a wheel specific to your machine built for the pip cache and isn't manylinux compatible / distributable (it links against whatever version of libicu you have installed on your machine) |
If this is too instructive, forgive me. Just wanted to require minimal effort from original author since this is me asking you to add something to your repository.
Summary of Merge Request
@ovalhub, this merge request creates a “Universal Wheel” to resolve an issue on the MacOS where pyicu's inability to fully install via
pip install pyicucauses an error withpolyglot, which is a multilingual NLP library created by @aboSamoor .Details of Request
Request @ovalhub accept merge request, and run the following commands to update the PyICU PyPi packaging. This will include a universal wheel in the
Filesection.The problem encountered and fix are discussed in detail here.
What's the Change?
Just created a simple
setup.cfgfile as described in the Universal Wheels section of the Python Packaging User Guide. The exact content of the simple setup.cfg file is here.How to "update" your Packaging
After this file is in the master
pyicurepository, you run the following commands, as described in the Packaging User Guide:Now, the last step is just uploading the distribution to PyPi. If you use twine (recommended approach, see here for security discussion):
>>> twine upload dist/*Or the
setuptoolsalternative (not recommended, see here for security discussion):This should solve one of the problems discussed in the
polyglotissue.