Support wheels

[StrikerRUS]

Since rgf_python hasn't any special requirements (for compiler, environment, etc.), I think it good idea to have wheels on PyPI site (and the sources in .tar.gz, of course). I believe providing successfully compiled binaries will prevent many strange errors like recent ones.

We need wheels for two platforms: first for macOS and Linux and second for Windows.

The final result should be similar to this one:

But each wheel for each platform should have 32bit and 64bit version.

Binaries we could get from Travis and Appveyor as artifacts (I can do this). The one problem I see now is that Travis hasn't 32bit machines, but I believe we'll overcome this problem 😃 .

@fukatani When you'll have time, please search how to appropriate name wheels according to target platforms and how to post them at PyPI. Or I can do it more later.

[fukatani]

Very good point and valuable suggestion for users and developers!

OK! I start to investigate.
I use my passward to upload PyPi, these are my tasks basically.

memo:
https://docs.travis-ci.com/user/deployment/pypi/
https://refi64.com/posts/using-appveyor-to-distribute-python-wheels.html

[StrikerRUS]

Working example.

Appveyor:

1. https://github.com/Microsoft/LightGBM/blob/750a11b08cba9adcd05e6bcb92d30495783572da/appveyor.yml#L52
2. https://github.com/Microsoft/LightGBM/blob/750a11b08cba9adcd05e6bcb92d30495783572da/appveyor.yml#L67-L68
3. As I understand it's optional publishing on GitHub, but we can grab artifacts manually:
   https://github.com/Microsoft/LightGBM/blob/750a11b08cba9adcd05e6bcb92d30495783572da/appveyor.yml#L72-L81

Result:

Travis:

1. https://github.com/Microsoft/LightGBM/blob/750a11b08cba9adcd05e6bcb92d30495783572da/.travis/test.sh#L85-L96
2. https://github.com/Microsoft/LightGBM/blob/750a11b08cba9adcd05e6bcb92d30495783572da/.travis.yml#L62-L74

Result:
Release on GitHub, but I may be wrong.

[StrikerRUS]

OK, what we have after #83:

• sources rgf_python-2.1.1.tar.gz (artifact from Appveyor)

• Win x32 rgf_python-2.1.1-py2.py3-none-win32.whl (artifact from Appveyor)

• Win x64 rgf_python-2.1.1-py2.py3-none-win_amd64.whl (artifact from Appveyor)

• Linux x32 rgf_python-2.1.1-py2.py3-none-manylinux1_i686.whl (I'll create them locally on VM)

• Linux x64 rgf_python-2.1.1-py2.py3-none-manylinux1_x86_64.whl (release on GitHub from Travis)

• macOS x32 (no need, see UPD)

• macOS x64 rgf_python-2.1.1-py2.py3-none-any.whl (release on GitHub from Travis)

Wheels for Linux and macOS we could get when you set up releases from Travis to GitHub (https://docs.travis-ci.com/user/deployment/releases/).

So you'll be able to grab wheels and source archive from Appveyor and GitHub and then upload all this stuff to PyPI manually. I think it's easier then uploading directly to PyPI because we have many sources (how to sync them?) and don't want to update PyPI after each commit.

The only remaining question is where we can find Linux and macOS x32?.. Travis hasn't x32 machines and this is sad. I see four variants:

• search for another CI service with 32-bit machines (the worst variant)
• install remaining OSs locally and build wheels manually (is macOS available as VM image for home usege?)
• get free VM (AWS, Google Cloud) and build wheels there manually (I like this variant)
• leave all as is, so users with Linux and macOS x32 have to build rgf_python from sources themselves

UPD: I've found that the latest 32-bit mac OS is 10.6 (released in 2011 and unsupported as of February 25, 2014) and next macOS wont support 32-bit app at all. So, I think we shouldn't worry about macOS x32. I've updated lists according to this.

[fukatani]

Both XGB and LightGBM do not support 32 bit, I think there is no pressing need.
So personally, I would like to choose the last option. If someone want to support 32bit OS, please upload them.

[StrikerRUS]

Yeah, maybe you are right. But both XGB and LGBM require 64-bit machines while RGF doen't.

For example, scikit-learn provides 32-bit wheels..

BTW, can we set plat-name to any_x86_64?

UPD:

BTW, can we set plat-name to any_x86_64?

It seems to be that no 😢

[StrikerRUS]

I've spent the whole day searching 32-bit AMIs on AWS and found there are no them for free. To be honest, I found 5-10 free AMIs but all of them are weird, for example, very old CentOS, Oracle Linux with password.

So I'm downloading VirtualBox and will create rgf_python-2.1.1-py2.py3-none-manylinux1_i686.whl on my local machine (2nd variant). Since we do not update rgf_python frequently, it wont be hard to me.

When you decide to update PyPI just open new issue and ping me. I'll attach the wheel to the response. Other wheels and source archive you'll grab from Appveyor and GitHub draft.

The last remaining thing - configure (#81 (comment)) Travis to upload artifacts. I can't do this because your secure key is required for uploading.

[StrikerRUS]

GitHub says it doesn't support uploading .whl files, so I've made archive:
rgf_python-2.1.1-py2.py3-none-manylinux1_i686.zip

[StrikerRUS]

@fukatani Let's close this issue ASAP. At least we should have one version of rgf_python on PyPI with wheels before adding FastRGF into the package- it would be our checkpoint which users will install if they have problems with FastRGF. :-)

[fukatani]

Sorry.
I have stopped with some troubles.
I missed artifacts download link with in appveyor.

[StrikerRUS]

What do you mean?
Go to Appveyor, select any build from master, then select one job with 64-bit image and the second one with 32-bit image, wheels and archive will be located on "artifacts" tab.

[StrikerRUS]

@fukatani I've succeeded to upload all needed files to test.pypi.
Let me share with you steps I performed.

1. pip install twine
2. create into your home directory file .pypirc (https://docs.python.org/3.2/distutils/packageindex.html#the-pypirc-file)
3. create folder (dist) and place in it all files to be uploaded
4. twine upload -r pypitest dist/rgf_python* (replace pypitest with reponame from your .pypirc file)

And the result:

$ D:\Users\nekit\Downloads\test>twine upload -r pypitest dist/rgf_python*

Uploading distributions to https://test.pypi.org/legacy/
Uploading rgf_python-2.1.1-py2.py3-none-manylinux1_i686.whl
Uploading rgf_python-2.1.1-py2.py3-none-win32.whl
Uploading rgf_python-2.1.1-py2.py3-none-win_amd64.whl
Uploading rgf_python-2.1.1.tar.gz

https://test.pypi.org/project/rgf-python/#files

So, feel free to ask me if you have any problems.

[fukatani]

Ah, I see.
I didn't realize that I need to click each envirionment.

And I wrote my thought.

1. Where to upload.
   Either way is fine, but I prefer github release. Because I/F is easy to see.

2. When to upload.
   I prefer on tag.
   If we release binary every time we change the master branch, the release will overflow.

[StrikerRUS]

I totally agree with you!

Such use case will bring us full control over PyPI releases. And we don't produce, for example, nightly builds, so manual uploads wouldn't be very hard and annoying.

About tag. Am I right that the algorithm will be the following:

1. Decision that we need a new version.
2. Pushing commit to master branch with tag (let say commit with bumping the version).
3. Travis by this commit creates draft release on GitHub with new tag.
4. You open new issue and ping me in it.
5. I compile i686 wheel for Linux and attach it to the issue.
6. You collect all artifacts from Appveyor (theay are regular, tag independent), GitHub release, issue and upload this stuff to PyPI manually.

If I'm not mistaken in the pipeline, I like it :-).

[fukatani]

1. Decision that we need a new version.
2. Pushing commit to master branch with tag (let say commit with bumping the version).
3. Travis by this commit creates draft release on GitHub with new tag.
4. Upload Artifact by appveyor automatically.
5. Upload pip manually.

Your earnestness is wonderful, but in honest, I want to forget 32bit OS.
Basically, RGF (C ++) is not updated and I think that it can be enough to upload binary in the next release only. 🐤 🐤 🐤

[StrikerRUS]

can be enough to upload binary in the next release only

Could you please explain this part?

Did you mean that we could create i686 wheel on x64 machine by --plat-name and then replace RGF executable file manually (because it's invariant to rgf_python versions)?

[fukatani]

I failed to upload release from travis.
It seems the same problem with travis-ci/dpl#270 .
I'll send email to travis support team.

[StrikerRUS]

You may try this travis-ci/travis-ci#8018 (comment) while waiting for response from them.

[fukatani]

Could you please explain this part?

This is my misunderstandings...

Did you mean that we could create i686 wheel on x64 machine by --plat-name and then replace RGF executable file manually

Is this possible? If so, can we replace exe with travis?

[fukatani]

Thanks to your advice, I succeeded to upload release, but rgf_python-2.1.1.tar.gz uploaded twice..
One is for python2.7 and the other is Python3.6..

[fukatani]

As Lightgbm, condition: "$TASK = bdist" or some appropriate conditional limitation should be added.

[fukatani]

I found that deployment automation is interesting. 😃

[StrikerRUS]

Thanks to your advice, I succeeded to upload release, but rgf_python-2.1.1.tar.gz uploaded twice..
One is for python2.7 and the other is Python3.6..

Did you delete these releases because I can't find them?

They say for normal tag name we shoud choose one of the options:

GitHub Releases uses git tags. If the build commit does not have any tags, one will be created in the form of untagged-*, where * is a random hex string.
If this is not what you want, either set your build to deploy only when the build already has a tag using on.tags: true as shown in the previous example .travis.yml, or tag the commit with git tag in before_deploy:
https://docs.travis-ci.com/user/deployment/releases/

Also I see tag_name in LightGBM:
https://github.com/Microsoft/LightGBM/blob/a69f41e2c5dc329fd343be0e3f5638bbdf07635f/.travis.yml#L67

In addition, they say

If you need to overwrite existing files, add overwrite: true to the deploy section of your .travis.yml.

Maybe this can be an alternative to specifying condition (in this case last job will overwrite artifacts from previous)?

[StrikerRUS]

OK, now you should add creating wheels:

    if [[ $TRAVIS_OS_NAME == "osx" ]]; then
        python setup.py bdist_wheel --plat-name=any --universal
    else
        python setup.py bdist_wheel --plat-name=manylinux1_x86_64 --universal
    fi

Also maybe it's better to specify only wheels to upload (.tar.gz we could get from Appveyor):
file: dist/*.whl?

[StrikerRUS]

Did you mean that we could create i686 wheel on x64 machine by --plat-name and then replace RGF executable file manually
Is this possible? If so, can we replace exe with travis?

I mean following. When you need to create new PyPI release, you push fake commit which changes
python setup.py bdist_wheel --plat-name=manylinux1_x86_64 --universal to
python setup.py bdist_wheel --plat-name=manylinux1i686 --universal,
download from GitHub created wheel and on your local machine replace executable file with one attached in this issue. You need to replace executable file because the wheel is still creating on x64 machine and I suppose it's unsafe to leave it as is.

Or you can force Travis to produce 2 versions of Linux wheels permanently. I mean, there will be 2 lines in .travis.yml in Linux if branch:

    else
        python setup.py bdist_wheel --plat-name=manylinux1_x86_64 --universal
        python setup.py bdist_wheel --plat-name=manylinux1_i686 --universal
    fi

But you anyway need to replace executable file in i686 wheel manually on your machine.

Or you can just ping me and I'll attach wheel for i686 😄 .

[StrikerRUS]

And please abort builds you are not needed anymore, or we'll wait for ages 🙄 .

[fukatani]

But you anyway need to replace executable file in i686 wheel manually on your machine.

If we upload the 32-bit executable file to github release only once,
can we wget 32bit executable from and replace in .travis.yml?

[fukatani]

But deploy automation is not completed, I published release manually.
We should start to merge fastRGF.

Future work:

• tag_name doesn't work well. (Tying GitHub Releases to existing tags travis-ci/travis-ci#8248 is related?)
  https://travis-ci.org/fukatani/rgf_python/jobs/310369738
• Upload gz file from appveyor on tags.

[StrikerRUS]

If we upload the 32-bit executable file to github release only once,
can we wget 32bit executable from and replace in .travis.yml?

Good idea! But I don't know is it possible or not.

[StrikerRUS]

Maybe tag_name fails because there is no tags: true condition? In such case deploy will be triggered only on commit with tag.

[StrikerRUS]

I'm not sure that changing any to osx will work, because in most cases it's osx_.... It's safer to use any.

You forgot to unzip i686 wheel.

[StrikerRUS]

And what is this rgf_python-2.1.1-py2.py3-none-win32.tar.gz?

We need two wheels for Windows from x64 and x32 jobs (I don't see them) and one source archive named rgf_python-2.1.1.tar.gz from any job.

[fukatani]

Thanks, I fixed release 😄

[StrikerRUS]

Thank you very much!

It's remained only to add rgf_python-2.1.1.tar.gz and finally that's all! 😄

[fukatani]

Mac OS wheel name is right?

rgf_python-2.1.2-py2.py3-none-macosx_10_6_x86_64.macosx_10_7_x86_64.macosx_10_8_x86_64.macosx_10_9_x86_64.macosx_10_10_x86_64.macosx_10_11_x86_64.macosx_10_12_x86_64.macosx_10_13_x86_64.whl

It is long name..

[StrikerRUS]

Should be right. We'll test it after uploading wheels to PyPI.

any should work too, but it's a little be confusing and may be used for non-supported systems (macOS x32 for instance). Also I'm afraid that macOS may prefer manilinux wheel to any. That's why I decided to give the wheel more concrete name.