Publish bindings on pypi

[breznak]

#1
Depends on binary releases #361
Use this as an example RedFT/Hexy@ddc9d01

• fix format so whl is published on PYPI
• fix whl content so we have all needed files there
  • people should be able to install by pip install my.whl
• get token to publish to "real" PYPI
• update Readme with instructions

[breznak]

This is an interesting alternative to PyPI, pip can install directly from github URL
https://github.com/ceddlyburge/python_use_hello_world

[breznak]

When our HTM is available on PyPI, I'll be able to simplify the install process for HTMcore (our HTM) NAB detector, and offer it to numenta/NAB for official score-board.

[dkeeney]

...working on it.

• I am now able to build the distribution packages for both PyPI and GitHub on appveyor. The GitHub package contains the PyPI wheel.
• Need to do the same thing on Travis and circleCI. (mostly there)
• GitVersion increment is not working yet. Comes up with wrong version on windows.
• I think I need the PR build to check-in (git commit) both the Version file and the two distribution packages because the built packages need to exists when we merge to master. I don't think it does a build at that point.
• Need to deploy the two packages during merge to master.

[dkeeney]

Oh, I will be out-of-town for the next 5 days. I will be checking email but no coding. I will push what I have so far.

I get to do the grandfatherly thing and build a tree house for my grandson :-)

[dkeeney]

Actually if anyone with experience with this type of thing wants to help out on this the help would be appreciated. I am having to learn how to do this from scratch.

[dkeeney]

Reference PR #573

[breznak]

I am now able to build the distribution packages for both PyPI and GitHub on appveyor.

great! glad you've managed to get it work! 👍

I get to do the grandfatherly thing and build a tree house for my grandson :-)

awesome, enjoy your time! And don't forget to (unit)test it.
PS: we want a treehouse in HTM now 😄

[breznak]

@dkeeney would you please look at:
The setup.py does need to link with htmcore.a which can now be in 2 locations: EDIT: needs not.

• in source/developer build: look in /build/Release/ (this takes preference)
• in binary install: look in path relative to the .whl? (../lib/) ?

[dkeeney]

The setup.py does need to link with htmcore.a which can now be in 2 locations:

setup.py uses CMake to create the htm.core library which is installed in /build/Release/lib. The python extensions are then statically linked to that and are put into the .whl along with the py code. The wheel along with lib, include, and bin from /build/Release go into the GitHub package.

So, only the GitHub package contains the htm.core library. The wheel contains the py extensions.

[breznak]

The python extensions are then statically linked to that and are put into the .whl along with the py code.

does it mean that all I need to run binary nupic is in the .whl file? Meaning I can move the wheel to a clean system (that has only py installed) and all nupic python would run? The file seems suspiciously slim for that (2.2MB).

[dkeeney]

does it mean that all I need to run binary nupic is in the .whl file?

That is all you need to run with python. It is the extensions and the .py code. It does not contain the C++ parts like the htm.core library or include files.

[breznak]

It does not contain the C++ parts like the htm.core library or include files.

I'll test that, IMHO it must contain some form of the .so/.a library, there is no implementation of, say SP, in python, and the code needs to call the c++ backend to run.

[breznak]

I have into another error, unsupported arch?

• downloaded the binary release from GH (should be Linux, 64b)
• when I run $ pip3 install --verbose htm.core-2.0.0-cp37-cp37m-linux_x86_64.whl

it fails with

Created temporary directory: /tmp/pip-ephem-wheel-cache-8dhgr3vf
Created temporary directory: /tmp/pip-req-tracker-28bg3xj7
Created requirements tracker '/tmp/pip-req-tracker-28bg3xj7'
Created temporary directory: /tmp/pip-install-mvijj2v3
Cleaning up...
Removed build tracker '/tmp/pip-req-tracker-28bg3xj7'
ERROR: htm.core-2.0.0-cp37-cp37m-linux_x86_64.whl is not a supported wheel on this platform.
Exception information:
Traceback (most recent call last):
File "/mnt/store/devel/test/env/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 188, in main
status = self.run(options, args)
File "/mnt/store/devel/test/env/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 319, in run
self.name, wheel_cache
File "/mnt/store/devel/test/env/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 281, in populate_requirement_set
requirement_set.add_requirement(req_to_add)
File "/mnt/store/devel/test/env/lib/python3.6/site-packages/pip/_internal/req/req_set.py", line 91, in add_requirement
wheel.filename
pip._internal.exceptions.InstallationError: htm.core-2.0.0-cp37-cp37m-linux_x86_64.whl is not a supported wheel on this platform.

I'm on ubuntu 64b, with py3.

[breznak]

cp37

the CI is using py3.7, wonder if that's the problem and it is so sensitive?

[breznak]

this could be interesting
https://github.com/pypa/gh-action-pypi-publish

runs nicely! But currently only for Linux host. Reported upstream:
pypa/gh-action-pypi-publish#15

Let's see, alternatively we can use twine from PYPI

[breznak]

@dkeeney my test.pypi.org has trouble registering project htm.core, this is likely due to the fact that you've registered that with previous attempts.
Could you please either:

• send me the unencrypted (pypi-...) token by email/forums PM?
• or unregister,delete your htmcore project on pypi?

[dkeeney]

Oh, I should have known that might cause a conflict.
I deleted htm.core project from my account.
You should be able to use it now.

[breznak]

one step done, that is working!

Now a new problem,
https://github.com/htm-community/htm.core/runs/237586188#step:14:17

HTTPError: 400 Client Error: Binary wheel 'htm.core-2.0.14-cp37-cp37m-linux_x86_64.whl' has an unsupported platform tag 'linux_x86_64'. for url: https://test.pypi.org/legacy/

seems the platform is encoded in the package name, so we'll have to change that correctly.

[breznak]

get token to publish to "real" PYPI

In parallel we could start with this.
@rhyolight would you please?

• setup a htm-community account on PYPI
• optionally add me as a maintainer
• create and send me by mall/PM a token for project htm.core
  • this would be the string pypi-some_hash
• so I can add it to our github secrets and we'd have working PYPI releases

TODO: fix for other platforms, see #19 (comment)

[dkeeney]

unsupported platform tag 'linux_x86_64'. for url: https://test.pypi.org/legacy/

yes, that is the bug I looked into some time back. Binary wheels cannot be made for linux in general because they cannot be guaranteed to work. The work-around was to build on a version of linux that is older (or as old) as anyone that would currently use it. There is a different syntax for the platform tag that contains the date of that OS.

Could not find a link to the issue or PR where I reported it.

[breznak]

The work-around was to build on a version of linux that is older (or as old) as anyone that would currently use it. There is a different syntax for the platform tag that contains the date of that OS.

yes, I remember, this is the manylinux docker builds. But I thought the inability to run on older toolchain is only a consequence, not requirement by PYPI (so I could publish pypi from our CI, but people would need the same/newer toolchain.)

• what is the correct format of the filename? (isn't it only .x86_64. ,)
• how do we deal with it (manylinux)? I'll have a look at Numenta's builds. We probably don't want to build on old docker images all the time (we're running bleeding edge C++ features), so for PYPI publish we'd have to setup Docker builds?
  • question is if we can compile on some old toolchain at all(?)

[breznak]

Our setu.py will need some work.
This looks like a good tutorial:
https://python-packaging.readthedocs.io/en/latest/minimal.html

I'm failing already on the first step: pip install .

[breznak]

Two interesting guides on publishing:
pypa/packaging.python.org#647
https://www.python.org/dev/peps/pep-0517/

[dkeeney]

@breznak sorry I did not get back right away....life getting in the way.
Ok, I took the wheel file that is generated and tried to pip install from it. It gives me some errors but says it installed. And indeed it is in the installed packages list.

Now I just need to figure out how to run the py tests.

[breznak]

Update:

• CI now stores artifacts (now I'm saving dist/) after each "push tag" build. You can access the artifacts on top right corner of the CI in "Publish to PYPI" job.

Run ls dist*
dist1:
htm.core-2.0.16-cp37-cp37m-linux_x86_64.whl
htm.core-2.0.16-py3.7-linux-x86_64.egg
requirements.txt

dist2:
htm.core-2.0.16-cp37-cp37m-macosx_10_14_x86_64.whl
htm.core-2.0.16-py3.7-macosx-10.14-x86_64.egg
requirements.txt

dist3:
htm.core-2.0.16-cp37-cp37m-win_amd64.whl
htm.core-2.0.16-py3.7-win-amd64.egg
requirements.txt

Now I'm trying to figure if dist/ can contain multiple whl/egg files (from each platform)?

[dkeeney]

I think the wheel files replace the egg. So do we need to create the eggs?

[breznak]

I think the wheel files replace the egg. So do we need to create the eggs?

yep, we should use only wheels:

P.S. I see you're building eggs, it's a decade-deprecated format. Better drop those in favor of wheels.

[breznak]

The linux pypi package will be a pain,
we'll want to use the manylinux docker image,

• which comes only with gcc-4.8, so we'd have to update/build a newer (8.3) gcc there...!
  This is a good article
  https://thomastrapp.com/blog/building-a-pypi-package-for-a-modern-cpp-project/

On the possitive side, cmake is available on PYPI, so we could install from there and drop it as dependency (c++ users would know to install it anyway)
https://pypi.org/project/cmake/

[breznak]

I have been thinking how we deal with the "common baseline" for Linux - manylinux.
Manylinux is based on CentOS 5 (with gcc 4.8) (manylinux 2010, 2014 are in TODO pipeline)
pypa/manylinux#118

• we could try building on antient gcc 4.8 in c++11, and use original manylinux PYPI
  • we want to retry/reevaluate when manylinux 2010, 2014 are released
  • another option is to install (from source, repo?) a newer gcc (8.x would be ideal)
• we could fake the name for PYPI and build on the Ubuntu18.04 image from CI
  • name must be changed to "manylinux-x86_64" for pypi to accept the wheel
  • since we bind static libstdc++, I'd try this "fake" name option first

[breznak]

YES 🎉 we have PYPI release!
https://github.com/htm-community/htm.core/runs/246384722

[breznak]

we have a first PYPI release! at
https://test.pypi.org/project/htm.core/2.0.18/

I follow the install instructions:
pip install -i https://test.pypi.org/simple/ htm.core==2.0.18

but getting an error

$ pip install -i https://test.pypi.org/simple/ htm.core==2.0.18
Looking in indexes: https://test.pypi.org/simple/
Collecting htm.core==2.0.18
ERROR: Could not find a version that satisfies the requirement htm.core==2.0.18 (from versions: none)
ERROR: No matching distribution found for htm.core==2.0.18

Note, I also tried
$ pip install -i https://test.pypi.org/legacy/ htm.core==2.0.18
as we upload to /legacy/

CC @dkeeney

[pradyunsg]

Ah, that might be because pip's got an aggressive caching strategy and it's not being managed properly. Can you check what it shows with --verbose?

[breznak]

Can you check what it shows with --verbose?

Sure, thanks for the hint, @pradyunsg !

$ pip uninstall htm.core
WARNING: Skipping htm.core as it is not installed.
(pyenv3) mmm@mmm-U2442:~/devel/HTM/htm-community/nupic.cpp/build/scripts$ pip install -i https://test.pypi.org/simple/ --verbose htm.core==2.0.18
Created temporary directory: /tmp/pip-ephem-wheel-cache-fv69v6ze
Created temporary directory: /tmp/pip-req-tracker-htf02_9h
Created requirements tracker '/tmp/pip-req-tracker-htf02_9h'
Created temporary directory: /tmp/pip-install-paafdv68
Looking in indexes: https://test.pypi.org/simple/
Collecting htm.core==2.0.18
1 location(s) to search for versions of htm.core:
* https://test.pypi.org/simple/htm-core/
Getting page https://test.pypi.org/simple/htm-core/
Found index url https://test.pypi.org/simple/
Looking up "https://test.pypi.org/simple/htm-core/" in the cache
Request header has "max_age" as 0, cache bypassed
Starting new HTTPS connection (1): test.pypi.org:443
https://test.pypi.org:443 "GET /simple/htm-core/ HTTP/1.1" 304 0
Analyzing links from page https://test.pypi.org/simple/htm-core/
Skipping link: none of the wheel's tags match: cp37-cp37m-macosx_10_14_x86_64: https://test-files.pythonhosted.org/packages/5c/60/63398eec1c0d860b7fd8405bf6729beae6b8da929171ebd001cb4e3943b6/htm.core-2.0.16-cp37-cp37m-macosx_10_14_x86_64.whl#sha256=946e101bc183ffac801e19a2e708e674cdfbc2295bc36bcf69ec7f065552baef (from https://test.pypi.org/simple/htm-core/)
Skipping link: none of the wheel's tags match: cp37-cp37m-manylinux1_x86_64: https://test-files.pythonhosted.org/packages/cd/92/04fe8ccb054f46de11ad68a47b31aa5cd0d4ad21f79201811c95660703c0/htm.core-2.0.16-cp37-cp37m-manylinux1_x86_64.whl#sha256=d5a55a92ff30e49adce9c6a8428548de9b7b48257e52ff72b6148c4cb973e263 (from https://test.pypi.org/simple/htm-core/)
Skipping link: none of the wheel's tags match: cp37-cp37m-win_amd64: https://test-files.pythonhosted.org/packages/10/cf/3ee61d63cbf7aa9d2fed5b4c9c40397a104fe5c7f85f76fb6fbec65755a6/htm.core-2.0.16-cp37-cp37m-win_amd64.whl#sha256=5fec1a2546c71f15d8ac2ec02ea1fcfe06d4237354abad1eb451de33a4e94d2c (from https://test.pypi.org/simple/htm-core/)
Skipping link: unsupported archive format: .egg: https://test-files.pythonhosted.org/packages/09/2b/c407c551e465a2a6bf8a0314f12ccf419e34467a6bb3e50df49eaa340e53/htm.core-2.0.16-py3.7-linux-x86_64.egg#sha256=a543a3d37641b67c106a2f574dd46dd3e2efe696a1f151b74f8a36521b85b1c2 (from https://test.pypi.org/simple/htm-core/)
Skipping link: unsupported archive format: .egg: https://test-files.pythonhosted.org/packages/20/cd/f6298c4a3316022d758b2a64ace008286ef03e43dc7d24d53af05c3f476d/htm.core-2.0.16-py3.7-macosx-10.14-x86_64.egg#sha256=98ccd274ac11e80bcb3d8aa77adb382d00345913c7f7897efe494231549bfdda (from https://test.pypi.org/simple/htm-core/)
Skipping link: unsupported archive format: .egg: https://test-files.pythonhosted.org/packages/74/50/0f6f4809aa7bdc2521dd658efb7379f693031a2ccd700fae88f5650d196e/htm.core-2.0.16-py3.7-win-amd64.egg#sha256=4910d8aa5a5600988c4f5300a2ecff7c82d55c05edc033beddfdc3ea747216cf (from https://test.pypi.org/simple/htm-core/)
Skipping link: none of the wheel's tags match: cp37-cp37m-macosx_10_14_x86_64: https://test-files.pythonhosted.org/packages/90/be/d532c2a0e3c9c0b035928f5aee44846a84f22dbe02888a22400e257cb643/htm.core-2.0.17-cp37-cp37m-macosx_10_14_x86_64.whl#sha256=5bbef3c56e68d869084caeabd6f97a75560e86400e4270804672152f81ef80b3 (from https://test.pypi.org/simple/htm-core/)
Skipping link: none of the wheel's tags match: cp37-cp37m-manylinux1_x86_64: https://test-files.pythonhosted.org/packages/69/85/6b9702fe9ec08f0611367b2355d63b565aa446aeb50b2aedf03e97510fee/htm.core-2.0.17-cp37-cp37m-manylinux1_x86_64.whl#sha256=8c8f1e68299c84e2e263d9d85ea007b16ce115b3b26ea0a5f33943e67fbcddeb (from https://test.pypi.org/simple/htm-core/)
Skipping link: none of the wheel's tags match: cp37-cp37m-win_amd64: https://test-files.pythonhosted.org/packages/4d/1e/6455a70e98f34c7861acc2bc2b2e9e689ea14691bab9f846fea9a99f60d9/htm.core-2.0.17-cp37-cp37m-win_amd64.whl#sha256=6d6896bda448bbf15d6c143228047f1424d9a80557d254c0eb64fbd58031481e (from https://test.pypi.org/simple/htm-core/)
Skipping link: none of the wheel's tags match: cp37-cp37m-macosx_10_14_x86_64: https://test-files.pythonhosted.org/packages/79/93/716973c55ab27a57438002096617ca669ee899cfc5f81d2627104b936df9/htm.core-2.0.18-cp37-cp37m-macosx_10_14_x86_64.whl#sha256=f54668ef59787167cdc7cb1876c481a07e86f63d129ff1f5997c04e5b14421b8 (from https://test.pypi.org/simple/htm-core/)
Skipping link: none of the wheel's tags match: cp37-cp37m-manylinux1_x86_64: https://test-files.pythonhosted.org/packages/26/d0/adff7d1cf6650d4806cfabb5451ee3f93e575116eaca5121afd7998c97c3/htm.core-2.0.18-cp37-cp37m-manylinux1_x86_64.whl#sha256=305db87651fc2bd149e2f3b9fde75390d6ca14a4fda53671c28a010014fd00ec (from https://test.pypi.org/simple/htm-core/)
Skipping link: none of the wheel's tags match: cp37-cp37m-win_amd64: https://test-files.pythonhosted.org/packages/14/e3/04456112733050d3b2c0e2ed6c65ef7beda3b60aaac528fc9c1f35244118/htm.core-2.0.18-cp37-cp37m-win_amd64.whl#sha256=8acfc90a6b61d038a8ca7a4b28bf685d88111ebe5c879895afde0a51e3852b1c (from https://test.pypi.org/simple/htm-core/)
Given no hashes to check 0 links for project 'htm.core': discarding no candidates
ERROR: Could not find a version that satisfies the requirement htm.core==2.0.18 (from versions: none)
Cleaning up...
Removed build tracker '/tmp/pip-req-tracker-htf02_9h'
ERROR: No matching distribution found for htm.core==2.0.18
Exception information:
Traceback (most recent call last):
File "/mnt/store/devel/HTM/pyenv3/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 188, in main
status = self.run(options, args)
File "/mnt/store/devel/HTM/pyenv3/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 345, in run
resolver.resolve(requirement_set)
File "/mnt/store/devel/HTM/pyenv3/lib/python3.6/site-packages/pip/_internal/legacy_resolve.py", line 196, in resolve
self._resolve_one(requirement_set, req)
File "/mnt/store/devel/HTM/pyenv3/lib/python3.6/site-packages/pip/_internal/legacy_resolve.py", line 359, in _resolve_one
abstract_dist = self._get_abstract_dist_for(req_to_install)
File "/mnt/store/devel/HTM/pyenv3/lib/python3.6/site-packages/pip/_internal/legacy_resolve.py", line 307, in _get_abstract_dist_for
self.require_hashes
File "/mnt/store/devel/HTM/pyenv3/lib/python3.6/site-packages/pip/_internal/operations/prepare.py", line 134, in prepare_linked_requirement
req.populate_link(finder, upgrade_allowed, require_hashes)
File "/mnt/store/devel/HTM/pyenv3/lib/python3.6/site-packages/pip/_internal/req/req_install.py", line 211, in populate_link
self.link = finder.find_requirement(self, upgrade)
File "/mnt/store/devel/HTM/pyenv3/lib/python3.6/site-packages/pip/_internal/index.py", line 1228, in find_requirement
'No matching distribution found for %s' % req
pip._internal.exceptions.DistributionNotFound: No matching distribution found for htm.core==2.0.18

Collecting htm.core==2.0.18
1 location(s) to search for versions of htm.core:
* https://test.pypi.org/simple/htm-core/
Getting page https://test.pypi.org/simple/htm-core/
Found index url https://test.pypi.org/simple/
Looking up "https://test.pypi.org/simple/htm-core/" in the cache
Request header has "max_age" as 0, cache bypassed

seems the package exists on PyPI and pip is not using the cache.

Analyzing links from page https://test.pypi.org/simple/htm-core/
Skipping link: none of the wheel's tags match: cp37-cp37m-macosx_10_14_x86_64: https://test-files.pythonhosted.org/packages/5c/60/63398eec1c0d860b7fd8405bf6729beae6b8da929171ebd001cb4e3943b6/htm.core-2.0.16-cp37-cp37m-macosx_10_14_x86_64.whl#sha256=946e101bc183ffac801e19a2e708e674cdfbc2295bc36bcf69ec7f065552baef (from https://test.pypi.org/simple/htm-core/)
Skipping link: none of the wheel's tags match: cp37-cp37m-manylinux1_x86_64: https://test-files.pythonhosted.org/packages/cd/92/04fe8ccb054f46de11ad68a47b31aa5cd0d4ad21f79201811c95660703c0/htm.core-2.0.16-cp37-cp37m-manylinux1_x86_64.whl#sha256=d5a55a92ff30e49adce9c6a8428548de9b7b48257e52ff72b6148c4cb973e263 (from https://test.pypi.org/simple/htm-core/)
Skipping link: none of the wheel's tags match: cp37-cp37m-win_amd64: https://test-files.pythonhosted.org/packages/10/cf/3ee61d63cbf7aa9d2fed5b4c9c40397a104fe5c7f85f76fb6fbec65755a6/htm.core-2.0.16-cp37-cp37m-win_amd64.whl#sha256=5fec1a2546c71f15d8ac2ec02ea1fcfe06d4237354abad1eb451de33a4e94d2c (from https://test.pypi.org/simple/htm-core/)

Looks like my local machine (Ubuntu LTS, x64) does not match any of the signatures?

could this be because my "hack"? https://github.com/htm-community/htm.core/blob/master/.github/workflows/htmcore.yml#L135
where I simply renamed the file to manylinux1_x86_64 to pass PyPI checks?

Notice my hack, should it not pass? But the CI build image actually matches my system (OS Ubuntu 18.04, gcc 8.3, 64bit), libstdc++ is statically linked.

[breznak]

Ok, I figured this!

cp37-cp37m-manylinux1_x86_64

the CI package is compiled with python 3.7 (cp37) while on my machine only py 3.6 was available.
After installing locally python 3.7, the pip install works fine!

OT: @pradyunsg maybe pip could be more verbose on which condition was not met?

So the workaround would be to compile in CI with a least possible py3 version.
@dkeeney what is the oldest python 3.x version we should use? With regards to pybind11, ...

[pradyunsg]

OT: @pradyunsg maybe pip could be more verbose on which condition was not met?

pypa/pip#4649 (comment)
pypa/pip#6526