Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: 1.13.2 release fails upon import (Mac OS X & Linux/Python 3.5.1) #9786

Closed
adamosloizou opened this issue Sep 28, 2017 · 39 comments
Closed

BUG: 1.13.2 release fails upon import (Mac OS X & Linux/Python 3.5.1) #9786

adamosloizou opened this issue Sep 28, 2017 · 39 comments

Comments

@adamosloizou
Copy link

@adamosloizou adamosloizou commented Sep 28, 2017

Versions

  • OS X: 10.12.1 (16B2555)
  • Linux: Amazon Linux AMI release 2016.03
  • Python: 3.5.1
  • Pip: 9.0.1
  • Numpy: 1.13.2
    • numpy-1.13.2-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl for Mac OS X
    • numpy-1.13.2-cp35-cp35m-manylinux1_x86_64.whl for Linux

Description

We are getting import errors when attempting to use numpy:

Mac OS X

Python 3.5.1 (default, May  9 2017, 17:14:37)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/adamos.loizou/.virtualenvs/tmp-f692dfaf032dece/lib/python3.5/site-packages/numpy/__init__.py", line 166, in <module>
    from . import random
  File "/Users/adamos.loizou/.virtualenvs/tmp-f692dfaf032dece/lib/python3.5/site-packages/numpy/random/__init__.py", line 99, in <module>
    from .mtrand import *
ImportError: dlopen(/Users/adamos.loizou/.virtualenvs/tmp-f692dfaf032dece/lib/python3.5/site-packages/numpy/random/mtrand.cpython-35m-darwin.so, 2): Symbol not found: __PyThreadState_UncheckedGet
  Referenced from: /Users/adamos.loizou/.virtualenvs/tmp-f692dfaf032dece/lib/python3.5/site-packages/numpy/random/mtrand.cpython-35m-darwin.so
  Expected in: flat namespace
 in /Users/adamos.loizou/.virtualenvs/tmp-f692dfaf032dece/lib/python3.5/site-packages/numpy/random/mtrand.cpython-35m-darwin.so

Linux

Python 3.5.1 (default, Sep 13 2016, 18:48:37)
[GCC 4.8.3 20140911 (Red Hat 4.8.3-9)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/root/venv/local/lib/python3.5/dist-packages/numpy/__init__.py", line 166, in <module>
    from . import random
  File "/root/venv/local/lib/python3.5/dist-packages/numpy/random/__init__.py", line 99, in <module>
    from .mtrand import *
ImportError: /root/venv/local/lib/python3.5/dist-packages/numpy/random/mtrand.cpython-35m-x86_64-linux-gnu.so: undefined symbol: _PyThreadState_UncheckedGet

This also affects using pandas upon a fresh installation because pandas only specifies a minimum numpy version thus taking the latest.

@adamosloizou adamosloizou changed the title BUG: 1.13.2 release fails upon import (Mac OS X/Python 3.5.1) BUG: 1.13.2 release fails upon import (Mac OS X & Linux/Python 3.5.1) Sep 28, 2017
@charris

This comment has been minimized.

Copy link
Member

@charris charris commented Sep 28, 2017

Seems to be a cython error: cython/cython#1880.

@charris

This comment has been minimized.

Copy link
Member

@charris charris commented Sep 28, 2017

I've hidden the 1.13.2 release on PyPi.

@matthew-brett

This comment has been minimized.

Copy link
Contributor

@matthew-brett matthew-brett commented Sep 28, 2017

@charris

This comment has been minimized.

Copy link
Member

@charris charris commented Sep 28, 2017

Pip will still see it, but it won't be the default download. Maybe I should just delete it ...

@pv

This comment has been minimized.

Copy link
Member

@pv pv commented Sep 28, 2017

@matthew-brett

This comment has been minimized.

Copy link
Contributor

@matthew-brett matthew-brett commented Sep 28, 2017

@matthew-brett

This comment has been minimized.

Copy link
Contributor

@matthew-brett matthew-brett commented Sep 28, 2017

@charris

This comment has been minimized.

Copy link
Member

@charris charris commented Sep 28, 2017

A build tag changes the file name?

@matthew-brett

This comment has been minimized.

Copy link
Contributor

@matthew-brett matthew-brett commented Sep 28, 2017

@charris

This comment has been minimized.

Copy link
Member

@charris charris commented Sep 28, 2017

I removed the file, but I suspect PyPi will resort to the source. OTOH, who knows, it's PyPi ;)

@charris

This comment has been minimized.

Copy link
Member

@charris charris commented Sep 28, 2017

Hmm, this means the cythonized source files are also in error, so need to be uploaded also. @matthew-brett How does one add a build tag?

EDIT: Oh, I see. Does that also work for source files?

@charris

This comment has been minimized.

Copy link
Member

@charris charris commented Sep 28, 2017

Is it only 3.5 on OSX? Google seems to indicate that it is micro version dependent, so maybe just hasn't been noticed on other systems.

@pv

This comment has been minimized.

Copy link
Member

@pv pv commented Sep 28, 2017

@adamosloizou

This comment has been minimized.

Copy link
Author

@adamosloizou adamosloizou commented Sep 28, 2017

@charris please bear in mind that we are seeing failures in both Mac OS and Linux (although we get different error messages).
I don't know about Windows.

@charris

This comment has been minimized.

Copy link
Member

@charris charris commented Sep 28, 2017

OK, sounds as if all the 3.5 wheels need to be deleted.

@charris

This comment has been minimized.

Copy link
Member

@charris charris commented Sep 28, 2017

Looks like a build tag should be included in the appveyor and travis scripts.

@charris

This comment has been minimized.

Copy link
Member

@charris charris commented Sep 28, 2017

The discussion at #9787 might also be of interest. There is some indication that we may need to do another point release in order to support legacy scripts.

@carno

This comment has been minimized.

Copy link

@carno carno commented Sep 28, 2017

Pulling down just the cp35 wheels does not really help people who install numpy in environments without a ready compilation environment (cloud for the win! ;-)), as pip tries to compile from source and fails due to lack of compiler and/or dev libs needed for compilation.

@charris

This comment has been minimized.

Copy link
Member

@charris charris commented Sep 28, 2017

@carno We can't push new wheels until the version of cython used in the build is pinned to something usable. That said, there is an argument to be made for simply removing the release from pip, as there is no easy way to mark it as broken, and fixing with build tags might break downstream scripts.

@theacodes

This comment has been minimized.

Copy link

@theacodes theacodes commented Sep 28, 2017

(drive by comment): it's generally discouraged to delete anything from PyPI.

@matthew-brett

This comment has been minimized.

Copy link
Contributor

@matthew-brett matthew-brett commented Sep 28, 2017

Chuck - can't we just rebuild with Cython 0.26?

I think we do have to remove the release, because we can't fix the source archive for 1.13.2. How about 1.13.3 with the same content, but Cython 0.26 wheels / source?

@greatestape

This comment has been minimized.

Copy link

@greatestape greatestape commented Sep 28, 2017

@charris I don't remember what the conventions are with pypi, but I think you should delete 1.13.2 and push 1.13.3 once the issue is resolved. As it is, anyone who hasn't pinned their numpy requirement (and even people who pinned to 1.13.*, which should be safe) is going to have their builds start failing (which is what happened to us)

dhermes added a commit to dhermes/bezier that referenced this issue Sep 28, 2017
These are "bad" releases somehow:

numpy/numpy#9786
numpy/numpy#9787
https://readthedocs.org/projects/bezier/builds/6047158/
https://circleci.com/gh/dhermes/bezier/819

ISTM the algebraic intersection failure was caused by building
from source (since `1.13.2.post1` was being found). So the "from
source" build did not get linked with ATLAS, etc.
@matthew-brett

This comment has been minimized.

Copy link
Contributor

@matthew-brett matthew-brett commented Sep 28, 2017

Just to clarify - this is all due to cython/cython#1880

The bug appears when code generated with Cython 0.27 gets run on Python 3.5 < 3.5.2 - at least on Linux and OSX - maybe also on Windows.

@charris

This comment has been minimized.

Copy link
Member

@charris charris commented Sep 28, 2017

@matthew-brett To have the Python 3.7 fix we need Cython >= 0.26.1

@matthew-brett

This comment has been minimized.

Copy link
Contributor

@matthew-brett matthew-brett commented Sep 28, 2017

Can we compile with current master then?

@charris

This comment has been minimized.

Copy link
Member

@charris charris commented Sep 28, 2017

I think 0.26.1 is good, although there is always NumPy 1.13.4 if it isn't. I did try a build tag for the source release and it worked.

charris@fc [~]$ pip install --no-binary :all: --upgrade numpy --user 
Collecting numpy
  Using cached numpy-1.13.2-1.zip
Skipping bdist_wheel for numpy, due to binaries being disabled for it.
Installing collected packages: numpy
  Running setup.py install for numpy ... done
Successfully installed numpy-1.13.2
@matthew-brett

This comment has been minimized.

Copy link
Contributor

@matthew-brett matthew-brett commented Sep 28, 2017

Hmm - but I think what's happening there is pip thinks that this is numpy version 1.13.2-1 . I think that's why pip install --no-binary :all: numpy==1.13.2 fails and pip install --no-binary :all: numpy==1.13.2-1 succeeds.

@rgommers

This comment has been minimized.

Copy link
Member

@rgommers rgommers commented Sep 28, 2017

@rgommers

This comment has been minimized.

Copy link
Member

@rgommers rgommers commented Sep 28, 2017

Can we compile with current master then?

You meant Cython master, that could be okay. Sorry, replying to issue in email client is confusing:)

Still the rest of my comment doesn't change.

@rgommers

This comment has been minimized.

Copy link
Member

@rgommers rgommers commented Sep 28, 2017

(drive by comment): it's generally discouraged to delete anything from PyPI.

Eh, that doesn't apply to something that's clearly broken.

@charris

This comment has been minimized.

Copy link
Member

@charris charris commented Sep 28, 2017

@rgommers It was good for a bit of experimentation, which is one of the few ways to find out how PyPi actually works. I, at least, have learned something.

@matthew-brett

This comment has been minimized.

Copy link
Contributor

@matthew-brett matthew-brett commented Sep 28, 2017

Right - I meant Cython master - is it a problem compiling 1.13.3 with Cython master?

@rgommers

This comment has been minimized.

Copy link
Member

@rgommers rgommers commented Sep 28, 2017

Right - I meant Cython master - is it a problem compiling 1.13.3 with Cython master?

If the numpy-wheels builds are all green, it should be okay I'd think.

@charris

This comment has been minimized.

Copy link
Member

@charris charris commented Sep 28, 2017

Ralf, they were all green before. There are actually a fair number of commits queued up for Cython 0.27.1, and given that 0.27 already broke NumPy in two different ways I'd rather be conservative.

@charris

This comment has been minimized.

Copy link
Member

@charris charris commented Sep 28, 2017

OK, 1.13.2 is gone.

@rgommers

This comment has been minimized.

Copy link
Member

@rgommers rgommers commented Sep 28, 2017

I'd rather be conservative.

Yeah good idea

@charris

This comment has been minimized.

Copy link
Member

@charris charris commented Sep 30, 2017

Numpy 1.13.3 is out, built with Cython 0.26.1. Hopefully that will fix this problem.

@Stewori

This comment has been minimized.

Copy link

@Stewori Stewori commented Sep 30, 2017

For the record, this issue also affected NumPy compatibility with Jython/JyNI, see Stewori/JyNI#20. I had already found out that it was due to Cython 0.27, but was still assuming it was due to something in JyNI. Luckily, I can confirm this is fixed with NumPy 13.3!

@mattip

This comment has been minimized.

Copy link
Member

@mattip mattip commented Jul 19, 2018

Issue was solved by removing 1.13.2 from PyPI, closing

@mattip mattip closed this Jul 19, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
10 participants
You can’t perform that action at this time.