New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The package demands a build at install time #513

Open
ilatypov opened this Issue Aug 30, 2018 · 9 comments

Comments

Projects
None yet
3 participants
@ilatypov

ilatypov commented Aug 30, 2018

Summary:

The Cython maintainers recommend that packages using Cython ship their pre-compiled code to avoid having to depend on building at install time that would also mutate the result.

http://docs.cython.org/en/latest/src/userguide/source_files_and_compilation.html#distributing-cython-modules

Description:

We got stuck with installing fbprophet (as part of installing packages from a requirements.txt) without having Cython installed in advance. The log showed that pip downloaded Cython first, then downloaded pystan and ran its setup which failed because Cython was not installed.

Collecting Cython>=0.22 (from fbprophet==0.3.post2->-r requirements.txt (line 11))
  Downloading https://XXXX/artifactory/api/pypi/pypi/packages/76/64/110bacbf53fe2cf545b0ede8b92d787a1244c655d483472b6ed1964716bc/Cython-0.28.5-cp37-cp37m-win32.whl (2.2MB)
    100% |....| 2.2MB 4.5MB/s
Collecting pystan>=2.14 (from fbprophet==0.3.post2->-r requirements.txt (line 11))
  Downloading https://XXXX/artifactory/api/pypi/pypi/packages/40/69/259304776a6fca45b70e1577c184c2ef6de00630be2e26cd5d5a63581fb2/pystan-2.17.1.0.tar.gz (13.9MB)
    100% |....| 13.9MB 1.1MB/s
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\Users\XXXX\AppData\Local\Temp\pip-install-5bk2k1e_\pystan\setup.py", line 121, in <module>
        from Cython.Build.Inline import _get_build_extension
    ModuleNotFoundError: No module named 'Cython'
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in C:\Users\XXXX\AppData\Local\Temp\pip-install-5bk2k1e_\pystan\

I gather that it may still be possible to implement the build-time requirement of Cython as part of the install-time ones. An improvement in pip 6.1.0 introduced a topological sort of the requirements

pypa/pip@c85f1d6

but with regards to using Cython at install time, I still believe packages should remain packages rather than bootstrapping systems.

PyStan Version:

2.17.1.0

Python Version:

3.7 (I guess from another log)

Operating System:

Windows

@ilatypov

This comment has been minimized.

Show comment
Hide comment
@ilatypov

ilatypov Aug 30, 2018

Sorry I am not experienced with the packaging systems so I am just theorizing here.

Curiously, a StackOverflow answer points to setuptools as a preferred packager and installer. I guess pystan uses distutils opportunistically as a custom inherited from other packages.

https://stackoverflow.com/questions/6344076/differences-between-distribute-distutils-setuptools-and-distutils2

ilatypov commented Aug 30, 2018

Sorry I am not experienced with the packaging systems so I am just theorizing here.

Curiously, a StackOverflow answer points to setuptools as a preferred packager and installer. I guess pystan uses distutils opportunistically as a custom inherited from other packages.

https://stackoverflow.com/questions/6344076/differences-between-distribute-distutils-setuptools-and-distutils2

@riddell-stan

This comment has been minimized.

Show comment
Hide comment
@riddell-stan

riddell-stan Aug 30, 2018

Contributor
Contributor

riddell-stan commented Aug 30, 2018

@ahartikainen

This comment has been minimized.

Show comment
Hide comment
@ahartikainen

ahartikainen Aug 30, 2018

Collaborator

Also, please notice that we have not yet shipped official wheels for Python 3.7. I have unofficial wheels for Windows & py37 in https://github.com/ahartikainen/custom_wheels but if Cython is installed, pip install should work.

Also, after we have official wheels for py37 installation should work without problems.

Collaborator

ahartikainen commented Aug 30, 2018

Also, please notice that we have not yet shipped official wheels for Python 3.7. I have unofficial wheels for Windows & py37 in https://github.com/ahartikainen/custom_wheels but if Cython is installed, pip install should work.

Also, after we have official wheels for py37 installation should work without problems.

@riddell-stan

This comment has been minimized.

Show comment
Hide comment
@riddell-stan

riddell-stan Aug 30, 2018

Contributor
Contributor

riddell-stan commented Aug 30, 2018

@ahartikainen

This comment has been minimized.

Show comment
Hide comment
@ahartikainen

ahartikainen Aug 30, 2018

Collaborator

Windows py27 and py37 are missing.

But we are also missing py37 for osx and unix so I think we can support py37 for Windows at the same time we have working wheels for *nix.

Py27 wheels are new on Windows, because we didn't support it officially before MingW-w64.

Collaborator

ahartikainen commented Aug 30, 2018

Windows py27 and py37 are missing.

But we are also missing py37 for osx and unix so I think we can support py37 for Windows at the same time we have working wheels for *nix.

Py27 wheels are new on Windows, because we didn't support it officially before MingW-w64.

@riddell-stan

This comment has been minimized.

Show comment
Hide comment
@riddell-stan

riddell-stan Aug 31, 2018

Contributor
Contributor

riddell-stan commented Aug 31, 2018

@ahartikainen

This comment has been minimized.

Show comment
Hide comment
@ahartikainen

ahartikainen Aug 31, 2018

Collaborator

Yes, python 2.7 on Windows is broken.

Currently wheels are builded with ci, but we can drop the support for them on Windows.

I have no idea do we even have users on Python 2.7 + Windows, but with the correct compiler installed, pip will work even without wheels.

Collaborator

ahartikainen commented Aug 31, 2018

Yes, python 2.7 on Windows is broken.

Currently wheels are builded with ci, but we can drop the support for them on Windows.

I have no idea do we even have users on Python 2.7 + Windows, but with the correct compiler installed, pip will work even without wheels.

@ilatypov

This comment has been minimized.

Show comment
Hide comment
@ilatypov

ilatypov Aug 31, 2018

Sorry I was only concerned with the error message and have little knowledge of this package or even the Python packagers and installers. Perhaps I missed this, but is there a way to not use Cython at install time? I agree pystan may require Cython at runtime, but resolving the install time error can still allow the runtime use of Cython.

ilatypov commented Aug 31, 2018

Sorry I was only concerned with the error message and have little knowledge of this package or even the Python packagers and installers. Perhaps I missed this, but is there a way to not use Cython at install time? I agree pystan may require Cython at runtime, but resolving the install time error can still allow the runtime use of Cython.

@ahartikainen

This comment has been minimized.

Show comment
Hide comment
@ahartikainen

ahartikainen Aug 31, 2018

Collaborator

Yes, when wheels are build, it does not need to compile anything at the install time.

We have not yet released wheels for python 3.7, but we are working on it, so next release should have them.

Collaborator

ahartikainen commented Aug 31, 2018

Yes, when wheels are build, it does not need to compile anything at the install time.

We have not yet released wheels for python 3.7, but we are working on it, so next release should have them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment