support pip installation

[anthrotype]

there is a problem if one tries to build the Python extension using the pip installer. The problem is that the setup.py is not located in the root of the repository, but in the python binding subfolder. The way pip works is to copy the source files to a temporary folder and try to build from there. But since the C/C++ source files are located higher in the repository tree (../enc, ../dec) relative to setup.py, then pip does not (cannot?) copy these over to its temporary build folder, and therefore it fails to build the extension.

A solution would be to make a hard copy (instead of a symlink) of the enc and dec folders inside the python subfolder. I believe git can efficiently handle such duplicate files and store them under the same object, as long as they have the same content.

That means one would have to synchronise it every time there is a change. I don't know if anyone has a better solution...

The reason I want to use pip is to allow publishing the Brotli extension to the official Python Package Index (PyPI) repository. Once that is done, one could simply do pip install brotli to download, build and install the extension. Besides, one could add brotli to the list of dependencies for other packages (e.g. fontTools, etc.).

I wonder whether @khaledhosny has already thought about publishing the Brotli extension to PyPI?

I'm already experimenting in this direction in https://github.com/anthrotype/brotli-wheels
I'm trying to use Travis and Appveyor to automatically build pre-compiled Python wheel packages for Windows and OS X.

Please let me know what you think.
Thank you,

Cosimo

[khaledhosny]

How are you building with pip? I do pip install -e . routinely and had no issue with it.

[anthrotype]

I know, but that only runs python setup develop for testing a package while developing it.
The problem arises when doing pip install ., without the -e option.
If you look at the traceback, the reason why pip fails is because it cannot find the C/C++ sources, since they are referenced using a relative path which is "above" the setup.py working directory:

$ pip install .
Processing /Users/cosimolupo/Documents/Github/brotli/python
Installing collected packages: Brotli
  Running setup.py install for Brotli
    clang -fno-strict-aliasing -fno-common -dynamic -I/usr/local/include -I/usr/local/opt/sqlite/include -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -DOS_MACOSX=1 -I/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c /private/var/folders/1w/6098jtks69sgwz0vtz2ysmsm0000gn/T/dec/bit_reader.c -o build/temp.macosx-10.9-x86_64-2.7/private/var/folders/1w/6098jtks69sgwz0vtz2ysmsm0000gn/T/dec/bit_reader.o
    clang: error: no such file or directory: '/private/var/folders/1w/6098jtks69sgwz0vtz2ysmsm0000gn/T/dec/bit_reader.c'
    clang: error: no input files
    error: command 'clang' failed with exit status 1
    Complete output from command /usr/local/opt/python/bin/python2.7 -c "import setuptools, tokenize;__file__='/var/folders/1w/6098jtks69sgwz0vtz2ysmsm0000gn/T/pip-oaQuzp-build/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/1w/6098jtks69sgwz0vtz2ysmsm0000gn/T/pip-R4EIB0-record/install-record.txt --single-version-externally-managed --compile:
    running install

    running build

    running build_ext

    creating build

    creating build/temp.macosx-10.9-x86_64-2.7

    creating build/temp.macosx-10.9-x86_64-2.7/private

    creating build/temp.macosx-10.9-x86_64-2.7/private/var

    creating build/temp.macosx-10.9-x86_64-2.7/private/var/folders

    creating build/temp.macosx-10.9-x86_64-2.7/private/var/folders/1w

    creating build/temp.macosx-10.9-x86_64-2.7/private/var/folders/1w/6098jtks69sgwz0vtz2ysmsm0000gn

    creating build/temp.macosx-10.9-x86_64-2.7/private/var/folders/1w/6098jtks69sgwz0vtz2ysmsm0000gn/T

    creating build/temp.macosx-10.9-x86_64-2.7/private/var/folders/1w/6098jtks69sgwz0vtz2ysmsm0000gn/T/dec

    clang -fno-strict-aliasing -fno-common -dynamic -I/usr/local/include -I/usr/local/opt/sqlite/include -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -DOS_MACOSX=1 -I/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c /private/var/folders/1w/6098jtks69sgwz0vtz2ysmsm0000gn/T/dec/bit_reader.c -o build/temp.macosx-10.9-x86_64-2.7/private/var/folders/1w/6098jtks69sgwz0vtz2ysmsm0000gn/T/dec/bit_reader.o

    clang: error: no such file or directory: '/private/var/folders/1w/6098jtks69sgwz0vtz2ysmsm0000gn/T/dec/bit_reader.c'

    clang: error: no input files

    error: command 'clang' failed with exit status 1

    ----------------------------------------
    Command "/usr/local/opt/python/bin/python2.7 -c "import setuptools, tokenize;__file__='/var/folders/1w/6098jtks69sgwz0vtz2ysmsm0000gn/T/pip-oaQuzp-build/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/1w/6098jtks69sgwz0vtz2ysmsm0000gn/T/pip-R4EIB0-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /var/folders/1w/6098jtks69sgwz0vtz2ysmsm0000gn/T/pip-oaQuzp-build

[khaledhosny]

I think we need to fix sdist command, since it seems not to copy any of the Brotli source files. If this does not fix pip, it will at least allow uploading to PyPI.

[anthrotype]

I'll have look at sdist.
The thing is, the way pip (but also distutils, setuptools, etc.) expects python packages to be organised is with a setup.py at the root of the package, and then everything else positioned either on the same level or inside sub-folders reachable from the root. Whereas here we want to keep the python folder isolated from the C/C++ source files, which is understandable, but this seems to conflict with the way Python packaging works. Maybe the solution could simply be to clone brotli as a submodule in another repository with a copy of setup.py positioned higher in the directory tree.

[khaledhosny]

Maybe the solution could simply be to clone brotli as a submodule in another repository with a copy of setup.py positioned higher in the directory tree.

That is how it was done originally :)

[anthrotype]

yes, I do remember.

[khaledhosny]

Thinking about it more, I think it shouldn’t harm to move the setup.py file to the top level directory, and leave the C file where it is, if it will simplify things.

[anthrotype]

It would definitely make the Python extension's setup much easier.

On Tue, Mar 31, 2015 at 8:42 AM, Khaled Hosny notifications@github.com
wrote:

Thinking about it more, I think it shouldn’t harm to move the setup.py file to the top level directory, and leave the C file where it is, if it will simplify things.

Reply to this email directly or view it on GitHub:
#52 (comment)

[szabadka]

If it involves just the setup.py file, it should be ok to have it in the top-level directory.

[anthrotype]

I can confirm pip is happy after 5e3e97e