Page isn't finished: https://packaging.python.org/extensions/ #284

Open
zfrenchee opened this Issue Mar 22, 2017 · 6 comments

Comments

Projects
None yet
4 participants
@zfrenchee

I'm particularly interested in the last header: https://packaging.python.org/extensions/#publishing-binary-extensions which is completely empty =0

@ncoghlan

This comment has been minimized.

Show comment
Hide comment
@ncoghlan

ncoghlan Apr 9, 2017

Member

The two main options I'm currently aware of for this are:

  1. Use enscons as your build system (rather than distutils or setuptools): https://pypi.python.org/pypi/enscons/

That gives you a full-fledged build system to use (Scons), with some additions for also emitting Python packages as a build output, rather than having to figure out how to teach distutils/setuptools to build your extension modules.

  1. Tell distutils/setuptools how to build the extension: https://docs.python.org/3/extending/building.html#building-c-and-c-extensions-with-distutils

This option is OK for simple extension modules, including CFFI wrappers as described in http://cffi.readthedocs.io/en/latest/cdef.html

So a possible subsection structure that might work:

  • Publishing simple extension modules with setuptools

    • Pull the example from the CPython documentation into here and adapt it to use setuptools instead
  • Publishing CFFI extension modules with setuptools

    • Pull the setuptools specific snippet from the CFFI docs into here
  • Publishing complex extension modules with enscons

    • Explain that while possible, building complex extension modules directly with distutils or setuptools tends to be fragile and error-prone, so it's no longer a recommended approach
    • Point to enscons as a project that can be either used directly for projects that are willing and able to adopt SCons as their build system, or else used as the basis for wrapping another build system in an sdist-compatible build interface
  • Providing pre-built wheel files through PyPI

    • Explain the binary compatibility tagging system
    • Explain the stable ABI and the ability to preserve binary extension compatibility across releases

Other (also incomplete) sections that could potentially be referenced:

Member

ncoghlan commented Apr 9, 2017

The two main options I'm currently aware of for this are:

  1. Use enscons as your build system (rather than distutils or setuptools): https://pypi.python.org/pypi/enscons/

That gives you a full-fledged build system to use (Scons), with some additions for also emitting Python packages as a build output, rather than having to figure out how to teach distutils/setuptools to build your extension modules.

  1. Tell distutils/setuptools how to build the extension: https://docs.python.org/3/extending/building.html#building-c-and-c-extensions-with-distutils

This option is OK for simple extension modules, including CFFI wrappers as described in http://cffi.readthedocs.io/en/latest/cdef.html

So a possible subsection structure that might work:

  • Publishing simple extension modules with setuptools

    • Pull the example from the CPython documentation into here and adapt it to use setuptools instead
  • Publishing CFFI extension modules with setuptools

    • Pull the setuptools specific snippet from the CFFI docs into here
  • Publishing complex extension modules with enscons

    • Explain that while possible, building complex extension modules directly with distutils or setuptools tends to be fragile and error-prone, so it's no longer a recommended approach
    • Point to enscons as a project that can be either used directly for projects that are willing and able to adopt SCons as their build system, or else used as the basis for wrapping another build system in an sdist-compatible build interface
  • Providing pre-built wheel files through PyPI

    • Explain the binary compatibility tagging system
    • Explain the stable ABI and the ability to preserve binary extension compatibility across releases

Other (also incomplete) sections that could potentially be referenced:

@ncoghlan

This comment has been minimized.

Show comment
Hide comment
@ncoghlan

ncoghlan Apr 25, 2017

Member

I added a link to the CPython tutorial on writing extension modules by hand: 532268a

I also added a link from the publishing section back to this issue.

Member

ncoghlan commented Apr 25, 2017

I added a link to the CPython tutorial on writing extension modules by hand: 532268a

I also added a link from the publishing section back to this issue.

@ned-deily

This comment has been minimized.

Show comment
Hide comment
@ned-deily

ned-deily Jul 15, 2017

Perhaps another topic to include in this section, if not discussed elsewhere, is how to package extensions modules built with multiple ABIs, in particular, non-debug vs debug builds. For Python 3.x, PEP 3149 and follow-ons provide ABI-unique extension module file names so that they can be installed in the same directory. (Python 2 does not attempt to address this.)

Perhaps another topic to include in this section, if not discussed elsewhere, is how to package extensions modules built with multiple ABIs, in particular, non-debug vs debug builds. For Python 3.x, PEP 3149 and follow-ons provide ABI-unique extension module file names so that they can be installed in the same directory. (Python 2 does not attempt to address this.)

@asmodehn

This comment has been minimized.

Show comment
Hide comment
@asmodehn

asmodehn Sep 24, 2017

While looking around for this kind of information, I also found CMake python integration (see here and there)

Another (related) point:
While it is easier for the developer to build binary extension from the maintainer side, I find it "more friendly for the user" (ie. more pythonic/open) and easier to distribute, if the user installing the package also builds the extension. I believe I have seen some package taking this approach, but I haven't found any documentation about it.

While looking around for this kind of information, I also found CMake python integration (see here and there)

Another (related) point:
While it is easier for the developer to build binary extension from the maintainer side, I find it "more friendly for the user" (ie. more pythonic/open) and easier to distribute, if the user installing the package also builds the extension. I believe I have seen some package taking this approach, but I haven't found any documentation about it.

@ncoghlan

This comment has been minimized.

Show comment
Hide comment
@ncoghlan

ncoghlan Sep 24, 2017

Member

@asmodehn Yes, end users being free to build the extension for themselves is why we discourage wheel-only distribution - we prefer publishers to offer both an sdist (for folks that want to build from source) and wheels (for folks that may not even have the appropriate compilation toolchains available).

Member

ncoghlan commented Sep 24, 2017

@asmodehn Yes, end users being free to build the extension for themselves is why we discourage wheel-only distribution - we prefer publishers to offer both an sdist (for folks that want to build from source) and wheels (for folks that may not even have the appropriate compilation toolchains available).

@ncoghlan

This comment has been minimized.

Show comment
Hide comment
@ncoghlan

ncoghlan Sep 24, 2017

Member

scikit-build is another option for CMake integration: #381

Member

ncoghlan commented Sep 24, 2017

scikit-build is another option for CMake integration: #381

@kasbah kasbah referenced this issue in sambayless/monosat Dec 13, 2017

Open

PyPI package? #6

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment