Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for non pure python projects #11

Closed
sdispater opened this issue Mar 9, 2018 · 16 comments
Closed

Add support for non pure python projects #11

sdispater opened this issue Mar 9, 2018 · 16 comments
Labels
Feature

Comments

@sdispater
Copy link
Member

@sdispater sdispater commented Mar 9, 2018

We should have a way to build non pure-python projects (from instance if they need to build C extensions).

@feluxe
Copy link

@feluxe feluxe commented Apr 7, 2018

Hi!

Does this work with poetry?

Impressive work by the way! :)

@sdispater
Copy link
Member Author

@sdispater sdispater commented Apr 7, 2018

Thanks for your interest in poetry!

Yes, it works even though it's not currently documented because I do not know if it will stay this way since it relies on adding kwargs to the setup.py.

Anyway, it you want to test this, you have to specifiy a build script in your pyproject.toml:

[tool.poetry]
build = 'build.py'

that will define a build function that will add the necessary build keyword arguments to the call to setup() in the generated setup.py.

An example of this is in my pendulum project:

@feluxe
Copy link

@feluxe feluxe commented Apr 8, 2018

This look promising! Thanks!

I'll give it a shot for my next package.

@bruxisma
Copy link

@bruxisma bruxisma commented Sep 3, 2018

I know this issue is closed at this point, but I wanted to say I'm so glad this is here. I've been struggling with pipenv so much that I've stared into the abyss and we're now on a first name basis.

I'm currently working a library called brujería, that lets you treat your C and C++ modules as if they were python modules (both for REPL imports as well as setuptools builds). I'm still working out the kinks (I have to write another library so it can be decoupled from all of CMake's garbage), but I think it might help others in this situation. It's not in a working state at the moment as I'm doing janitorial work on both repos, but I still wanted to mention it also, @sdispater thank you for poetry. It's been so nice and I've only been using it for 10 minutes so far. :)

@salotz
Copy link

@salotz salotz commented Oct 11, 2018

Here is an example for building a cython module https://github.com/ADicksonLab/geomm/blob/master/build.py.

@shimil
Copy link

@shimil shimil commented May 14, 2019

Hey @sdispater, I love Poetry!
We want to adopt it as part of our tech stack, but consider support for C extensions an important feature. Can we count on continued support for it?

Many thanks

@AlJohri
Copy link

@AlJohri AlJohri commented Dec 20, 2019

How can I use this build.py idea for external Cython dependencies? Currently I want to install (from source) the spaCy package. I have to currently resort to using poetry run to achieve this.

# install non-cython dependencies
poetry install

# install spaCy from source using `build_ext`
git clone https://github.com/explosion/spaCy.git spaCy
poetry run python spaCy/setup.py build_ext --inplace
poetry run pip install -e spaCy

@zoj613
Copy link

@zoj613 zoj613 commented Dec 20, 2019

How can I use this build.py idea for external Cython dependencies? Currently I want to install (from source) the spaCy package. I have to currently resort to using poetry run to achieve this.

# install non-cython dependencies
poetry install

# install spaCy from source using `build_ext`
git clone https://github.com/explosion/spaCy.git spaCy
poetry run python spaCy/setup.py build_ext --inplace
poetry run pip install -e spaCy

you could bundle these steps as one Makefile command? Something like

poetry:
     poetry install
     git clone https://github.com/explosion/spaCy.git spaCy
     poetry run python spaCy/setup.py build_ext --inplace
     poetry run pip install -e spaCy

Then run all them using make poetry

@AlJohri
Copy link

@AlJohri AlJohri commented Dec 20, 2019

@zoj613 and I've certainly done exactly that (see screenshot). but I wish I could specify spaCy as a git project directly in the pyproject.toml with some markers specifying that the cython extension needs to be built. is there anyway to do this?

Screen Shot 2019-12-20 at 1 01 41 PM

@zoj613
Copy link

@zoj613 zoj613 commented Dec 20, 2019

@zoj613 and I've certainly done exactly that (see screenshot). but I wish I could specify spaCy as a git project directly in the pyproject.toml with some markers specifying that the cython extension needs to be built. is there anyway to do this?

Screen Shot 2019-12-20 at 1 01 41 PM

does it not build and install the cython packages by default when you install spaCy? If that is the case then you can just add the line
spaCy = {git = "https://github.com/explosion/spaCy.git"}
in your package list of pyproject.toml.

If it doesnt build it by default then I would clone the repo, modify the setup file to automatically build the Cython packages, then use your fork as the source in the pyrpoject.toml file like:
spaCy = {git = "https://github.com/AIJohri/spaCy.git"}

@AlJohri
Copy link

@AlJohri AlJohri commented Dec 20, 2019

It doesn’t, it says that it can’t find the package version?

For some reason when pip tries to install it, it installs it on some sort of isolated environment without Cython which is why it fails to parse the setup.py.

I tried forking as you suggested but couldn’t figure out how to get past the Cython not found issue. I can post a dockerized example if you’re interested in debugging further.

@AlJohri
Copy link

@AlJohri AlJohri commented Dec 21, 2019

@zoj613 in case you're curious, I found the root case was that the spaCy package has a pyproject.toml without the build dependencies defined: explosion/spaCy#4829

@teknico
Copy link

@teknico teknico commented Apr 10, 2020

We're also using this feature in tinydecred with Cython. Any chance in stabilizing it? Thanks.

@abn
Copy link
Member

@abn abn commented Apr 10, 2020

@teknico I'd recommend creating a new issue detailing short-commings of the current implementations with as much details as possible please. We are already aware of a few issues. Including #1516.

@geronm
Copy link

@geronm geronm commented Aug 18, 2020

salotz said:

Here is an example for building a cython module https://github.com/ADicksonLab/geomm/blob/master/build.py.

This comment seems to have fallen out of date.

I think this tag (v0.1.7) in the repo gives a stable reference with working build.py: https://github.com/ADicksonLab/geomm/tree/v0.1.7

@zoj613
Copy link

@zoj613 zoj613 commented Nov 28, 2020

I have a mordern approach that avoids using an explicit setup.py file. It contains a build script that takes wheel building into account. See: https://github.com/zoj613/htnorm/blob/main/build.py

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Feature
Projects
None yet
Development

No branches or pull requests