Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to use cibuildwheel under this "unusual" setup #13

Closed
sgeto opened this issue Jun 29, 2017 · 16 comments
Closed

How to use cibuildwheel under this "unusual" setup #13

sgeto opened this issue Jun 29, 2017 · 16 comments

Comments

@sgeto
Copy link

sgeto commented Jun 29, 2017

Hi @joerick

I could use some help.
The wheels I'm trying to build are a binding for a C library. The setup.py file is located in a sub folder (python) where the python build takes place (as part of the overall build or seperatly).
However, during build the compiler needs access to additional header and source file located in other sub folders (src include) located on the same level as the python folder.

cibuildwheel seems to only import the folder holding the setup.py file. And the build sadly fails. I tried everything. Setting global environment variables, adding the whole src for as project and adding cibuildwheel pre-build commands.

How can I use cibuildwheel in such a setup?

Thanks a lot

Project: https://github.com/sgeto/libdnet
Travis: https://travis-ci.org/sgeto/libdnet/

@joerick
Copy link
Contributor

joerick commented Jul 3, 2017

cibuildwheel does take a project path, like

cibuildwheel --output-dir wheelhouse python

But in your case, this is going to be a problem because cibuildwheel mounts the project folder on Linux, so the build machine will only have access to the python folder, not the sibling folders.

I can't think of an easy solution we could make... maybe it would work if you moved the setup.py to the root of the repo and build from there?

@sgeto
Copy link
Author

sgeto commented Jul 4, 2017

If cibuildwheel would not look for a setup.py file when attempting to mount the project folder, I could import the whole repo and add e.g. cd python as a pre-build command.
Do you think an option to not check for the presents of a setup.py file (at mount time) is a good idea?

@sgeto
Copy link
Author

sgeto commented Jul 4, 2017

By the way moving setup.py to the root of the repo is a possibility

@joerick
Copy link
Contributor

joerick commented Jul 4, 2017

Ah, the pre-build command executes in a subshell, so cd'ing won't affect the working dir.

@sgeto
Copy link
Author

sgeto commented Jul 4, 2017

I'm not sure I'm following. Is that good?

@joerick
Copy link
Contributor

joerick commented Jul 5, 2017

Sorry, I mean cd'ing won't work the way you're thinking because it won't change cibuildwheel's working dir.

@sgeto
Copy link
Author

sgeto commented Jul 5, 2017

I see.

I can't think of an easy solution we could make

Can you tell me more about the not so easy solution? Is it related to docker or more of a python limitation?

@tgarc
Copy link
Contributor

tgarc commented Jul 9, 2017

@sgeto I don't think your project setup is uncommon - it just isn't primarily a python package. cibuildwheel is still in early stages so I imagine these kind of use cases likely haven't come up yet. Anyway, just to rephrase the issue you're having:

Your setup.py is not in your root folder. cibuildwheel doesn't currently support this configuration as it expects everything required to be contained under the directory where setup.py is located.

In the future it probably makes sense to directly support configurations like yours - perhaps by allowing passing of additional options to docker and/or allowing a custom path for setup.py. But as a temporary workaround...if you moved your setup.py to your root project folder (and made the necessary changes to your setup.py) would that fix your issue?

@sgeto
Copy link
Author

sgeto commented Jul 9, 2017

@tgarc moving the setup.py file to für root directory fixes the issue.
Don't get me wrong, I am not complaining. I'd like to see this work. So I was just wondering what it would take and where to start.

@tgarc
Copy link
Contributor

tgarc commented Jul 9, 2017

Glad to hear that worked. (If I'm understanding you right.) Maybe adding a CIBW_SETUP_DIR to specify a folder containg setup.py that is different from the root would help in cases like yours.

@sgeto
Copy link
Author

sgeto commented Jul 9, 2017

It did, thanks.
I'll give it a try and report back.

@joerick
Copy link
Contributor

joerick commented Sep 7, 2017

I'm closing this out for now. Thanks @tgarc, I think CIBW_SETUP_DIR could be a solution to this problem. I don't want to implement based on only one data point though, so if we get a little more demand for this then we can consider implementing.

Comment below if this is a feature you'd find useful for your project!

@joerick joerick closed this as completed Sep 7, 2017
@SooluThomas
Copy link

Hi, I got into similar issue recently. Is there anyway to implement CIBW_SETUP_DIR flag? I can help contribute if I was pointed to the right direction

@henryiii
Copy link
Contributor

henryiii commented Jul 7, 2021

I have a feeling you wouldn't be helped by it. Lots of other parts of the tooling including pip will assume the root directory contains all the code and will copy it before building. I think your best bet is to put the project files at the top level (in a before-all script if you can't physically move them).

Does your SDist work? You should have a working SDist if possible.

@oliver-zehentleitner
Copy link

My project consists of several packages and several build actions.

This means that I have setup.py and pyproject.toml in a subdirectory.

I would also like to be able to say: @cibuildwheel, please cd to “xx/yy/ first and then start.

@oliver-zehentleitner
Copy link

I was able to successfully test this workaround for my problem. In the GitHub action, I revised the checkout as follows. Thus, the desired subdirectory is the root directory of CIBuildWheels:

steps:
  - name: Checkout specific directory
    uses: actions/checkout@v4
    with:
      repository: ${{ github.repository }}
      ref: ${{ github.ref }}
      path: your-repository
      sparse-checkout: 'path/to/your/subdirectory/'

  - name: Move directory contents
    run: |
      mv your-repository/path/to/your/subdirectory/* ./
      rm -rf your-repository

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants