-
Notifications
You must be signed in to change notification settings - Fork 392
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
TST: Test that packages can be build from sdists. #532
Conversation
I'm having trouble with this locally.
This should allow builds from the sdist to complete.
Hi @DWesl! Thank you very much for the contribution. You are totally right, I was quite focused on creating working wheels and I didn't realise that the sdist zip is missing the requirements files that later |
I appear to have broken the test workflow, though similar commands work fine on my laptop. Should I drop the attempt to test the source distribution and just include the changes to |
Most of the failures are coming because the precompiled GEOS library is not being found, and then all references to its symbols are identified as undeclared. I will need to take a look on it, probably during this evening. |
Old Python versions will not be using this anyway.
Sorry for coming late to the PR, I was a bit busy this week. For the moment I reverted the changes in the workflows, since they break the pipelines. I am not fully aware on how The most important part is your addition of the Something that would be easy to implement in the GitHub Actions is to also try to install the packages from the source distribution. At the moment it only tries to install the wheels. If I had added this in the test job, this error would have appeared before. |
It does look straightforward to build from the sdist; it might be as simple as changing the |
@DWesl Then that is the reason for
Once From what I see, I was a bit fast reverting the changes in the |
From a fast check in basemap/packages/basemap/setup.py Lines 96 to 107 in ba6e90d
So your approach with |
Getting an older version of NumPy into the GitHub Actions should be defining a variable with the directory the package gets checked out in, so it should be easy to make the directory variables absolute paths once I find the name of that variable and figure out how to set variables in PowerShell. |
@DWesl The environment variable with the working directory for checkout is Thanks for discovering me |
I was checking about the For a normal new project it would be ok, but there is a lot of legacy code depending on I would therefore prefer to rely on defining |
So would I'm keeping the NUMPY_INCLUDE_PATH steps in, since deleting it feels like more work. |
@DWesl I think it is a good idea to add the Python version markers in the |
Okay, I'll start pulling over the NumPy versions from the workflow files and the Cython versions from requirements-setup.txt. |
For cython I think you can keep it simple in the |
…r python 2.{6,7} and 3.4+
The dropped python_versions probobly won't be using pyproject.toml.
I have one question: the current If your last changes make the workflows pass, it would be possible to simplify some parts of the YAML files, in particular this: basemap/.github/workflows/basemap-for-manylinux.yml Lines 116 to 135 in ba6e90d
and this: basemap/.github/workflows/basemap-for-windows.yml Lines 134 to 156 in ba6e90d
Because all this manual handling of generating the |
Correct; the |
Some older workflow has failed (Python 3.8) because building NumPy 1.17.3 failed (it looks like for this version we would need to define
This would replace the previous |
The new method is (or was, at least) working better on python 2 than python 3. I really did not expect that. I opened pypa/pip#10883 to flag the issue with |
cd ${{ env.PKGDIR }} | ||
$env:GEOS_DIR = "extern" | ||
$env:NUMPY_INCLUDE_PATH = "extern/include" | ||
$env:GEOS_DIR = "$env:GITHUB_WORKSPACE/${{ env.PKGDIR }}/extern" | ||
$env:NUMPY_INCLUDE_PATH = "$env:GITHUB_WORKSPACE/${{ env.PKGDIR }}/extern/include" | ||
pip install -r requirements-setup.txt | ||
python setup.py sdist bdist_wheel | ||
python setup.py sdist | ||
pip wheel -w dist --no-deps (Get-Item ${{ env.PKGDIR }}/dist/*.zip) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wouldn't the cd ${{ env.PKKGDIR }}
on line 166 make the ${{ env.PKGDIR }}
in the Get-Item
on line 171 unnecessary?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, you are totally right. I messed it up there.
Given that generating from ZIP-format sdists then building a wheel from them keeps running into problems, would it make more sense to revert the wheel build to |
@DWesl What about doing |
@DWesl I was just thinking on my comment before and it is not a good idea, since we can only guarantee that we can build the wheel from the tar but not from the zip, which is what you will find in the end in PyPI when trying to install |
It would appear |
It seems that having |
0f16ca8
to
fcb8289
Compare
It was the best way to specify build-time dependencies until |
The limits for the Python versions are not that strict I would say. I took these requirement files from some templates that I have that would fit for I think it would be safe e.g. to remove the references to Python 3.2, since it brings other problems (pip < 8 therefore no manylinux wheel support, the I think Python 2.6-2.7 and 3.5+ is a good deal, Python 2.6 is not difficult to maintain because |
@DWesl I just merged the PR. Could you check if you can build from the sdist in your environment (Windows with Cygwin)? You can pick the zip file from the artifacts stored in the GitHub Actions. |
$ unzip artifacts-build-x86-2.7.zip
Archive: artifacts-build-x86-2.7.zip
inflating: basemap-1.3.1+dev-cp27-cp27mu-linux_i686.whl
inflating: basemap-1.3.1+dev-cp27-cp27mu-manylinux1_i686.whl
inflating: basemap-1.3.1+dev.zip
$ for ver in 2.7 3.{5,6,7,8,9};
> do
> python${ver} -m pip install basemap-1.3.1+dev.zip
> done
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
...
Building wheels for collected packages: basemap
Building wheel for basemap (PEP 517) ... done
Created wheel for basemap: filename=basemap-1.3.1+dev-cp27-cp27m-cygwin_3_3_4_x86_64.whl size=207644 sha256=b13ceea5af1eb639af8342f4d3230ab9577b65621038940e39cc8418618abe0d
Stored in directory: ${HOME}/.cache/pip/wheels/18/61/bc/04286efe1a42778576b35bebbf76986ee8939cecf85e357ecf
Successfully built basemap
...
Successfully installed basemap-1.3.1+dev
DEPRECATION: Python 3.5 reached the end of its life on September 13th, 2020. Please upgrade your Python as Python 3.5 is no longer maintained. pip 21.0 will drop support for Python 3.5 in January 2021. pip 21.0 will remove support for this functionality.
...
Building wheels for collected packages: basemap
Building wheel for basemap (PEP 517) ... done
Created wheel for basemap: filename=basemap-1.3.1+dev-cp35-cp35m-cygwin_3_3_4_x86_64.whl size=204099 sha256=b74dcd2ed16675e76dc5089809369156e24a7d597117334b561ddd6b0081d9ba
Stored in directory: ${HOME}/.cache/pip/wheels/7a/1b/3f/4df0af4d794f0daf480be0e561be949bb26f5470f5d03bdb8f
Successfully built basemap
...
Successfully installed basemap-1.3.1+dev six-1.15.0
... I think it works |
@DWesl May I ask you to try one last time using the latest sdist available in the GitHub Actions? It is labelled as 1.3.2. I made slight changes after the PR and would like to be sure that everything works for you before publishing to PyPI. |
$ python -m pip install basemap-1.3.2.zip
Defaulting to user installation because normal site-packages is not writeable
Looking in links: /usr/share/python-wheels
Processing ./basemap-1.3.2.zip
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: numpy<1.23,>=1.21 in ${HOME}/.local/lib/python3.9/site-packages (from basemap==1.3.2) (1.21.5)
Requirement already satisfied: pyproj<3.4.0,>=1.9.3 in /usr/lib/python3.9/site-packages (from basemap==1.3.2) (3.3.0)
Requirement already satisfied: matplotlib<3.6,>=1.5 in /usr/lib/python3.9/site-packages (from basemap==1.3.2) (3.5.1)
Requirement already satisfied: pyshp<2.2,>=1.2 in ${HOME}/.local/lib/python3.9/site-packages (from basemap==1.3.2) (2.1.3)
Requirement already satisfied: basemap-data<1.4,>=1.3 in /usr/lib/python3.9/site-packages (from basemap==1.3.2) (1.3.0)
Requirement already satisfied: six<1.16,>=1.10 in ${HOME}/.local/lib/python3.9/site-packages (from basemap==1.3.2) (1.15.0)
Requirement already satisfied: cycler>=0.10 in /usr/lib/python3.9/site-packages (from matplotlib<3.6,>=1.5->basemap==1.3.2) (0.11.0)
Requirement already satisfied: python-dateutil>=2.7 in /usr/lib/python3.9/site-packages (from matplotlib<3.6,>=1.5->basemap==1.3.2) (2.8.2)
Requirement already satisfied: fonttools>=4.22.0 in /usr/lib/python3.9/site-packages (from matplotlib<3.6,>=1.5->basemap==1.3.2) (4.28.5)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/lib/python3.9/site-packages (from matplotlib<3.6,>=1.5->basemap==1.3.2) (1.3.2)
Requirement already satisfied: pyparsing>=2.2.1 in /usr/lib/python3.9/site-packages (from matplotlib<3.6,>=1.5->basemap==1.3.2) (3.0.6)
Requirement already satisfied: packaging>=20.0 in /usr/lib/python3.9/site-packages (from matplotlib<3.6,>=1.5->basemap==1.3.2) (21.3)
Requirement already satisfied: pillow>=6.2.0 in /usr/lib/python3.9/site-packages (from matplotlib<3.6,>=1.5->basemap==1.3.2) (9.0.0)
Requirement already satisfied: certifi in /usr/lib/python3.9/site-packages (from pyproj<3.4.0,>=1.9.3->basemap==1.3.2) (2021.10.8)
Building wheels for collected packages: basemap
Building wheel for basemap (pyproject.toml) ... done
Created wheel for basemap: filename=basemap-1.3.2-cp39-cp39-cygwin_3_3_4_x86_64.whl size=235019 sha256=fcb6645099406b4d65a107325175e307d95ccc038c59ac39b978be5725d612ee
Stored in directory: ${HOME}/.cache/pip/wheels/3e/8c/72/1d6fd45bb09223f56e9c1d6845fce9c2f6e4f14ae77252eb2e
Successfully built basemap
Installing collected packages: basemap
Attempting uninstall: basemap
Found existing installation: basemap 1.3.1+dev
Uninstalling basemap-1.3.1+dev:
Successfully uninstalled basemap-1.3.1+dev
Successfully installed basemap-1.3.2 Looks like it works. |
@DWesl Thanks for the effort of the pull request and the testing. |
I ran into problems trying to install basemap on a somewhat unusual platform (Cygwin, but the relevant bit is that it doesn't have binary wheels).