Closed
Description
Description
I have used cmake to precompiled a .so file. But when I use build-python to create python support. It ended up create a unknown-0.0.0 package.
Reproducible example
sudo apt-get install --no-install-recommends git cmake build-essential libboost-dev libboost-system-dev libboost-filesystem-dev
sudo mkdir -p /mnt/workspace
sudo chown $(whoami):$(whoami) /mnt/workspace
cd /mnt/workspace
git clone --recursive https://github.com/microsoft/LightGBM
cd LightGBM
cmake -B build -S . -DUSE_GPU=1
cmake --build build -j$(nproc)
apt-get install python3-venv
pip install setuptools numpy scipy scikit-learn -U
sudo sh ./build-python.sh install --precompile
Environment info
LightGBM version or commit hash:
Command(s) you used to install LightGBM
root@5567918b09b1:/mnt/workspace/LightGBM# ./build-python.sh install --precompile
building lightgbm
Requirement already satisfied: build>=0.10.0 in /usr/local/lib/python3.10/dist-packages (1.2.2.post1)
Requirement already satisfied: pyproject_hooks in /usr/local/lib/python3.10/dist-packages (from build>=0.10.0) (1.2.0)
Requirement already satisfied: packaging>=19.1 in /usr/local/lib/python3.10/dist-packages (from build>=0.10.0) (24.1)
Requirement already satisfied: tomli>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from build>=0.10.0) (2.0.2)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
found pre-compiled lib_lightgbm.so
--- building sdist ---
* Creating isolated environment: virtualenv+pip...
* Installing packages in isolated environment:
- setuptools
* Getting build dependencies for sdist...
running egg_info
creating lightgbm.egg-info
writing lightgbm.egg-info/PKG-INFO
writing dependency_links to lightgbm.egg-info/dependency_links.txt
writing requirements to lightgbm.egg-info/requires.txt
writing top-level names to lightgbm.egg-info/top_level.txt
writing manifest file 'lightgbm.egg-info/SOURCES.txt'
reading manifest file 'lightgbm.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '*.dll' under directory 'lightgbm'
warning: no files found matching '*.dylib' under directory 'lightgbm'
adding license file 'LICENSE'
writing manifest file 'lightgbm.egg-info/SOURCES.txt'
* Building sdist...
running sdist
running egg_info
writing lightgbm.egg-info/PKG-INFO
writing dependency_links to lightgbm.egg-info/dependency_links.txt
writing requirements to lightgbm.egg-info/requires.txt
writing top-level names to lightgbm.egg-info/top_level.txt
reading manifest file 'lightgbm.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '*.dll' under directory 'lightgbm'
warning: no files found matching '*.dylib' under directory 'lightgbm'
adding license file 'LICENSE'
writing manifest file 'lightgbm.egg-info/SOURCES.txt'
running check
creating lightgbm-4.5.0.99
creating lightgbm-4.5.0.99/lightgbm
creating lightgbm-4.5.0.99/lightgbm.egg-info
creating lightgbm-4.5.0.99/lightgbm/lib
copying files to lightgbm-4.5.0.99...
copying LICENSE -> lightgbm-4.5.0.99
copying MANIFEST.in -> lightgbm-4.5.0.99
copying README.rst -> lightgbm-4.5.0.99
copying pyproject.toml -> lightgbm-4.5.0.99
copying lightgbm/__init__.py -> lightgbm-4.5.0.99/lightgbm
copying lightgbm/basic.py -> lightgbm-4.5.0.99/lightgbm
copying lightgbm/callback.py -> lightgbm-4.5.0.99/lightgbm
copying lightgbm/compat.py -> lightgbm-4.5.0.99/lightgbm
copying lightgbm/dask.py -> lightgbm-4.5.0.99/lightgbm
copying lightgbm/engine.py -> lightgbm-4.5.0.99/lightgbm
copying lightgbm/libpath.py -> lightgbm-4.5.0.99/lightgbm
copying lightgbm/plotting.py -> lightgbm-4.5.0.99/lightgbm
copying lightgbm/py.typed -> lightgbm-4.5.0.99/lightgbm
copying lightgbm/sklearn.py -> lightgbm-4.5.0.99/lightgbm
copying lightgbm.egg-info/PKG-INFO -> lightgbm-4.5.0.99/lightgbm.egg-info
copying lightgbm.egg-info/SOURCES.txt -> lightgbm-4.5.0.99/lightgbm.egg-info
copying lightgbm.egg-info/dependency_links.txt -> lightgbm-4.5.0.99/lightgbm.egg-info
copying lightgbm.egg-info/requires.txt -> lightgbm-4.5.0.99/lightgbm.egg-info
copying lightgbm.egg-info/top_level.txt -> lightgbm-4.5.0.99/lightgbm.egg-info
copying lightgbm/lib/lib_lightgbm.so -> lightgbm-4.5.0.99/lightgbm/lib
copying lightgbm.egg-info/SOURCES.txt -> lightgbm-4.5.0.99/lightgbm.egg-info
Writing lightgbm-4.5.0.99/setup.cfg
Creating tar archive
removing 'lightgbm-4.5.0.99' (and everything under it)
Successfully built lightgbm-4.5.0.99.tar.gz
--- installing lightgbm ---
Looking in links: .
Processing ./lightgbm-4.5.0.99.tar.gz
Installing build dependencies ... done
Getting requirements to build wheel ... done
Installing backend dependencies ... done
Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: UNKNOWN
Building wheel for UNKNOWN (pyproject.toml) ... done
Created wheel for UNKNOWN: filename=UNKNOWN-0.0.0-py3-none-any.whl size=1794 sha256=31cf85902f1e3ffcb3b79747c9d3629bdf509a4dfe6f2dda9c7bf91c58ce4506
Stored in directory: /tmp/pip-ephem-wheel-cache-bfyn99ka/wheels/97/48/5c/b2fa3ad0b5ff4da83888744e25e16ce761d05dc33923b317f1
Successfully built UNKNOWN
Installing collected packages: UNKNOWN
Attempting uninstall: UNKNOWN
Found existing installation: UNKNOWN 0.0.0
Uninstalling UNKNOWN-0.0.0:
Successfully uninstalled UNKNOWN-0.0.0
Successfully installed UNKNOWN-0.0.0
Activity
jameslamb commentedon Oct 7, 2024
What Linux distribution are you using? That'd help us to replicate this.
ZhenyiLin commentedon Oct 7, 2024
[-]UNKNOWN python package[/-][+][python-package] UNKNOWN python package[/+]jameslamb commentedon Oct 14, 2024
I was able to reproduce this in an Ubuntu 22.04 container today (on my M2 mac).
logs (click me)
Later, I'll try removing this line so I can inspect the build directory:
LightGBM/build-python.sh
Line 379 in bbeecc0
I suspect this MIGHT be related to #6624. Will provide more updates soon.
jameslamb commentedon Oct 15, 2024
Nope, this was not it.
@ZhenyiLin I found that just updating to a more recent
pip
resolves this. Can you please try that?I saw that work with
pip
24.2.pip --version # pip 24.2 from /usr/local/lib/python3.10/dist-packages/pip (python 3.10)
I'm not sure what range of versions will work.
Root Cause
It looks like
python3-pip
/python3-venv
on Ubuntu 22.04 ships withpip==22.0.2
andsetuptools==59.6.0
.setuptools
did not add support forpyproject.toml
until v61.0.0 (see the setuptools docs).Your script is running
pip install -U setuptools
(where-U
is an alias for--upgrade
), which installs the latest compatiblesetuptools
at/usr/local/lib/python3.10/dist-packages
. But I think the system-installedpip
at/usr/lib/python3/dist-packages
is ignoring that and loading thesetuptools
at/usr/lib/python3/dist-packages
.pip install -U pip
puts a newpip
into/usr/local/lib/python3.10/dist-packages
, which then uses the newersetuptools
there.What
lightgbm
can doWe could have
build-python.sh
write asetup.cfg
file when--precompile
is passed, so other users won't face this.Ubuntu 22.04 is only 2.5 years... it won't reach End of Standard Support until April 2027 (https://ubuntu.com/about/release-cycle). I think this is very worth supporting.
[ci] [python-package] support setuptools<61 in build-python.sh (fixes #…
ZhenyiLin commentedon Oct 16, 2024
Thanks, updating pip works!