python:: local installation of python client/bindings #1641
Comments
|
@matthewfeickert : could you please comment on this one (we really do appreciate your time and help!) |
|
oh, also, our build recipe is no longer working, in the sense that, python bindings are no longer found by our python installation |
|
more info: so this is the build log and i noticed this: which is wrong as this should not see the system (or user side) installation .. |
|
I will not have time to look at this seriously until after Snowmass white papers are submitted. As always, minimal reproducible examples are nice, so please provide one. |
|
Though to provide you something from #1594 (comment) $ docker run --rm -ti xrootd/xrootd:issue-1594
[root@9cc1f196bef4 /]# . /opt/rh/devtoolset-7/enable
[root@9cc1f196bef4 /]# python3 -m venv issue-1641
[root@9cc1f196bef4 /]# . issue-1641/bin/activate
(issue-1641) [root@9cc1f196bef4 /]# python -m pip --quiet install --upgrade pip setuptools wheel
Cache entry deserialization failed, entry ignored
(issue-1641) [root@9cc1f196bef4 /]# python -m pip --quiet install --upgrade build
(issue-1641) [root@9cc1f196bef4 /]# cd xrootd/
(issue-1641) [root@9cc1f196bef4 xrootd]# cp packaging/wheel/* .
(issue-1641) [root@9cc1f196bef4 xrootd]# python -m build --sdist .
...
Successfully built xrootd-unknown.tar.gz
(issue-1641) [root@9cc1f196bef4 xrootd]# cd .. # Move xrootd.egg-info off PYTHONPATH
(issue-1641) [root@9cc1f196bef4 /]# python -m pip install --upgrade xroot/dist/xrootd-unknown.tar.gz
Processing ./dist/xrootd-unknown.tar.gz
Preparing metadata (setup.py) ... done
Building wheels for collected packages: xrootd
Building wheel for xrootd (setup.py) ... done
WARNING: Legacy build of wheel for 'xrootd' created no files.
Command arguments: /issue-1641/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-m80_htkl/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-m80_htkl/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-9yheewvo
Command output: [use --verbose to show]
Running setup.py clean for xrootd
Failed to build xrootd
Installing collected packages: xrootd
Running setup.py install for xrootd ... done
Successfully installed xrootd-5.4.2
(issue-1641) [root@9cc1f196bef4 /]# python -m pip show xrootd
WARNING: No metadata found in /issue-1641/lib/python3.6/site-packages
WARNING: No metadata found in /issue-1641/lib/python3.6/site-packages
WARNING: No metadata found in /issue-1641/lib/python3.6/site-packages
Name: xrootd
Version: 5.4.2
Summary:
Home-page:
Author:
Author-email:
License:
Location: /issue-1641/lib/python3.6/site-packages
Requires:
Required-by: this is the exact thing done in the CI: xrootd/.github/workflows/build.yml Lines 570 to 577 in 3d64cda You could have done the same thing outside a venv too (issue-1641) [root@9cc1f196bef4 xrootd]# deactivate
[root@9cc1f196bef4 xrootd]# python3 -m pip --quiet install --upgrade pip setuptools wheel
[root@9cc1f196bef4 xrootd]# cd .. # Move xrootd.egg-info off PYTHONPATH
[root@9cc1f196bef4 /]# python3 -m pip install --upgrade --user xroot/dist/xrootd-unknown.tar.gz
Processing ./dist/xrootd-unknown.tar.gz
Preparing metadata (setup.py) ... done
Building wheels for collected packages: xrootd
Building wheel for xrootd (setup.py) ... done
WARNING: Legacy build of wheel for 'xrootd' created no files.
Command arguments: /usr/bin/python3 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-3ajy3hoe/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-3ajy3hoe/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-44hqsp87
Command output: [use --verbose to show]
Running setup.py clean for xrootd
Failed to build xrootd
Installing collected packages: xrootd
Running setup.py install for xrootd ... done
Successfully installed xrootd-5.4.2
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
[root@9cc1f196bef4 /]# python3 -m pip show xrootd
WARNING: No metadata found in /root/.local/lib/python3.6/site-packages
WARNING: No metadata found in /root/.local/lib/python3.6/site-packages
WARNING: No metadata found in /root/.local/lib/python3.6/site-packages
Name: xrootd
Version: 5.4.2
Summary:
Home-page:
Author:
Author-email:
License:
Location: /root/.local/lib/python3.6/site-packages
Requires:
Required-by: |
|
@matthewfeickert so, sorry if i was not very clear: i was asking for a recipe for local installation as the old one no longer works. in my first post i put a link to the recipe that used to work when being run from the root of repository, and my question was, how can that be replaced. the second problem relates to the ALICE software building, and if you want i can provide you with the recipe, but first we have to establish why there is that "cannot uninstall error" |
|
@matthewfeickert thanks a lot! so, i just successfully tried your recipe: the package was installed: but the problem is that i have this: any idea how this can be fixed? prior, this used to be either the git tag, or the date followed by |
Sure. To get the version too just run the xrootd/.github/workflows/build.yml Lines 570 to 577 in 3d64cda )
xrootd/packaging/wheel/publish.sh Lines 3 to 6 in 3d64cda |
|
By the way, that before you c.f. 2cc617c and #1604 (comment) for more details here. Also unless you're on a system with line in your that will only get the Though this requires Python 3.7 or later. |
|
@matthewfeickert thanks!! yeah, i noticed that and followed, and i can confirm that with on a tagged head i get the valid result. and i see in dist : just i case i messed up the instructions, this is the current script that i use for building: https://github.com/adriansev/bin-scripts/blob/master/xrootdpy_rebuild Thank you!! |
This is just because you're asking python3 -m pip install --user --upgrade ${BNAME}/dist/xrootd-*.tar.gzas you have multiple sdists under It looks like you're trying to do that here, but I think you're in the wrong directory(?). Also xrootd/packaging/wheel/publish.sh Line 37 in 3d64cda so you can remove this line. Probably easier to just do # ...
python3 -m pip --quiet install --upgrade build
command cp -f -r packaging/wheel/* .
rm dist/*
./publish.sh
cd ..
python3 -m pip install --user --upgrade ${BNAME}/dist/xrootd-*.tar.gz
# ... |
|
@matthewfeickert thanks a lot! yes, it seems that the second bdist line was wrong. now, on the matter of normal cmake based xrootd installation https://github.com/alisw/alidist/blob/master/xrootd.sh#L49 and this should not happen, as ideally nor the system installed python(and packages) nor user installed packages should be seen... this happens in an environment that have it's own python. until this is fixed in xrootd, is there anything that we can do to intercept and fix the problem? Maybe something like build without python, and then build/install the python bindings based on what we just compiled, but with our python infrastructure and locations? Thanks a lot! |
|
@matthewfeickert so, i found the problem and the solution was this: |
Correct. As an explicit example with the image built from this Dockerfile:ARG BASE_IMAGE=gitlab-registry.cern.ch/linuxsupport/cc7-base:latest
FROM ${BASE_IMAGE} as base
SHELL [ "/bin/bash", "-c" ]
RUN yum update -y && \
yum install -y \
cmake3 \
make \
krb5-devel \
libuuid-devel \
libxml2-devel \
openssl-devel \
systemd-devel \
zlib-devel \
devtoolset-7-gcc-c++ \
python3-devel \
python3-setuptools \
git \
cppunit-devel && \
yum clean all && \
python3 -m pip install --no-cache-dir --upgrade pip setuptools wheel
COPY . /code/xrootd
WORKDIR /code
RUN source scl_source enable devtoolset-7 && \
cmake3 \
-DCMAKE_INSTALL_PREFIX=/usr/local/ \
-DPYTHON_EXECUTABLE=$(command -v python3) \
-DENABLE_TESTS=ON \
-DPIP_VERBOSE=ON \
-S xrootd \
-B build && \
cmake3 build -LH && \
cmake3 \
--build build \
--clean-first \
--parallel $(($(nproc) - 1)) && \
cmake3 --build build --target install && \
cd / && \
python3 -m pip list
WORKDIR /
SHELL [ "/usr/bin/scl", "enable", "devtoolset-7", "/bin/bash", "-c"]if you build it with then you can see that the bindings got installed under the
I don't think there's anything to fix here. The c.f. pypa/pip#5247 (comment) for a concise technical summary from the PyPA on why errors like this happen. This is strange though, as this sort of thing is what motivated me to open up Issue #1579 in the first place and make the changes that went into I'm a bit confused too why
The short answer is just manually delete the old
...yes this will also indeed work. c.f. #1594 (comment) for an example of building with CMake and then installing the Python bindings from PyPI. |
|
in my case 5.4.1 is distro installed, also usually i have another |
|
also, related to #1594 (comment) is there a solution to use the already present source? can i do |
Ah okay. Thanks for explaining. So let me see if I have this all correct.
Okay, so if I have that all right, then it seems that you're trying to mix installing Python packages from package managers and from Okay, assuming I have that right (my apologies if not) my recommendation for the
Does this seem reasonable? Sorry if I've confused the situation or missed anything.
You could use the current source that you have, but you'd need to do the xrootd/.github/workflows/build.yml Lines 570 to 577 in 3d64cda
Currently no. The pattern that you describe here with being able to install from Git repos is a very reasonable one, but that requires the repository structure to be setup in a way that allows for that. The XRootD system is setup in a way that requires you to do the xrootd/.github/workflows/build.yml Lines 570 to 577 in 3d64cda sort of workflow as the |
|
As a follow up, here's a Dockerfile where I've intentionally recreated the problem you describe, and then show how to get around it if you decided you need to install Python bindings globally with Dockerfile:ARG BASE_IMAGE=gitlab-registry.cern.ch/linuxsupport/cc7-base:latest
FROM ${BASE_IMAGE} as base
SHELL [ "/bin/bash", "-c" ]
RUN yum update -y && \
yum install -y \
cmake3 \
make \
krb5-devel \
libuuid-devel \
libxml2-devel \
openssl-devel \
systemd-devel \
zlib-devel \
devtoolset-7-gcc-c++ \
python3-devel \
python3-setuptools \
git \
cppunit-devel && \
yum clean all && \
python3 -m pip install --no-cache-dir --upgrade pip setuptools wheel
RUN yum update -y && \
yum install -y \
xrootd-5.4.1 \
xrootd-client-5.4.1 \
python36-xrootd-5.4.1 && \
yum clean all && \
xrootd -v && \
xrdcp --version && \
python3 -m pip list && \
python3 -m pip show xrootd # 5.4.1
# The following would fail with the error:
#
# ERROR: Cannot uninstall 'xrootd'. It is a distutils installed project and thus we
# cannot accurately determine which files belong to it which would lead to only a
# partial uninstall.
#
# as you're asking pip to try to deal with something it didn't install.
# RUN . /opt/rh/devtoolset-7/enable && \
# python3 -m pip install --upgrade --force-reinstall 'xrootd==5.4.2' && \
# xrootd -v && \
# xrdcp --version && \
# python3 -m pip list && \
# python3 -m pip show xrootd
# The solution is to uninstall the RPM package and _then_ to install things
# A better solution is to not be installing Python packages globally as pip
# warns us about:
#
# WARNING: Running pip as the 'root' user can result in broken permissions and
# conflicting behaviour with the system package manager. It is recommended to
# use a virtual environment instead: https://pip.pypa.io/warnings/venv
#
# and to instead either use an automatically activated virtual envrionment or to
# use pipx
RUN yum remove python36-xrootd -y && \
. /opt/rh/devtoolset-7/enable && \
python3 -m pip --verbose install \
--upgrade \
--force-reinstall \
'xrootd==5.4.2' && \
xrootd -v && \
xrdcp --version && \
python3 -m pip list && \
python3 -m pip show xrootd && \
python3 -c 'import XRootD; print(XRootD)' && \
python3 -c 'import pyxrootd; print(pyxrootd)' && \
python3 -c 'from XRootD import client; print(client.FileSystem("root://someserver:1094"))'
WORKDIR /
SHELL [ "/usr/bin/scl", "enable", "devtoolset-7", "/bin/bash", "-c"]the commented out snippet RUN . /opt/rh/devtoolset-7/enable && \
python3 -m pip install --upgrade --force-reinstall 'xrootd==5.4.2' && \snippet fails with your error: The part that allows the build to succeed is the bit below where the rpm is first removed to avoid collisions like this that RUN yum remove python36-xrootd -y && \
. /opt/rh/devtoolset-7/enable && \
python3 -m pip --verbose install \
--upgrade \
--force-reinstall \
'xrootd==5.4.2' && \
...Does this all make sense? |
|
So, for the record (as a lot was settled in private communication) a short summary with context, problem and workable solution for us (ALICE) is the following:
this 5.4.1 being the system level installed package.
moreover, maybe part of the context, i should mention that on clusters, or any multi-user machines, that xrootd might be needed system-level and this should not hinder the ALICE users to build the ALICE software; |
|
@adriansev & @matthewfeickert : could you just quickly let me know what is the status of this one? is it covered by #1648? |
|
this was answered and the actual recipe look like this: https://github.com/adriansev/bin-scripts/blob/e6d2b4a857e4f1876864853b9917faca17d5ed21/xrootdpy_rebuild#L13 |
Hi! After the new improvements the recipe for local installation from local repository no longer works. What is the new procedure for local python bindings install? Note that the need is to also test uncommitted source modifications.
what i tried so far is:
The text was updated successfully, but these errors were encountered: