Add check for nmstate C library on pip install

[jona42-ui]

Added check for the presence of the nmstate C library in the setup.py script.

This commit modifies the setup.py script to check for the presence of the nmstate C library using ctypes.cdll.LoadLibrary("libnmstate.so.2"). If the library is not found, a RuntimeError is raised, guiding the user to install the nmstate C library separately before installing the Python package. This enhancement ensures that users are informed about the dependency on the nmstate C library before proceeding with the installation.

Resolves: #2575

[kubevirt-bot]

Hi @jona42-ui. Thanks for your PR.

I'm waiting for a nmstate member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[jona42-ui]

does this seem to address the issue??

cc @ihuguet

[ihuguet]

I don't know how is this normally done. I would bet that setuptools has a way to do this, but as I say, I don't know. Let's see what others think.

[jona42-ui]

I don't know how is this normally done. I would bet that setuptools has a way to do this, but as I say, I don't know. Let's see what others think.

thanks @ihuguet for the feedback.

@cathay4t @ffmancera are you having sometime to look at this

[cathay4t]

Please provide proof that setup.py can actually install C library of nmstate. From what I see, it will never works

[jona42-ui]

oops seems am getting the workflow here wrongly,
so I anticipated that when the nmstate is installed it will eventually install the c lib, which I see Is not the case.
so my proposal would be now to explicitly handle the C library installation, meaning depending on the host OS(maybe debian&fedora) only.

[jona42-ui]

@cathay4t you can correct me if am still wrong

[jona42-ui]

what would be the best appraoch here.
either provide user with explicit installation steps of the nmstae c lib or install the nmstate c lib(seemingly requires more work, but I am willing to take it on if guided)
@cathay4t @ffmancera

[cathay4t]

Pip should not compile C library, so I think the installation should fail with error message guilding user to install nmstate C library. I recall python gobject binding required gobject C binding been installed, so I guess you may check how their setup.py looks like.

[jona42-ui]

Pip should not compile C library, so I think the installation should fail with error message guilding user to install nmstate C library. I recall python gobject binding required gobject C binding been installed, so I guess you may check how their setup.py looks like.

thanks on it

[jona42-ui]

from the reference you gave me https://github.com/pygobject/pgi/blob/master/setup.py (I hope its the one)
It turns out from my observation that, the setup.py script typically doesn't handle the installation of the underlying C libraries directly. Instead, it usually focuses on setting up the Python bindings and ensuring that the necessary Python dependencies are met.

so yes I think your suggestion would work best to allow the user install the nmstate c lib separately

[cathay4t]

Yes. I was shocked when you have pip install inside of setup.py, that's a loop as pip install actually invoke setup.py.

Enjoy hacking.

[cathay4t]

BTW, to check whether nmstate C library is installed, you may call ctypes.cdll.LoadLibrary("libnmstate.so.2")

[jona42-ui]

@jona42-ui , do you have this file ? /usr/local/lib64/libnmstate.so.2

yes I do

[liangwen12year]

@jona42-ui , do you have this file ? /usr/local/lib64/libnmstate.so.2

yes I do

Can you do echo $LD_LIBRARY_PATH ?

Notice that ctypes.cdll.LoadLibrary() relies on the LD_LIBRARY_PATH to find the shared library path, otherwise, you have to append the /usr/local/lib64 to the shared library path via:

export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/lib64"

[liangwen12year]

@jona42-ui , do you have this file ? /usr/local/lib64/libnmstate.so.2

yes I do

Please use this command to install PREFIX=/usr make install rather than make install, then ctypes.cdll.LoadLibrary("libnmstate.so.2") will load successfully. (As described in https://nmstate.io/user/install.html#install-from-source. )

[jona42-ui]

@jona42-ui , do you have this file ? /usr/local/lib64/libnmstate.so.2

yes I do

Please use this command to install PREFIX=/usr make install rather than make install, then ctypes.cdll.LoadLibrary("libnmstate.so.2") will load successfully. (As described in https://nmstate.io/user/install.html#install-from-source. )

okay

[jona42-ui]

@jona42-ui , do you have this file ? /usr/local/lib64/libnmstate.so.2

yes I do

Please use this command to install PREFIX=/usr make install rather than make install, then ctypes.cdll.LoadLibrary("libnmstate.so.2") will load successfully. (As described in https://nmstate.io/user/install.html#install-from-source. )

its like I have no lib4 directory, I have lib with the libnmstate.so.2 file

[liangwen12year]

@jona42-ui , do you have this file ? /usr/local/lib64/libnmstate.so.2

yes I do

Please use this command to install PREFIX=/usr make install rather than make install, then ctypes.cdll.LoadLibrary("libnmstate.so.2") will load successfully. (As described in https://nmstate.io/user/install.html#install-from-source. )

its like I have no lib4 directory, I have lib with the libnmstate.so.2 file

After running PREFIX=/usr make install, then you can check by using:

>>> import ctypes
>>> ctypes.cdll.LoadLibrary("libnmstate.so.2")
<CDLL 'libnmstate.so.2', handle 55c9e667c390 at 0x7f33c93c8440>

Please let me know if you encountered the same error OSError: libnmstate.so.2: cannot open shared object file: No such file or directory this time ?

[jona42-ui]

ctypes.cdll.LoadLibrary("libnmstate.so.2")

in the shell after importing ctypes,
I run ctypes.cdll.LoadLibrary("libnmstate.so.2")
it out puts <CDLL 'libnmstate.so.2', handle 55c9e667c390 at 0x7f33c93c8440>
without error

is this what we expected?

[jona42-ui]

I also tried ;
`import ctypes

try:
libnmstate = ctypes.cdll.LoadLibrary("libnmstate.so.2")
print("Library loaded successfully!")
print(libnmstate)
except OSError as e:
print("Error:", e)
`

it says
Library loaded successfully! and out puts
<CDLL 'libnmstate.so.2', handle 19a0f20 at 0x7a3cab7bdfd0>

[liangwen12year]

ctypes.cdll.LoadLibrary("libnmstate.so.2")

in the shell after importing ctypes, I run ctypes.cdll.LoadLibrary("libnmstate.so.2") it out puts <CDLL 'libnmstate.so.2', handle 55c9e667c390 at 0x7f33c93c8440> without error

is this what we expected?

Then you did not encounter the error anymore.

[jona42-ui]

ctypes.cdll.LoadLibrary("libnmstate.so.2")

in the shell after importing ctypes, I run ctypes.cdll.LoadLibrary("libnmstate.so.2") it out puts <CDLL 'libnmstate.so.2', handle 55c9e667c390 at 0x7f33c93c8440> without error
is this what we expected?

Then you did not encounter the error anymore.

I cant tell why the CI really fails

[liangwen12year]

ctypes.cdll.LoadLibrary("libnmstate.so.2")

in the shell after importing ctypes, I run ctypes.cdll.LoadLibrary("libnmstate.so.2") it out puts <CDLL 'libnmstate.so.2', handle 55c9e667c390 at 0x7f33c93c8440> without error
is this what we expected?

Then you did not encounter the error anymore.

I cant tell why the CI really fails

You can try to run the command .github/workflows/build_rpm.sh el9 locally and check libnmstate.so.2 at the beginning of the function def check_nmstate_c_library() by printing out the result of command find / -name "libnmstate.so.2".

[jona42-ui]

ctypes.cdll.LoadLibrary("libnmstate.so.2")

in the shell after importing ctypes, I run ctypes.cdll.LoadLibrary("libnmstate.so.2") it out puts <CDLL 'libnmstate.so.2', handle 55c9e667c390 at 0x7f33c93c8440> without error
is this what we expected?

Then you did not encounter the error anymore.

I cant tell why the CI really fails

You can try to run the command .github/workflows/build_rpm.sh el9 locally and check libnmstate.so.2 at the beginning of the function def check_nmstate_c_library() by printing out the result of command find / -name "libnmstate.so.2".

the first command terminates with error;
Traceback (most recent call last): File "/root/rpmbuild/BUILD/nmstate-2.2.31/rust/src/python/setup.py", line 20, in check_nmstate_c_library ctypes.cdll.LoadLibrary("libnmstate.so.2") File "/usr/lib64/python3.9/ctypes/__init__.py", line 452, in LoadLibrary return self._dlltype(name) File "/usr/lib64/python3.9/ctypes/__init__.py", line 374, in __init__ self._handle = _dlopen(self._name, mode) OSError: libnmstate.so.2: cannot open shared object file: No such file or directory The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/root/rpmbuild/BUILD/nmstate-2.2.31/rust/src/python/setup.py", line 30, in <module> check_nmstate_c_library() File "/root/rpmbuild/BUILD/nmstate-2.2.31/rust/src/python/setup.py", line 22, in check_nmstate_c_library raise RuntimeError( RuntimeError: Error: nmstate C library not found. Please install nmstate C library separately before installing the Python package.See: https://nmstate.io/user/install.html error: Bad exit status from /var/tmp/rpm-tmp.kARUHq (%build) RPM build errors: Bad exit status from /var/tmp/rpm-tmp.kARUHq (%build) make: *** [Makefile:177: rpm] Error 1 (base) thembo@workspace:~/fedora/nmstate/.github/workflows$

[jona42-ui]

ctypes.cdll.LoadLibrary("libnmstate.so.2")

in the shell after importing ctypes, I run ctypes.cdll.LoadLibrary("libnmstate.so.2") it out puts <CDLL 'libnmstate.so.2', handle 55c9e667c390 at 0x7f33c93c8440> without error
is this what we expected?

Then you did not encounter the error anymore.

I cant tell why the CI really fails

You can try to run the command .github/workflows/build_rpm.sh el9 locally and check libnmstate.so.2 at the beginning of the function def check_nmstate_c_library() by printing out the result of command find / -name "libnmstate.so.2".

the first command terminates with error;
Traceback (most recent call last): File "/root/rpmbuild/BUILD/nmstate-2.2.31/rust/src/python/setup.py", line 20, in check_nmstate_c_library ctypes.cdll.LoadLibrary("libnmstate.so.2") File "/usr/lib64/python3.9/ctypes/__init__.py", line 452, in LoadLibrary return self._dlltype(name) File "/usr/lib64/python3.9/ctypes/__init__.py", line 374, in __init__ self._handle = _dlopen(self._name, mode) OSError: libnmstate.so.2: cannot open shared object file: No such file or directory The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/root/rpmbuild/BUILD/nmstate-2.2.31/rust/src/python/setup.py", line 30, in <module> check_nmstate_c_library() File "/root/rpmbuild/BUILD/nmstate-2.2.31/rust/src/python/setup.py", line 22, in check_nmstate_c_library raise RuntimeError( RuntimeError: Error: nmstate C library not found. Please install nmstate C library separately before installing the Python package.See: https://nmstate.io/user/install.html error: Bad exit status from /var/tmp/rpm-tmp.kARUHq (%build) RPM build errors: Bad exit status from /var/tmp/rpm-tmp.kARUHq (%build) make: *** [Makefile:177: rpm] Error 1 (base) thembo@workspace:~/fedora/nmstate/.github/workflows$

@liangwen12year can we jump on call and get this off the radar

[liangwen12year]

ctypes.cdll.LoadLibrary("libnmstate.so.2")

in the shell after importing ctypes, I run ctypes.cdll.LoadLibrary("libnmstate.so.2") it out puts <CDLL 'libnmstate.so.2', handle 55c9e667c390 at 0x7f33c93c8440> without error
is this what we expected?

Then you did not encounter the error anymore.

I cant tell why the CI really fails

You can try to run the command .github/workflows/build_rpm.sh el9 locally and check libnmstate.so.2 at the beginning of the function def check_nmstate_c_library() by printing out the result of command find / -name "libnmstate.so.2".

the first command terminates with error;
Traceback (most recent call last): File "/root/rpmbuild/BUILD/nmstate-2.2.31/rust/src/python/setup.py", line 20, in check_nmstate_c_library ctypes.cdll.LoadLibrary("libnmstate.so.2") File "/usr/lib64/python3.9/ctypes/__init__.py", line 452, in LoadLibrary return self._dlltype(name) File "/usr/lib64/python3.9/ctypes/__init__.py", line 374, in __init__ self._handle = _dlopen(self._name, mode) OSError: libnmstate.so.2: cannot open shared object file: No such file or directory The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/root/rpmbuild/BUILD/nmstate-2.2.31/rust/src/python/setup.py", line 30, in <module> check_nmstate_c_library() File "/root/rpmbuild/BUILD/nmstate-2.2.31/rust/src/python/setup.py", line 22, in check_nmstate_c_library raise RuntimeError( RuntimeError: Error: nmstate C library not found. Please install nmstate C library separately before installing the Python package.See: https://nmstate.io/user/install.html error: Bad exit status from /var/tmp/rpm-tmp.kARUHq (%build) RPM build errors: Bad exit status from /var/tmp/rpm-tmp.kARUHq (%build) make: *** [Makefile:177: rpm] Error 1 (base) thembo@workspace:~/fedora/nmstate/.github/workflows$

@liangwen12year can we jump on call and get this off the radar

Sure, give me your availability. I will schedule the call/meeting.

[jona42-ui]

ctypes.cdll.LoadLibrary("libnmstate.so.2")

in the shell after importing ctypes, I run ctypes.cdll.LoadLibrary("libnmstate.so.2") it out puts <CDLL 'libnmstate.so.2', handle 55c9e667c390 at 0x7f33c93c8440> without error
is this what we expected?

Then you did not encounter the error anymore.

I cant tell why the CI really fails

You can try to run the command .github/workflows/build_rpm.sh el9 locally and check libnmstate.so.2 at the beginning of the function def check_nmstate_c_library() by printing out the result of command find / -name "libnmstate.so.2".

the first command terminates with error;
Traceback (most recent call last): File "/root/rpmbuild/BUILD/nmstate-2.2.31/rust/src/python/setup.py", line 20, in check_nmstate_c_library ctypes.cdll.LoadLibrary("libnmstate.so.2") File "/usr/lib64/python3.9/ctypes/__init__.py", line 452, in LoadLibrary return self._dlltype(name) File "/usr/lib64/python3.9/ctypes/__init__.py", line 374, in __init__ self._handle = _dlopen(self._name, mode) OSError: libnmstate.so.2: cannot open shared object file: No such file or directory The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/root/rpmbuild/BUILD/nmstate-2.2.31/rust/src/python/setup.py", line 30, in <module> check_nmstate_c_library() File "/root/rpmbuild/BUILD/nmstate-2.2.31/rust/src/python/setup.py", line 22, in check_nmstate_c_library raise RuntimeError( RuntimeError: Error: nmstate C library not found. Please install nmstate C library separately before installing the Python package.See: https://nmstate.io/user/install.html error: Bad exit status from /var/tmp/rpm-tmp.kARUHq (%build) RPM build errors: Bad exit status from /var/tmp/rpm-tmp.kARUHq (%build) make: *** [Makefile:177: rpm] Error 1 (base) thembo@workspace:~/fedora/nmstate/.github/workflows$

@liangwen12year can we jump on call and get this off the radar

Sure, give me your availability. I will schedule the call/meeting.

am available now.
and usually from 9:00 am to 9:00pm EAT.
next two days is a public hoiliday here.
so now is okay

[liangwen12year]

ctypes.cdll.LoadLibrary("libnmstate.so.2")

in the shell after importing ctypes, I run ctypes.cdll.LoadLibrary("libnmstate.so.2") it out puts <CDLL 'libnmstate.so.2', handle 55c9e667c390 at 0x7f33c93c8440> without error
is this what we expected?

Then you did not encounter the error anymore.

I cant tell why the CI really fails

You can try to run the command .github/workflows/build_rpm.sh el9 locally and check libnmstate.so.2 at the beginning of the function def check_nmstate_c_library() by printing out the result of command find / -name "libnmstate.so.2".

the first command terminates with error;
Traceback (most recent call last): File "/root/rpmbuild/BUILD/nmstate-2.2.31/rust/src/python/setup.py", line 20, in check_nmstate_c_library ctypes.cdll.LoadLibrary("libnmstate.so.2") File "/usr/lib64/python3.9/ctypes/__init__.py", line 452, in LoadLibrary return self._dlltype(name) File "/usr/lib64/python3.9/ctypes/__init__.py", line 374, in __init__ self._handle = _dlopen(self._name, mode) OSError: libnmstate.so.2: cannot open shared object file: No such file or directory The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/root/rpmbuild/BUILD/nmstate-2.2.31/rust/src/python/setup.py", line 30, in <module> check_nmstate_c_library() File "/root/rpmbuild/BUILD/nmstate-2.2.31/rust/src/python/setup.py", line 22, in check_nmstate_c_library raise RuntimeError( RuntimeError: Error: nmstate C library not found. Please install nmstate C library separately before installing the Python package.See: https://nmstate.io/user/install.html error: Bad exit status from /var/tmp/rpm-tmp.kARUHq (%build) RPM build errors: Bad exit status from /var/tmp/rpm-tmp.kARUHq (%build) make: *** [Makefile:177: rpm] Error 1 (base) thembo@workspace:~/fedora/nmstate/.github/workflows$

@liangwen12year can we jump on call and get this off the radar

Sure, give me your availability. I will schedule the call/meeting.

am available now. and usually from 9:00 am to 9:00pm EAT. next two days is a public hoiliday here. so now is okay

Let us follow up in Element channel.

[jona42-ui]

ctypes.cdll.LoadLibrary("libnmstate.so.2")

in the shell after importing ctypes, I run ctypes.cdll.LoadLibrary("libnmstate.so.2") it out puts <CDLL 'libnmstate.so.2', handle 55c9e667c390 at 0x7f33c93c8440> without error
is this what we expected?

Then you did not encounter the error anymore.

I cant tell why the CI really fails

You can try to run the command .github/workflows/build_rpm.sh el9 locally and check libnmstate.so.2 at the beginning of the function def check_nmstate_c_library() by printing out the result of command find / -name "libnmstate.so.2".

the first command terminates with error;
Traceback (most recent call last): File "/root/rpmbuild/BUILD/nmstate-2.2.31/rust/src/python/setup.py", line 20, in check_nmstate_c_library ctypes.cdll.LoadLibrary("libnmstate.so.2") File "/usr/lib64/python3.9/ctypes/__init__.py", line 452, in LoadLibrary return self._dlltype(name) File "/usr/lib64/python3.9/ctypes/__init__.py", line 374, in __init__ self._handle = _dlopen(self._name, mode) OSError: libnmstate.so.2: cannot open shared object file: No such file or directory The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/root/rpmbuild/BUILD/nmstate-2.2.31/rust/src/python/setup.py", line 30, in <module> check_nmstate_c_library() File "/root/rpmbuild/BUILD/nmstate-2.2.31/rust/src/python/setup.py", line 22, in check_nmstate_c_library raise RuntimeError( RuntimeError: Error: nmstate C library not found. Please install nmstate C library separately before installing the Python package.See: https://nmstate.io/user/install.html error: Bad exit status from /var/tmp/rpm-tmp.kARUHq (%build) RPM build errors: Bad exit status from /var/tmp/rpm-tmp.kARUHq (%build) make: *** [Makefile:177: rpm] Error 1 (base) thembo@workspace:~/fedora/nmstate/.github/workflows$

@liangwen12year can we jump on call and get this off the radar

Sure, give me your availability. I will schedule the call/meeting.

am available now. and usually from 9:00 am to 9:00pm EAT. next two days is a public hoiliday here. so now is okay

Let us follow up in Element channel.

okay.
thanks

[liangwen12year]

@jona42-ui , I ran the CI locally, indeed, I can not find libnmstate.so.2 using find / -name libnmstate.so.2 command.

[jona42-ui]

@jona42-ui , I ran the CI locally, indeed, I can not find libnmstate.so.2 using find / -name libnmstate.so.2 command.

oops!
any kmown work around?

[liangwen12year]

@jona42-ui , I ran the CI locally, indeed, I can not find libnmstate.so.2 using find / -name libnmstate.so.2 command.

oops! any kmown work around?

rpm_build (el9) CI failure seems to be expected, if you look at the script .github/workflows/build_rpm.sh, make rpm was called in the end, however, make rpm will not install the file libnmstate.so.2. (make install will install the file libnmstate.so.2)

[jona42-ui]

@jona42-ui , I ran the CI locally, indeed, I can not find libnmstate.so.2 using find / -name libnmstate.so.2 command.

oops! any kmown work around?

rpm_build (el9) CI failure seems to be expected, if you look at the script .github/workflows/build_rpm.sh, make rpm was called in the end, however, make rpm will not install the file libnmstate.so.2. (make install will install the file libnmstate.so.2)

thanks. makes sense. so we need a tweak to the workflow script??

[cathay4t]

we need to tell whether we are in make install or pip install. For make install or other method of install, we skip the c library check. For pip install, we validate.

So you need to find a way to tell whether the command invoker is using pip or not.

Meanwhile, the setup.py install is deprecated, any changes to it should move to new python suggested way: PEP-518 https://peps.python.org/pep-0518/ and PEP 517 https://peps.python.org/pep-0517/

I do not have urgent need on changes to any python code, so take your time and read PEP 517 and 518.

[liangwen12year]

thanks. makes sense. so we need a tweak to the workflow script??

You can take a look Gris's comment, for make install or make rpm, we can skip the c library check by defining an environment variable.