[BUG] onedir blocks installation of shared libraries for pygit2 gitfs in 3006.0 #64121
Comments
|
The Salt master does not not run as root anymore, see the release notes: |
|
Ah, well that was half of it. I moved them to a directory owned by root and readable by the new Now I am getting an error probably because the The |
DaAwesomeP
changed the title
DaAwesomeP
changed the title
|
I stand corrected, there is a home directory assigned to it at the default EDIT: See #64141 |
|
Running I verified that the |
|
OK, I found a version inconsistency that may make dealing this this quite horrible:
So it seems you have to pick Was gitfs on Debian stable tested as a part of the 3006.0 release? I'm sure this is a very common configuration. If not, how can we be sure this sort of thing is tested going forward? I imagine this is also a mess on RHEL 6 and other older stable platforms. |
|
I tried I also tried I will try building the wheel separately and see if I see why Salt moved to Python 3.10 but I'm sure this is not the only package version inconsistency that will be found on other stable/LTS distros. |
|
Another workaround: Since Salstack still support Python 3.9 just fine and Debian 11 packages are built around Python 3.9 as the latest available version, then the Salt packages could use Python 3.9 on Debian 11. |
No, you have to install pygit2 via wheel: The system package of libgit2 does not usually work, as it doesn't have SSH support included. |
|
@OrangeDog To try that out I removed salt, wiped the Python 3.10 and package folders to start afresh, and reinstalled. Pip installs pygit2 but it complains of missing I have verified that package Given that it can't find # uninstall 3006
$ rm -rf /opt/saltstack/salt/lib/python3.10 /opt/saltstack/salt/extras-3.10
# install 3006
$ salt-pip install --only-binary=:all: pygit2
Collecting pygit2
Using cached pygit2-1.12.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.9 MB)
Collecting cffi>=1.9.1
Using cached cffi-1.15.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (441 kB)
Collecting pycparser
Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Installing collected packages: pycparser, cffi, pygit2
Do not adjust rpath of /tmp/pip-target-mvjvbnk5/lib/python/_cffi_backend.cpython-310-x86_64-linux-gnu.so
Do not adjust rpath of /tmp/pip-target-mvjvbnk5/lib/python/pygit2.libs/libcrypto-1b9ded14.so.3
WARNING: Unable to find library libssl-d3387b0f.so.3 linked from /tmp/pip-target-mvjvbnk5/lib/python/pygit2.libs/libgit2-dc37b70b.so.1.6.3
WARNING: Unable to find library libcrypto-1b9ded14.so.3 linked from /tmp/pip-target-mvjvbnk5/lib/python/pygit2.libs/libgit2-dc37b70b.so.1.6.3
WARNING: Unable to find library libpcre-9513aab5.so.1.2.0 linked from /tmp/pip-target-mvjvbnk5/lib/python/pygit2.libs/libgit2-dc37b70b.so.1.6.3
WARNING: Unable to find library libssh2-3ed487a7.so.1.0.1 linked from /tmp/pip-target-mvjvbnk5/lib/python/pygit2.libs/libgit2-dc37b70b.so.1.6.3
Do not adjust rpath of /tmp/pip-target-mvjvbnk5/lib/python/pygit2.libs/libgit2-dc37b70b.so.1.6.3
Do not adjust rpath of /tmp/pip-target-mvjvbnk5/lib/python/pygit2.libs/libpcre-9513aab5.so.1.2.0
WARNING: Unable to find library libssl-d3387b0f.so.3 linked from /tmp/pip-target-mvjvbnk5/lib/python/pygit2.libs/libssh2-3ed487a7.so.1.0.1
WARNING: Unable to find library libcrypto-1b9ded14.so.3 linked from /tmp/pip-target-mvjvbnk5/lib/python/pygit2.libs/libssh2-3ed487a7.so.1.0.1
Do not adjust rpath of /tmp/pip-target-mvjvbnk5/lib/python/pygit2.libs/libssh2-3ed487a7.so.1.0.1
WARNING: Unable to find library libcrypto-1b9ded14.so.3 linked from /tmp/pip-target-mvjvbnk5/lib/python/pygit2.libs/libssl-d3387b0f.so.3
Do not adjust rpath of /tmp/pip-target-mvjvbnk5/lib/python/pygit2.libs/libssl-d3387b0f.so.3
WARNING: In `rpath_only mode` but /tmp/pip-target-mvjvbnk5/lib/python/pygit2/../pygit2.libs/libgit2-dc37b70b.so.1.6.3 is not in /opt/saltstack/salt
WARNING: In `rpath_only mode` but /tmp/pip-target-mvjvbnk5/lib/python/pygit2/../pygit2.libs/libssl-d3387b0f.so.3 is not in /opt/saltstack/salt
WARNING: In `rpath_only mode` but /tmp/pip-target-mvjvbnk5/lib/python/pygit2/../pygit2.libs/libcrypto-1b9ded14.so.3 is not in /opt/saltstack/salt
WARNING: In `rpath_only mode` but /tmp/pip-target-mvjvbnk5/lib/python/pygit2/../pygit2.libs/libpcre-9513aab5.so.1.2.0 is not in /opt/saltstack/salt
WARNING: In `rpath_only mode` but /tmp/pip-target-mvjvbnk5/lib/python/pygit2/../pygit2.libs/libssh2-3ed487a7.so.1.0.1 is not in /opt/saltstack/salt
Do not adjust rpath of /tmp/pip-target-mvjvbnk5/lib/python/pygit2/_libgit2.abi3.so
WARNING: In `rpath_only mode` but /tmp/pip-target-mvjvbnk5/lib/python/pygit2/../pygit2.libs/libgit2-dc37b70b.so.1.6.3 is not in /opt/saltstack/salt
WARNING: In `rpath_only mode` but /tmp/pip-target-mvjvbnk5/lib/python/pygit2/../pygit2.libs/libssl-d3387b0f.so.3 is not in /opt/saltstack/salt
WARNING: In `rpath_only mode` but /tmp/pip-target-mvjvbnk5/lib/python/pygit2/../pygit2.libs/libcrypto-1b9ded14.so.3 is not in /opt/saltstack/salt
WARNING: In `rpath_only mode` but /tmp/pip-target-mvjvbnk5/lib/python/pygit2/../pygit2.libs/libpcre-9513aab5.so.1.2.0 is not in /opt/saltstack/salt
WARNING: In `rpath_only mode` but /tmp/pip-target-mvjvbnk5/lib/python/pygit2/../pygit2.libs/libssh2-3ed487a7.so.1.0.1 is not in /opt/saltstack/salt
Do not adjust rpath of /tmp/pip-target-mvjvbnk5/lib/python/pygit2/_pygit2.cpython-310-x86_64-linux-gnu.so
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
salt 3006.0 requires cffi==1.14.6, but you have cffi 1.15.1 which is incompatible.
Successfully installed cffi-1.15.1 pycparser-2.21 pygit2-1.12.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
$ systemctl restart salt-master.serviceResulting log: |
|
It's not looking for |
DaAwesomeP
changed the title
|
@OrangeDog I see, that makes sense. Has this been observed with any other bundled shared libraries? Is there a workaround? |
|
@DaAwesomeP @OrangeDog @dwoz observing the same issue, looking for a workaround. UPDATE: it seems to work for me with $ salt-pip install --only-binary=:all: --no-deps pygit2==1.7.0
[..]
$ salt-pip show pygit2
Name: pygit2
Version: 1.7.0
Summary: Python bindings for libgit2.
Home-page: http://github.com/libgit2/pygit2
Author:
Author-email:
License: GPLv2 with linking exception
Location: /opt/saltstack/salt/extras-3.10
Requires: cffi
Required-by: Latest version (1.12.0) throws exactly the same error as @DaAwesomeP described in #64121 (comment). |
|
What happens if you update salt's pip to the latest version first? salt-pip install --upgrade pip |
Just reinstalled everything on master from scratch and did this: Then installed latest pygit2: Started salt-master service and got this: Stopped salt-master, installed pygit2==1.7.0: Started salt-master again, got remote repositories cloned in /var/cache/salt/master/gitfs, no errors in log. (This is on Debian 11 "bullseye" amd64.) |
|
I can confirm that |
|
For what it's worth. We are expecting that people will want to install from source and use the system's libraries. For pygit2 I did this: As far as |
|
It sounds like additional bundled shared libraries in wheels may have some issue with onedir, but there must be something particular since pygit2 v1.7.0 works fine but pygit2 v1.12.0 does not. I'm sure this issue will come up again at some point with this package or another. Locking into an older version (either v1.6.1 or v1.7.0 via wheel or source) seems more like a workaround.
I have had a lot of trouble with Python packages built from source across different systems, particularly with Salstack. Using the wheels/bundled libraries with pygit2 solves a lot of portability issues across many systems where the libgit2 could be really any version and the Python version included in onedir will increment as time goes on. More and more Python packages with bindings are bundling their dependencies so that they are consistent across platforms and independent from OS source versions. Having to install dependencies and compile the modules on each machine seems counter to onedir making salt more portable/consistent across installations. Building from source on Debian 11 uses a now quite old libgit2 that does not support newer SSH keys. The workaround for this is to use the pygit2 wheel (which became always the case with onedir prior to 3006). See #61790. Also, pygit2 v1.6.1 does not officially support Python 3.10: https://www.pygit2.org/install.html#version-numbers (so again the ability to install the latest version via wheel is much more future-proofed).
I have observed cffi version mismatches when installing additional packages alongside Salt for a while now. So far no issues (I have tried both installing the later version and keeping the older version), but it would be great if Salt updated that dependency. There are other additional packages too that rely on different cffi versions. |
In general, no. That's a big burden that many people really want to avoid (to the extent that they won't upgrade to onedir because of it). Especially for Your build probably won't work. See #58898. |
|
@DaAwesomeP Can you please test using |
|
@dwoz I'm not able to test this out on the same Debian machine at the moment (maybe next week or week after), but I did try it out on a RHEL8 machine. No luck unfortunately. |
OrangeDog
added
Regression
@whytewolf version locking or manually building a wheel may work now, but it is only a matter of time before this breaks and we need a new version or there is risk of a CVE preventing this version from being used. Debian 12 is also right around the corner which would affect how that wheel builds outside of the relenv. While it may not be a "hack" it is definitely a temporary workaround. Using these modules has become a constant stream of workarounds, and surely this was not tested prior to 3006 release or these workarounds wouldn't be necessary or would be well documented. |
|
Looks like this is an issue with how the Salt Master process is being run. When running on Ubuntu 20.04 and some of the other OSes systemd is launching the Salt master without a HOME environment variable defined. This results in the That explains why we see it in A temporary work around could be to modify the Salt Master's service file to include a user. Salt has a method that set's the HOME environment variable, |
|
From https://git-scm.com/book/en/v2/Git-Internals-Environment-Variables
If HOME is required then |
|
We can modify check_user to set HOME even if the key is not in https://github.com/saltstack/salt/blob/master/salt/cli/daemons.py#L182 |
|
|
I don't think Arch is using the onedir solution nor the salt user account yet. It seems to be running as root and has pieces of itself all over the filesystem on the master. |
|
Spent a few days chasing this in our project too and what I found was that |
|
Would it possible to fix it in 3006.4 ? I guess, a lot of people using git with salt. |
|
After upgrading on the good ol' CentOS 7 from 3005 to 3006.3, I also ran into this issue.
Here and there, it is mentioned that Salt-Master now runs as non-root, but this was not the case here; it still runs as root just as before. But the HOME environment variable was indeed missing. Adding this to my |
|
Thanks to @MartinEmrich i've got it working on AlmaLinux 8.8 and salt-master 3006.3. In my case the steps to take where:
|
|
To sum up, this seems to have been two separate issues.
The latter problem appears to be dependent on the version of pygit2, and affects all versions of Salt. |
Yesterday I did a fresh install of 3006.4 on Debian 12 (using packages from Debian 11) and was able to get it to install with:
Yes, I am currently solving this with a Systemd unit modifier in [Service]
WorkingDirectory=~Note that I am using If the install-specific parts in this issue have been resolved then we should also turn our attention to:
|
I definitely had library load issues when installing wheels the last time I tested onedir.
Similarly, this one fixes 3004.2, and probably any classic packaging running as [Service]
Environment=HOME=/root |
|
Seems I managed to make my 3006.5 (onedir package from repo.saltstack.io on Debian11) to work by:
Maybe it's just me, but official docs seems a bit lagging about the onedir package, recommending to install via OS tools (one for all: pip instead of salt-pip)... |
|
Had to do this again today with a more recent update modified
did the trick. |
I was able to get it going even without pinning to |
This saves me now a lot of pain! i am using pygit2 from salt-pip: My code: I am very sad that saltstack do not have a "one shoot" solution for debian users and we must do such a lot of "workarounds" while using salt-master with a git repo and ssh. :( |
|
for me in centos 7 a workaround that worked was downgrading pygit2 to 1.10.1
and restarting the salt-master of course |
|
EDIT: apologies, my error was not the same as what's listed in this thread. |
|
@jamest-pin you need to install readelf and also maybe patchelf into the operating system. these should be installed through your normal package manager. |
|
Thanks @whytewolf that's what I ended up doing, although |
|
|
|
@mazhenyong1 see this comment: #64121 (comment). All you need is the Systemd modifier and ensure that the |
To piggyback this, Salt 3006.8 on Rocky Linux 9 does not have a home directory created for the Salt user. I installed with the bootstrap method. The SystemD environment solution mentioned above resolved it for me as well. |
|
My workaround with It uses an ancient version of libgit2, which does not (yet) support non-RSA host keys: libgit2/pygit2#552 (comment) After trying literally all versions after 1.9.0 of pygit2, I am stuck now. All up to and including 1.10.1 will refuse to connect to our new Git server without RSA host key. The next ist 1.11.1, which will bail out with the missing known_hosts file :( Using the latest 1.15.x will produce lots of errors in the Salt Master log. |
and others
Description
I did not change any configuration between 3005.1 and 3006.0 but suddently salt-master cannot find my public keys.
Setup
Steps to Reproduce the behavior
Upgrade from 3005.1 to 3006.0. Run
salt-pip install pygit2.Expected behavior
Should fetch gitfs normally.
Screenshots
Versions Report
salt --versions-report
(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)Additional context
Running
catandlson those files show they exist fine. The files have permission0400/-r--------and are owned byroot:root.The text was updated successfully, but these errors were encountered: