You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe
The dnf.packages and yum.packages operations use the RpmPackageProvides fact internally to map binaries to package names. For example dnf.packages(packages=["vim"]) first runs dnf repoquery --whatprovides vim and then executes dnf install -y vim-enhanced, since vim is provided by vim-enhanced on RHEL derived distros.
This is inconsistent with other package manager operations and also somewhat surprising (at least I would never expect it).
This also slows things down considerably, since the RpmPackageProvides fact is checked for each package sequentially. For example, given the following script and all the listed packages already installed:
$ time pyinfra --dry -v --ssh-user=root test deploy.py
--> Loading config...
--> Loading inventory...
--> Connecting to hosts...
[test] Connected
--> Preparing operations...
--> Preparing Operations...
Loading: deploy.py
[test] Loaded fact rpm.RpmPackages
[test] Loaded fact rpm.RpmPackageProvides (name=sudo)
[test] noop: package sudo is installed (1.9.5p2-1.el8_9)
[test] Loaded fact rpm.RpmPackageProvides (name=tmux)
[test] noop: package tmux is installed (2.7-3.el8)
[test] Loaded fact rpm.RpmPackageProvides (name=htop)
[test] noop: package htop is installed (3.2.1-1.el8)
[test] Loaded fact rpm.RpmPackageProvides (name=rsync)
[test] noop: package rsync is installed (3.1.3-19.el8_7.1)
[test] Loaded fact rpm.RpmPackageProvides (name=net-tools)
[test] noop: package net-tools is installed (2.0-0.52.20160912git.el8)
[test] Loaded fact rpm.RpmPackageProvides (name=less)
[test] noop: package less is installed (530-2.el8_9)
[test] Loaded fact rpm.RpmPackageProvides (name=nnn)
[test] noop: package nnn is installed (4.6-1.el8)
[test] Loaded fact rpm.RpmPackageProvides (name=ncdu)
[test] noop: package ncdu is installed (1.19-1.el8)
[test] Loaded fact rpm.RpmPackageProvides (name=hostname)
[test] noop: package hostname is installed (3.20-6.el8)
[test] Loaded fact rpm.RpmPackageProvides (name=unzip)
[test] noop: package unzip is installed (6.0-46.el8)
[test] Loaded fact rpm.RpmPackageProvides (name=tree)
[test] noop: package tree is installed (1.7.0-15.el8)
[test] Loaded fact rpm.RpmPackageProvides (name=time)
[test] noop: package time is installed (1.9-3.el8)
[test] Loaded fact rpm.RpmPackageProvides (name=lsof)
[test] noop: package lsof is installed (4.93.2-1.el8)
[test] Loaded fact rpm.RpmPackageProvides (name=tar)
[test] noop: package tar is installed (1.30-9.el8)
[test] Loaded fact rpm.RpmPackageProvides (name=patch)
[test] noop: package patch is installed (2.7.6-11.el8)
[test] Loaded fact rpm.RpmPackageProvides (name=smartmontools)
[test] noop: package smartmontools is installed (7.1-2.el8)
[test] Loaded fact rpm.RpmPackageProvides (name=sysbench)
[test] noop: package sysbench is installed (1.0.20-5.el8)
[test] Loaded fact rpm.RpmPackageProvides (name=sysstat)
[test] noop: package sysstat is installed (11.7.3-11.el8)
[test] Loaded fact rpm.RpmPackageProvides (name=msr-tools)
[test] noop: package msr-tools is installed (1.3-17.el8)
[test] Ready: deploy.py
--> Detected changes:
Operation Change Conditional Change
dnf.packages (packages=['sudo', 'tmux', 'htop', 'rsync', 'net-tools', 'less', 'nnn', 'ncdu', 'hostname', 'unzip', 'tree', 'time', 'lsof', 'tar', 'patch', 'smartmontools', 'sysbench', 'sysstat', 'msr-tools'], update=False) - -
--> Disconnecting from hosts...
pyinfra --dry -v --ssh-user=root test deploy.py 0.67s user 0.06s system 4% cpu 16.234 total
That is 16s just to determine that nothings needs to be done.
Without expand_package_fact=RpmPackageProvides it can be much faster:
$ time pyinfra --dry -v test deploy.py
--> Loading config...
--> Loading inventory...
--> Connecting to hosts...
[test] Connected
[pyinfra.connectors.ssh] Command exit status: 0
--> Preparing operations...
--> Preparing Operations...
Loading: deploy.py
[test] Loaded fact rpm.RpmPackages
[test] noop: package sudo is installed (1.9.5p2-1.el8_9)
[test] noop: package tmux is installed (2.7-3.el8)
[test] noop: package htop is installed (3.2.1-1.el8)
[test] noop: package rsync is installed (3.1.3-19.el8_7.1)
[test] noop: package net-tools is installed (2.0-0.52.20160912git.el8)
[test] noop: package less is installed (530-2.el8_9)
[test] noop: package nnn is installed (4.6-1.el8)
[test] noop: package ncdu is installed (1.19-1.el8)
[test] noop: package hostname is installed (3.20-6.el8)
[test] noop: package unzip is installed (6.0-46.el8)
[test] noop: package tree is installed (1.7.0-15.el8)
[test] noop: package time is installed (1.9-3.el8)
[test] noop: package lsof is installed (4.93.2-1.el8)
[test] noop: package tar is installed (1.30-9.el8)
[test] noop: package patch is installed (2.7.6-11.el8)
[test] noop: package smartmontools is installed (7.1-2.el8)
[test] noop: package sysbench is installed (1.0.20-5.el8)
[test] noop: package sysstat is installed (11.7.3-11.el8)
[test] noop: package msr-tools is installed (1.3-17.el8)
[test] Ready: deploy.py
--> Detected changes:
Operation Change Conditional Change
dnf.packages (packages=['sudo', 'tmux', 'htop', 'rsync', 'net-tools', 'less', 'nnn', 'ncdu', 'hostname', 'unzip', 'tree', 'time', 'lsof', 'tar', 'patch', 'smartmontools', 'sysbench', 'sysstat', 'msr-tools'], update=False) - -
--> Disconnecting from hosts...
pyinfra --dry -v test deploy.py 0.49s user 0.04s system 45% cpu 1.164 total
Describe the solution you'd like
Drop expand_package_fact from dnf.packages or at least don't make it the default. It's both slow and probably not what most users would expect. If it needs to stay, an optimization would be to run a single repoquery with all the package names.
If you agree, I can provide a PR.
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem? Please describe
The
dnf.packages
andyum.packages
operations use theRpmPackageProvides
fact internally to map binaries to package names. For examplednf.packages(packages=["vim"])
first runsdnf repoquery --whatprovides vim
and then executesdnf install -y vim-enhanced
, since vim is provided byvim-enhanced
on RHEL derived distros.This is inconsistent with other package manager operations and also somewhat surprising (at least I would never expect it).
This also slows things down considerably, since the
RpmPackageProvides
fact is checked for each package sequentially. For example, given the following script and all the listed packages already installed:Currently:
That is 16s just to determine that nothings needs to be done.
Without
expand_package_fact=RpmPackageProvides
it can be much faster:Describe the solution you'd like
Drop
expand_package_fact
fromdnf.packages
or at least don't make it the default. It's both slow and probably not what most users would expect. If it needs to stay, an optimization would be to run a singlerepoquery
with all the package names.If you agree, I can provide a PR.
The text was updated successfully, but these errors were encountered: