dpkg_lowpkg: Do not access dpkg internals directly #58735

bdrung · 2020-10-15T17:22:28Z

salt contains modules, which directly access the dpkg internal database, instead of using one of the public interfaces provided by dpkg. This is a problem for several reasons, because even though the layout and format of the dpkg database is administrator friendly, and it is expected that those might need to mess with it, in case of emergency, this “interface” does not extend to other programs besides the dpkg suite of tools. The admindir can also be configured differently at dpkg build or run-time. And finally, the contents and its format, will be changing in the near future.

So use something like:

dpkg-query --showformat '${db-fsys:Last-Modified}\n' --show $pkg

to get the mtime from /var/lib/dpkg/info/.list files.

/var/lib/dpkg/available should not be read directly. Instead dpkg-query --print-avail should be used. The /var/lib/dpkg/available
file is only kept up-to-date when using dselect, but nowadays apt is used, and therefore this file does not provide much more information than dpkg-query -W. Users of APT-based frontends should use apt show or in our case the pkg.show salt module.

This merge request addresses the issues in dpkg_lowpkg. A separate merge request for alternatives will follow. Please have a look at the individual commits for more details of the changes.

Bug: #52605 and drive by fix for #58735
Bug-Debian: https://bugs.debian.org/944970

cmcmarrow · 2020-10-21T17:42:05Z

@bdrung thanks for the fix!

salt contains modules, which directly access the dpkg internal database, instead of using one of the public interfaces provided by dpkg. This is a problem for several reasons, because even though the layout and format of the dpkg database is administrator friendly, and it is expected that those might need to mess with it, in case of emergency, this “interface” does not extend to other programs besides the dpkg suite of tools. The admindir can also be configured differently at dpkg build or run-time. And finally, the contents and its format, will be changing in the near future. So use something like: ``` dpkg-query --showformat '${db-fsys:Last-Modified}\n' --show $pkg ``` to get the mtime from /var/lib/dpkg/info/<package>.list files. Bug: saltstack#52605 Bug-Debian: https://bugs.debian.org/944970 Forwarded: saltstack#58735 Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>

salt contains modules, which directly access the dpkg internal database, instead of using one of the public interfaces provided by dpkg. This is a problem for several reasons, because even though the layout and format of the dpkg database is administrator friendly, and it is expected that those might need to mess with it, in case of emergency, this “interface” does not extend to other programs besides the dpkg suite of tools. The admindir can also be configured differently at dpkg build or run-time. And finally, the contents and its format, will be changing in the near future. `/var/lib/dpkg/available` should not be read directly. Instead `dpkg-query --print-avail` should be used. The `/var/lib/dpkg/available` file is only kept up-to-date when using dselect, but nowadays `apt` is used, and therefore this file does not provide much more information than `dpkg-query -W`. Users of APT-based frontends should use `apt show` or in our case the `pkg.show` salt module. `dpkg-query --print-avail` provides following keys (on Ubuntu 20.04): ``` $ dpkg-query --print-avail | sed 's/:.*$//' | sort | uniq Architecture Breaks Bugs Build-Essential Built-Using Cnf-Extra-Commands Cnf-Priority-Bonus Conflicts Depends Description Description-md5 Enhances Essential Filename Homepage Important Installed-Size Maintainer MD5sum Multi-Arch Origin Original-Maintainer Package Pre-Depends Priority Provides Recommends Replaces Section SHA1 SHA256 Size Source Suggests Supported Task Version ``` Following keys cannot be retrieved by `dpkg-query -W`: ``` Build-Essential Built-Using Cnf-Extra-Commands Cnf-Priority-Bonus Important Multi-Arch Original-Maintainer SHA1 SHA256 Supported ``` Please use `pkg.show` for those keys. Following additional keys could be retrieved if needed: ``` Breaks Enhances Essential Filename Pre-Depends Priority Suggests ``` Without calling `_get_pkg_ds_avail`, `info` just calls `_get_pkg_info` and converts it into a dict and add a license field. Since `_get_pkg_ds_avail` is used nowhere else, move it into `info`. Do not query Installed-Size, because it was stripped as "technical" key previously. Bug: saltstack#52605 Bug-Debian: https://bugs.debian.org/944970 Forwarded: saltstack#58735 Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>

salt contains modules, which directly access the dpkg internal database, instead of using one of the public interfaces provided by dpkg. This is a problem for several reasons, because even though the layout and format of the dpkg database is administrator friendly, and it is expected that those might need to mess with it, in case of emergency, this “interface” does not extend to other programs besides the dpkg suite of tools. The admindir can also be configured differently at dpkg build or run-time. And finally, the contents and its format, will be changing in the near future. So use something like: ``` dpkg-query --showformat '${db-fsys:Last-Modified}\n' --show $pkg ``` to get the mtime from /var/lib/dpkg/info/<package>.list files. Bug: saltstack#52605 Bug-Debian: https://bugs.debian.org/944970 Forwarded: saltstack#58735 Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>

salt contains modules, which directly access the dpkg internal database, instead of using one of the public interfaces provided by dpkg. This is a problem for several reasons, because even though the layout and format of the dpkg database is administrator friendly, and it is expected that those might need to mess with it, in case of emergency, this “interface” does not extend to other programs besides the dpkg suite of tools. The admindir can also be configured differently at dpkg build or run-time. And finally, the contents and its format, will be changing in the near future. `/var/lib/dpkg/available` should not be read directly. Instead `dpkg-query --print-avail` should be used. The `/var/lib/dpkg/available` file is only kept up-to-date when using dselect, but nowadays `apt` is used, and therefore this file does not provide much more information than `dpkg-query -W`. Users of APT-based frontends should use `apt show` or in our case the `pkg.show` salt module. `dpkg-query --print-avail` provides following keys (on Ubuntu 20.04): ``` $ dpkg-query --print-avail | sed 's/:.*$//' | sort | uniq Architecture Breaks Bugs Build-Essential Built-Using Cnf-Extra-Commands Cnf-Priority-Bonus Conflicts Depends Description Description-md5 Enhances Essential Filename Homepage Important Installed-Size Maintainer MD5sum Multi-Arch Origin Original-Maintainer Package Pre-Depends Priority Provides Recommends Replaces Section SHA1 SHA256 Size Source Suggests Supported Task Version ``` Following keys cannot be retrieved by `dpkg-query -W`: ``` Build-Essential Built-Using Cnf-Extra-Commands Cnf-Priority-Bonus Important Multi-Arch Original-Maintainer SHA1 SHA256 Supported ``` Please use `pkg.show` for those keys. Following additional keys could be retrieved if needed: ``` Breaks Enhances Essential Filename Pre-Depends Priority Suggests ``` Without calling `_get_pkg_ds_avail`, `info` just calls `_get_pkg_info` and converts it into a dict and add a license field. Since `_get_pkg_ds_avail` is used nowhere else, move it into `info`. Do not query Installed-Size, because it was stripped as "technical" key previously. Bug: saltstack#52605 Bug-Debian: https://bugs.debian.org/944970 Forwarded: saltstack#58735 Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>

salt contains modules, which directly access the dpkg internal database, instead of using one of the public interfaces provided by dpkg. This is a problem for several reasons, because even though the layout and format of the dpkg database is administrator friendly, and it is expected that those might need to mess with it, in case of emergency, this “interface” does not extend to other programs besides the dpkg suite of tools. The admindir can also be configured differently at dpkg build or run-time. And finally, the contents and its format, will be changing in the near future. So use something like: ``` dpkg-query --showformat '${db-fsys:Last-Modified}\n' --show $pkg ``` to get the mtime from /var/lib/dpkg/info/<package>.list files. Bug: saltstack#52605 Bug-Debian: https://bugs.debian.org/944970 Forwarded: saltstack#58735 Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>

salt contains modules, which directly access the dpkg internal database, instead of using one of the public interfaces provided by dpkg. This is a problem for several reasons, because even though the layout and format of the dpkg database is administrator friendly, and it is expected that those might need to mess with it, in case of emergency, this “interface” does not extend to other programs besides the dpkg suite of tools. The admindir can also be configured differently at dpkg build or run-time. And finally, the contents and its format, will be changing in the near future. `/var/lib/dpkg/available` should not be read directly. Instead `dpkg-query --print-avail` should be used. The `/var/lib/dpkg/available` file is only kept up-to-date when using dselect, but nowadays `apt` is used, and therefore this file does not provide much more information than `dpkg-query -W`. Users of APT-based frontends should use `apt show` or in our case the `pkg.show` salt module. `dpkg-query --print-avail` provides following keys (on Ubuntu 20.04): ``` $ dpkg-query --print-avail | sed 's/:.*$//' | sort | uniq Architecture Breaks Bugs Build-Essential Built-Using Cnf-Extra-Commands Cnf-Priority-Bonus Conflicts Depends Description Description-md5 Enhances Essential Filename Homepage Important Installed-Size Maintainer MD5sum Multi-Arch Origin Original-Maintainer Package Pre-Depends Priority Provides Recommends Replaces Section SHA1 SHA256 Size Source Suggests Supported Task Version ``` Following keys cannot be retrieved by `dpkg-query -W`: ``` Build-Essential Built-Using Cnf-Extra-Commands Cnf-Priority-Bonus Important Multi-Arch Original-Maintainer SHA1 SHA256 Supported ``` Please use `pkg.show` for those keys. Following additional keys could be retrieved if needed: ``` Breaks Enhances Essential Filename Pre-Depends Priority Suggests ``` Without calling `_get_pkg_ds_avail`, `info` just calls `_get_pkg_info` and converts it into a dict and add a license field. Since `_get_pkg_ds_avail` is used nowhere else, move it into `info`. Do not query Installed-Size, because it was stripped as "technical" key previously. Bug: saltstack#52605 Bug-Debian: https://bugs.debian.org/944970 Forwarded: saltstack#58735 Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>

salt contains modules, which directly access the dpkg internal database, instead of using one of the public interfaces provided by dpkg. This is a problem for several reasons, because even though the layout and format of the dpkg database is administrator friendly, and it is expected that those might need to mess with it, in case of emergency, this “interface” does not extend to other programs besides the dpkg suite of tools. The admindir can also be configured differently at dpkg build or run-time. And finally, the contents and its format, will be changing in the near future. So use something like: ``` dpkg-query --showformat '${db-fsys:Last-Modified}\n' --show $pkg ``` to get the mtime from /var/lib/dpkg/info/<package>.list files. Bug: saltstack#52605 Bug-Debian: https://bugs.debian.org/944970 Forwarded: saltstack#58735 Signed-off-by: Benjamin Drung <benjamin.drung@ionos.com> Gbp-Pq: Name dpkg_lowpkg-Do-not-access-var-lib-dpkg-info-package-.patch

salt contains modules, which directly access the dpkg internal database, instead of using one of the public interfaces provided by dpkg. This is a problem for several reasons, because even though the layout and format of the dpkg database is administrator friendly, and it is expected that those might need to mess with it, in case of emergency, this “interface” does not extend to other programs besides the dpkg suite of tools. The admindir can also be configured differently at dpkg build or run-time. And finally, the contents and its format, will be changing in the near future. `/var/lib/dpkg/available` should not be read directly. Instead `dpkg-query --print-avail` should be used. The `/var/lib/dpkg/available` file is only kept up-to-date when using dselect, but nowadays `apt` is used, and therefore this file does not provide much more information than `dpkg-query -W`. Users of APT-based frontends should use `apt show` or in our case the `pkg.show` salt module. `dpkg-query --print-avail` provides following keys (on Ubuntu 20.04): ``` $ dpkg-query --print-avail | sed 's/:.*$//' | sort | uniq Architecture Breaks Bugs Build-Essential Built-Using Cnf-Extra-Commands Cnf-Priority-Bonus Conflicts Depends Description Description-md5 Enhances Essential Filename Homepage Important Installed-Size Maintainer MD5sum Multi-Arch Origin Original-Maintainer Package Pre-Depends Priority Provides Recommends Replaces Section SHA1 SHA256 Size Source Suggests Supported Task Version ``` Following keys cannot be retrieved by `dpkg-query -W`: ``` Build-Essential Built-Using Cnf-Extra-Commands Cnf-Priority-Bonus Important Multi-Arch Original-Maintainer SHA1 SHA256 Supported ``` Please use `pkg.show` for those keys. Following additional keys could be retrieved if needed: ``` Breaks Enhances Essential Filename Pre-Depends Priority Suggests ``` Without calling `_get_pkg_ds_avail`, `info` just calls `_get_pkg_info` and converts it into a dict and add a license field. Since `_get_pkg_ds_avail` is used nowhere else, move it into `info`. Do not query Installed-Size, because it was stripped as "technical" key previously. Bug: saltstack#52605 Bug-Debian: https://bugs.debian.org/944970 Forwarded: saltstack#58735 Signed-off-by: Benjamin Drung <benjamin.drung@ionos.com> Gbp-Pq: Name dpkg_lowpkg-Drop-reading-var-lib-dpkg-available-dire.patch

salt contains modules, which directly access the dpkg internal database, instead of using one of the public interfaces provided by dpkg. This is a problem for several reasons, because even though the layout and format of the dpkg database is administrator friendly, and it is expected that those might need to mess with it, in case of emergency, this “interface” does not extend to other programs besides the dpkg suite of tools. The admindir can also be configured differently at dpkg build or run-time. And finally, the contents and its format, will be changing in the near future. So use something like: ``` dpkg-query --showformat '${db-fsys:Last-Modified}\n' --show $pkg ``` to get the mtime from /var/lib/dpkg/info/<package>.list files. Bug: saltstack#52605 Bug-Debian: https://bugs.debian.org/944970 Forwarded: saltstack#58735 Signed-off-by: Benjamin Drung <benjamin.drung@ionos.com>

salt contains modules, which directly access the dpkg internal database, instead of using one of the public interfaces provided by dpkg. This is a problem for several reasons, because even though the layout and format of the dpkg database is administrator friendly, and it is expected that those might need to mess with it, in case of emergency, this “interface” does not extend to other programs besides the dpkg suite of tools. The admindir can also be configured differently at dpkg build or run-time. And finally, the contents and its format, will be changing in the near future. `/var/lib/dpkg/available` should not be read directly. Instead `dpkg-query --print-avail` should be used. The `/var/lib/dpkg/available` file is only kept up-to-date when using dselect, but nowadays `apt` is used, and therefore this file does not provide much more information than `dpkg-query -W`. Users of APT-based frontends should use `apt show` or in our case the `pkg.show` salt module. `dpkg-query --print-avail` provides following keys (on Ubuntu 20.04): ``` $ dpkg-query --print-avail | sed 's/:.*$//' | sort | uniq Architecture Breaks Bugs Build-Essential Built-Using Cnf-Extra-Commands Cnf-Priority-Bonus Conflicts Depends Description Description-md5 Enhances Essential Filename Homepage Important Installed-Size Maintainer MD5sum Multi-Arch Origin Original-Maintainer Package Pre-Depends Priority Provides Recommends Replaces Section SHA1 SHA256 Size Source Suggests Supported Task Version ``` Following keys cannot be retrieved by `dpkg-query -W`: ``` Build-Essential Built-Using Cnf-Extra-Commands Cnf-Priority-Bonus Important Multi-Arch Original-Maintainer SHA1 SHA256 Supported ``` Please use `pkg.show` for those keys. Following additional keys could be retrieved if needed: ``` Breaks Enhances Essential Filename Pre-Depends Priority Suggests ``` Without calling `_get_pkg_ds_avail`, `info` just calls `_get_pkg_info` and converts it into a dict and add a license field. Since `_get_pkg_ds_avail` is used nowhere else, move it into `info`. Do not query Installed-Size, because it was stripped as "technical" key previously. Bug: saltstack#52605 Bug-Debian: https://bugs.debian.org/944970 Forwarded: saltstack#58735 Signed-off-by: Benjamin Drung <benjamin.drung@ionos.com>

Ch3LL · 2022-12-14T20:11:26Z

Looks like some of the test failures are related. I can give a review once they are cleaned up. Thanks

salt contains modules, which directly access the dpkg internal database, instead of using one of the public interfaces provided by dpkg. This is a problem for several reasons, because even though the layout and format of the dpkg database is administrator friendly, and it is expected that those might need to mess with it, in case of emergency, this “interface” does not extend to other programs besides the dpkg suite of tools. The admindir can also be configured differently at dpkg build or run-time. And finally, the contents and its format, will be changing in the near future. So use something like: ``` dpkg-query --showformat '${db-fsys:Last-Modified}\n' --show $pkg ``` to get the mtime from /var/lib/dpkg/info/<package>.list files. Bug: saltstack#52605 Bug-Debian: https://bugs.debian.org/944970 Signed-off-by: Benjamin Drung <benjamin.drung@ionos.com>

salt contains modules, which directly access the dpkg internal database, instead of using one of the public interfaces provided by dpkg. This is a problem for several reasons, because even though the layout and format of the dpkg database is administrator friendly, and it is expected that those might need to mess with it, in case of emergency, this “interface” does not extend to other programs besides the dpkg suite of tools. The admindir can also be configured differently at dpkg build or run-time. And finally, the contents and its format, will be changing in the near future. `/var/lib/dpkg/available` should not be read directly. Instead `dpkg-query --print-avail` should be used. The `/var/lib/dpkg/available` file is only kept up-to-date when using dselect, but nowadays `apt` is used, and therefore this file does not provide much more information than `dpkg-query -W`. Users of APT-based frontends should use `apt show` or in our case the `pkg.show` salt module. `dpkg-query --print-avail` provides following keys (on Ubuntu 20.04): ``` $ dpkg-query --print-avail | sed 's/:.*$//' | sort | uniq Architecture Breaks Bugs Build-Essential Built-Using Cnf-Extra-Commands Cnf-Priority-Bonus Conflicts Depends Description Description-md5 Enhances Essential Filename Homepage Important Installed-Size Maintainer MD5sum Multi-Arch Origin Original-Maintainer Package Pre-Depends Priority Provides Recommends Replaces Section SHA1 SHA256 Size Source Suggests Supported Task Version ``` Following keys cannot be retrieved by `dpkg-query -W`: ``` Build-Essential Built-Using Cnf-Extra-Commands Cnf-Priority-Bonus Important Multi-Arch Original-Maintainer SHA1 SHA256 Supported ``` Please use `pkg.show` for those keys. Following additional keys could be retrieved if needed: ``` Breaks Enhances Essential Filename Pre-Depends Priority Suggests ``` Without calling `_get_pkg_ds_avail`, `info` just calls `_get_pkg_info` and converts it into a dict and add a license field. Since `_get_pkg_ds_avail` is used nowhere else, move it into `info`. Do not query Installed-Size, because it was stripped as "technical" key previously. Bug: saltstack#52605 Bug-Debian: https://bugs.debian.org/944970 Signed-off-by: Benjamin Drung <benjamin.drung@ionos.com>

`dpkg_lowpkg.info` only returns information about installed packages. To query packges available in the configured APT archive(s), use the higher level aptpkg module. Signed-off-by: Benjamin Drung <benjamin.drung@ionos.com>

Avoid joining and splitting command parameters to `cmd.run_all` by passing directly a list as `cmd` parameter. Signed-off-by: Benjamin Drung <benjamin.drung@ionos.com>

`dpkg-query` does not know following keys: * binary:Revision (Revision exists, but is deprecated) * SHA1 * SHA256 * Summary Remove those keys since they will be always be empty and then stripped. Signed-off-by: Benjamin Drung <benjamin.drung@ionos.com>

bdrung requested a review from a team as a code owner October 15, 2020 17:22

bdrung requested review from cmcmarrow and removed request for a team October 15, 2020 17:22

ghost requested a review from krionbsd October 15, 2020 17:22

bdrung force-pushed the fix-dpkg branch 2 times, most recently from 43c63cb to d2b619e Compare October 16, 2020 09:16

bdrung changed the title ~~Do not access /var/lib/dpkg/info/<package>.list directly~~ Do not access dpkg internals directly Oct 16, 2020

bdrung force-pushed the fix-dpkg branch from a59417d to 4125277 Compare October 16, 2020 11:15

bdrung changed the title ~~Do not access dpkg internals directly~~ dpkg_lowpkg: Do not access dpkg internals directly Oct 16, 2020

bdrung force-pushed the fix-dpkg branch 2 times, most recently from f1450b2 to 5385163 Compare October 19, 2020 20:28

cmcmarrow previously approved these changes Oct 21, 2020

View reviewed changes

krionbsd previously approved these changes Nov 9, 2020

View reviewed changes

bdrung dismissed stale reviews from krionbsd and cmcmarrow via ebb595e April 28, 2021 18:19

bdrung force-pushed the fix-dpkg branch from c8b8a33 to ebb595e Compare April 28, 2021 18:19

bdrung force-pushed the fix-dpkg branch from e5d1067 to cb24cd1 Compare November 30, 2022 20:40

dwoz force-pushed the fix-dpkg branch from cb24cd1 to 6eca413 Compare December 10, 2023 10:11

dwoz had a problem deploying to ci December 10, 2023 10:38 — with GitHub Actions Error

bdrung added 5 commits December 10, 2023 03:45

dpkg_lowpkg: Clarify scope of dpkg_lowpkg.info

4dede27

`dpkg_lowpkg.info` only returns information about installed packages. To query packges available in the configured APT archive(s), use the higher level aptpkg module. Signed-off-by: Benjamin Drung <benjamin.drung@ionos.com>

dpkg_lowpkg: Pass command parameters as list

4f0d25c

Avoid joining and splitting command parameters to `cmd.run_all` by passing directly a list as `cmd` parameter. Signed-off-by: Benjamin Drung <benjamin.drung@ionos.com>

dwoz force-pushed the fix-dpkg branch from 6eca413 to 4fc3936 Compare December 10, 2023 10:45

dwoz temporarily deployed to ci December 10, 2023 17:29 — with GitHub Actions Inactive

dwoz temporarily deployed to ci December 10, 2023 17:30 — with GitHub Actions Inactive

dwoz temporarily deployed to ci December 11, 2023 02:00 — with GitHub Actions Inactive

dwoz temporarily deployed to ci December 11, 2023 02:01 — with GitHub Actions Inactive

dwoz temporarily deployed to ci December 11, 2023 02:02 — with GitHub Actions Inactive

dwoz temporarily deployed to ci December 11, 2023 03:31 — with GitHub Actions Inactive

dwoz temporarily deployed to ci December 11, 2023 06:57 — with GitHub Actions Inactive

dwoz temporarily deployed to ci December 11, 2023 06:58 — with GitHub Actions Inactive

dwoz added this to the Argon v3008.0 milestone Dec 18, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

dpkg_lowpkg: Do not access dpkg internals directly #58735

dpkg_lowpkg: Do not access dpkg internals directly #58735

bdrung commented Oct 15, 2020 •

edited

cmcmarrow commented Oct 21, 2020

Ch3LL commented Dec 14, 2022

dpkg_lowpkg: Do not access dpkg internals directly #58735

Are you sure you want to change the base?

dpkg_lowpkg: Do not access dpkg internals directly #58735

Conversation

bdrung commented Oct 15, 2020 • edited

cmcmarrow commented Oct 21, 2020

Ch3LL commented Dec 14, 2022

bdrung commented Oct 15, 2020 •

edited