Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

apt_info.py can hang for hours #179

Closed
anarcat opened this issue Oct 11, 2023 · 1 comment · Fixed by #181
Closed

apt_info.py can hang for hours #179

anarcat opened this issue Oct 11, 2023 · 1 comment · Fixed by #181

Comments

@anarcat
Copy link
Contributor

anarcat commented Oct 11, 2023

We have a situation here where numerous machines are seeing the apt_info.py script hang for hours. This has been reported in Debian as bug https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1028212 and is being tracked in our internal tracker as https://gitlab.torproject.org/tpo/tpa/team/-/issues/41355

This is possibly an upstream issue, part of python-apt, requires further investigation. Current workaround includes installing a TimeoutStartSec=30s in the systemd unit [Service] block.
@anarcat anarcat mentioned this issue Oct 11, 2023
Closed
@anarcat
Copy link
Contributor Author

anarcat commented Oct 13, 2023

In https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1028212#62, @julian-klode has an excellent point that apt_info.py should not trigger apt update on its own. There are out of band mechanisms to do this (e.g. unattended-upgrades or apt itself, actually asked jak for clarification on that) that are supposed to take care of this. apt_info.py's job is simply to collect the numbers and dump them into something prometheus can parse. It should do the least possible for this.

For out of date mirrors or data, it should report the cache age (#180), not try to pre-emptively update it.

So i'll make a PR to fix that shortly.

anarcat added a commit to anarcat/node-exporter-textfile-collector-scripts that referenced this issue Oct 13, 2023
@anarcat
do not run apt update or simulate apt dist-upgrade
dc995a9 
This is causing all sorts of problems. The first of which is that
we're hitting our poor mirrors every time the script is ran, which, in
the Debian package configuration, is *every 15 minutes* (!!).

The second is that this locks the cache and makes this script
needlessly stumble upon a possible regression in APT from Debian
bookworm and Ubuntu 22.06:

https://bugs.launchpad.net/ubuntu/+source/apt/+bug/2003851

That still has to be confirmed: it's possible that `apt update` can
hang for a long time, but that shouldn't concern us if we delegate
this work out of band.

I also do not believe actually performing the `dist-upgrade`
calculations is necessary to compute the pending upgrades at all. I've
done work with python-apt for other projects and haven't found that to
be required: the cache has the necessary information about pending
upgrades.

Closes: prometheus-community#179
@anarcat anarcat mentioned this issue Oct 13, 2023
Merged
anarcat added a commit to anarcat/node-exporter-textfile-collector-scripts that referenced this issue Oct 13, 2023
@anarcat
do not run apt update or simulate apt dist-upgrade
3b17a4d 
This is causing all sorts of problems. The first of which is that
we're hitting our poor mirrors every time the script is ran, which, in
the Debian package configuration, is *every 15 minutes* (!!).

The second is that this locks the cache and makes this script
needlessly stumble upon a possible regression in APT from Debian
bookworm and Ubuntu 22.06:

https://bugs.launchpad.net/ubuntu/+source/apt/+bug/2003851

That still has to be confirmed: it's possible that `apt update` can
hang for a long time, but that shouldn't concern us if we delegate
this work out of band.

I also do not believe actually performing the `dist-upgrade`
calculations is necessary to compute the pending upgrades at all. I've
done work with python-apt for other projects and haven't found that to
be required: the cache has the necessary information about pending
upgrades.

Closes: prometheus-community#179

Signed-off-by: Antoine Beaupré <anarcat@debian.org>
anarcat added a commit to anarcat/node-exporter-textfile-collector-scripts that referenced this issue Oct 14, 2023
@anarcat
do not run apt update or simulate apt dist-upgrade
fddbd16 
This is causing all sorts of problems. The first of which is that
we're hitting our poor mirrors every time the script is ran, which, in
the Debian package configuration, is *every 15 minutes* (!!).

The second is that this locks the cache and makes this script
needlessly stumble upon a possible regression in APT from Debian
bookworm and Ubuntu 22.06:

https://bugs.launchpad.net/ubuntu/+source/apt/+bug/2003851

That still has to be confirmed: it's possible that `apt update` can
hang for a long time, but that shouldn't concern us if we delegate
this work out of band.

I also do not believe actually performing the `dist-upgrade`
calculations is necessary to compute the pending upgrades at all. I've
done work with python-apt for other projects and haven't found that to
be required: the cache has the necessary information about pending
upgrades.

Closes: prometheus-community#179

Signed-off-by: Antoine Beaupré <anarcat@debian.org>
anarcat added a commit to anarcat/node-exporter-textfile-collector-scripts that referenced this issue Oct 15, 2023
@anarcat @dswarbrick
do not run apt update or simulate apt dist-upgrade
524dbae 
This is causing all sorts of problems. The first of which is that
we're hitting our poor mirrors every time the script is ran, which, in
the Debian package configuration, is *every 15 minutes* (!!).

The second is that this locks the cache and makes this script
needlessly stumble upon a possible regression in APT from Debian
bookworm and Ubuntu 22.06:

https://bugs.launchpad.net/ubuntu/+source/apt/+bug/2003851

That still has to be confirmed: it's possible that `apt update` can
hang for a long time, but that shouldn't concern us if we delegate
this work out of band.

I also do not believe actually performing the `dist-upgrade`
calculations is necessary to compute the pending upgrades at all. I've
done work with python-apt for other projects and haven't found that to
be required: the cache has the necessary information about pending
upgrades.

Closes: prometheus-community#179

Signed-off-by: Antoine Beaupré <anarcat@debian.org>
Co-authored-by: Daniel Swarbrick <daniel.swarbrick@gmail.com>
anarcat added a commit to anarcat/node-exporter-textfile-collector-scripts that referenced this issue Oct 16, 2023
@anarcat
do not run apt update or simulate apt dist-upgrade
28c179d 
This is causing all sorts of problems. The first of which is that
we're hitting our poor mirrors every time the script is ran, which, in
the Debian package configuration, is *every 15 minutes* (!!).

The second is that this locks the cache and makes this script
needlessly stumble upon a possible regression in APT from Debian
bookworm and Ubuntu 22.06:

https://bugs.launchpad.net/ubuntu/+source/apt/+bug/2003851

That still has to be confirmed: it's possible that `apt update` can
hang for a long time, but that shouldn't concern us if we delegate
this work out of band.

I also do not believe actually performing the `dist-upgrade`
calculations is necessary to compute the pending upgrades at all. I've
done work with python-apt for other projects and haven't found that to
be required: the cache has the necessary information about pending
upgrades.

Closes: prometheus-community#179

Signed-off-by: Antoine Beaupré <anarcat@debian.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

do not run apt update or simulate apt dist-upgrade anarcat/node-exporter-textfile-collector-scripts
1 participant
@anarcat