Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

apt info should report cache age #180

Closed
anarcat opened this issue Oct 11, 2023 · 4 comments · Fixed by #182
Closed

apt info should report cache age #180

anarcat opened this issue Oct 11, 2023 · 4 comments · Fixed by #182

Comments

@anarcat
Copy link
Contributor

anarcat commented Oct 11, 2023 •
edited

We cannot alert on stale apt caches. Combined with #179, we can quickly end up in a situation where critical security upgrades are delayed or never installed.

We should have a timestamp metric showing when the last apt update was ran. Optionnally, we could have that metric per mirror as well.
@anarcat anarcat mentioned this issue Oct 13, 2023
Closed
@anarcat
Copy link
Contributor Author

anarcat commented Oct 13, 2023

@julian-klode what would be the canonical way to check when apt update ran last? maybe the last mod time on /var/cache/apt/pkgcache.bin? i guess we could also populate a separate metric based on the /var/lib/apt/lists/*InRelease timestamp, or should we parse the Date: field in those??

thanks, and sorry for the ping if that's inappropriate...

anarcat added a commit to anarcat/node-exporter-textfile-collector-scripts that referenced this issue Oct 13, 2023
@anarcat
report the apt cache timestamp
3a85cac 
We use the `pkgcache.bin` modification time as a heuristic, but there
might be a better way to do this.

We also silently ignore errors when pulling that file to avoid broken
systems from breaking the other metrics. I believe this will lead to a
null metric which is probably what we want anyway.

A possible improvement to this would be to individually inspect the
`InRelease` files and report the mirror's timestamps as well, but
that's more involved. It's also not a priority compared to this fix,
because we want the update timestamp if we remove the `apt update`
run (in prometheus-community#181).

Closes: prometheus-community#180

Signed-off-by: Antoine Beaupré <anarcat@debian.org>
@anarcat anarcat mentioned this issue Oct 13, 2023
Merged
@anarcat
Copy link
Contributor Author

anarcat commented Oct 13, 2023

i picked only pkgcache.bin in #182 for now, with the understanding that it's an important metric to add once we remove the auto-update in #181. This could be extended to cover mirror ages later on, but I suspect that would trigger a lot of noise in a larger fleet, as each server would individually report that its list is out of date... I guess an alerting rule could be tweaked to avoid alerting too often, but still... seems overkill for now.

@geor-g
Copy link

geor-g commented Oct 13, 2023

apt ships apt.systemd.daily, which gets installed into /usr/lib/apt/. It provides check_stamp() and relies on /var/lib/apt/periodic/update-stamp. I can't tell if it makes sense to rely on this code, but in any case this might provide an idea how to implement this.

@anarcat
Copy link
Contributor Author

anarcat commented Oct 14, 2023

yeah, in #181 i actually document how that works too, or at least an example of it.

anarcat added a commit to anarcat/node-exporter-textfile-collector-scripts that referenced this issue Oct 14, 2023
@anarcat @SuperQ
report the apt cache timestamp
dc6568e 
We use the `pkgcache.bin` modification time as a heuristic, but there
might be a better way to do this.

We also silently ignore errors when pulling that file to avoid broken
systems from breaking the other metrics. I believe this will lead to a
null metric which is probably what we want anyway.

A possible improvement to this would be to individually inspect the
`InRelease` files and report the mirror's timestamps as well, but
that's more involved. It's also not a priority compared to this fix,
because we want the update timestamp if we remove the `apt update`
run (in prometheus-community#181).

Closes: prometheus-community#180

Signed-off-by: Antoine Beaupré <anarcat@debian.org>
Co-authored-by: Ben Kochie <superq@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

report the apt cache timestamp anarcat/node-exporter-textfile-collector-scripts
2 participants
@anarcat @geor-g