Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update systemService apt monitoring #1290

Merged
merged 5 commits into from
May 3, 2024

Conversation

MorningLightMountain713
Copy link
Contributor

What this pull does

  • Simplifies aptMonitoring logic. We no longer wait for 12 hours and resume the apt queue if there is a failure. We just empty the queue and start fresh. resuming after 12 hours didn't make sense - the commands would be stale by then anyway.
  • We now gracefully, then forcefully terminate ANY process that holds a lock on the apt locks. This uses fuser in combindation with -k -TERM to gracefully ask the processes to quit, followed by -k -KILL if that fails. This is good as it looks at what processes are holding the locks on the actual files, not just killing off apt or apt-get.
  • Fix up tests related to the above changes.
  • Remove some old code from syncthing that was no longer used as of 5.6.0.
  • Rename some fifo queue variables that didn't make sense.

I've tested this on a live node by doing the following:

  • Downgrade syncthing to 1.27.5.
  • Run sudo apt-get remove syncthing but don't press y or n, so it hangs.
  • Reboot FluxOS, so it tries to install syncthing.
  • Apt Queue Does the following:
    • 3 minutes for initial try.
    • 5 retries @ 3 minutes.
    • 1 minute between retries ( 5 minutes)
    • Total of 23 minutes
  • Monitor apt queue then does:
    • 3 checks @ 10 minutes.
    • Total 30 minutes
    • Kills any processes that hold locks
  • Total time to resolve 53 minutes.

If this was run on a node that didn't have a newer version of apt, would take 35 minutes before the processes are killed.

All tests passing.

Copy link
Member

@Cabecinha84 Cabecinha84 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ack. Great work

@Cabecinha84 Cabecinha84 merged commit 0632380 into development May 3, 2024
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants