Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: make sure to clean up any lingering processes #1214

Merged
merged 2 commits into from
Jul 12, 2023
Merged

Conversation

Arqu
Copy link
Collaborator

@Arqu Arqu commented Jul 11, 2023

Description

Occasionally there are lingering iroh/derper nodes running from tests (probably netsim failed runs) which might not be cleaned up properly (and might impact future runs on those boxes). This makes sure it kills anything remaining.

Notes & open questions

Change checklist

  • Self-review.
  • Documentation updates if relevant.
  • Tests if relevant.

@Arqu Arqu added the ci continous integration label Jul 11, 2023
@Arqu Arqu self-assigned this Jul 11, 2023
Copy link
Contributor

@flub flub left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this looks like a great reason to use cgroups in some capacity. So that an entire CI run is contained in some resource groups and you can make sure everything spawned in that groups is killed at the end.

systemd probably has facilities for this, though I'd also have to spend a while figuring those out. Anyway, only an idea to maybe use in the future. I'm kind of surprised this isn't part of the github runner already and that you have to deal with this here.

@Arqu
Copy link
Collaborator Author

Arqu commented Jul 12, 2023

It's a bit more involved I think. So the GH runner actually does do it's own stuff correctly. The issue here afaict is that the netsim depends on an external process that's already running on the box and manages coordination and spawning (which in this case is outside of the GH jurisdiction) on failed runs can leave loose nodes running. If I see it again I can check on the process tree to see exactly. Anyways this is a cheap way to do it for now.

@Arqu Arqu merged commit f782fef into main Jul 12, 2023
@Arqu Arqu deleted the arqu/ci_cleanup branch July 12, 2023 08:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ci continous integration
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

2 participants