Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🐛 fix/low-CPU-load healthcheck for migration service #3477

Merged
merged 4 commits into from
Oct 27, 2022

Conversation

pcrespov
Copy link
Member

@pcrespov pcrespov commented Oct 26, 2022

What do these changes do?

New healthcheck function for the migration service with low CPU footprint.

Why this change ?

Previous healthcheck has been producing an almost constant (~13 %) CPU usage. See @mrnicegyu11's promQL request
Screen Shot 2022-10-26 at 16 03 42

After a couple of days, that ended up consuming all CPU credit in AWS. See outcome from @Surfict's research:
image

This is probably what caused the instabilities we have been experiencing in the last days in the e2e.

🚑️ This should be hot-fixed to staging before ITISFoundation/osparc-issues#753

@pcrespov pcrespov changed the title 🐛 fix/low-CPU migration healthcheck WIP: 🐛 fix/low-CPU migration healthcheck Oct 26, 2022
@pcrespov pcrespov self-assigned this Oct 26, 2022
@pcrespov pcrespov changed the title WIP: 🐛 fix/low-CPU migration healthcheck WIP: 🐛 fix/low-CPU intensity healthcheck for migration service Oct 26, 2022
@pcrespov pcrespov changed the title WIP: 🐛 fix/low-CPU intensity healthcheck for migration service WIP: 🐛 fix/low-CPU-load healthcheck for migration service Oct 26, 2022
@pcrespov pcrespov added this to the Katherine Switzer milestone Oct 26, 2022
Copy link
Member

@mrnicegyu11 mrnicegyu11 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks looks good, I guess by checking the existence of the "done" file you avoid pgSQL queries. Looks promissing

@pcrespov pcrespov changed the title WIP: 🐛 fix/low-CPU-load healthcheck for migration service 🐛 fix/low-CPU-load healthcheck for migration service Oct 26, 2022
@pcrespov pcrespov marked this pull request as ready for review October 26, 2022 14:13
Copy link
Member

@sanderegg sanderegg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

waow, that is an interesting side effect... I'd like to have some more info about that CPU credit thing to understand it better.

@sonarcloud
Copy link

sonarcloud bot commented Oct 27, 2022

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

No Coverage information No Coverage information
0.0% 0.0% Duplication

@pcrespov pcrespov merged commit 6e3ecfb into ITISFoundation:master Oct 27, 2022
@pcrespov pcrespov deleted the fix/migration-healthcheck branch October 27, 2022 08:10
@pcrespov
Copy link
Member Author

Before - After

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants