Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[alerts] increase GitpodWorkspaceStuckOnStopping for time to 30min to reduce flakiness #12316

Merged
merged 1 commit into from
Aug 23, 2022

Conversation

sagor999
Copy link
Contributor

Description

This alert sometimes fires incorrectly due to the number of workspaces that we have running in the cluster.
This is a short term mitigation for this alert.

Long term mitigation could be re-thinking on how we alert on this (change it to % of workspaces in the cluster for example).

Related Issue(s)

Fixes #

How to test

Release Notes

none

Documentation

Werft options:

  • /werft with-preview

@sagor999 sagor999 requested a review from a team August 23, 2022 18:30
@github-actions github-actions bot added the team: workspace Issue belongs to the Workspace team label Aug 23, 2022
@roboquat roboquat merged commit cc79d75 into main Aug 23, 2022
@roboquat roboquat deleted the pavel/fix-alert branch August 23, 2022 18:32
@kylos101
Copy link
Contributor

@sagor999 🙏 thank you! If you want to push this to active clusters for this week to reduce on-caller noise, now that this PR is merged, the balance of the process is:

  1. Run this and once it's done
  2. Run our enable alerts werft job once for each workspace cluster

@roboquat roboquat added deployed: workspace Workspace team change is running in production deployed Change is completely running in production labels Aug 24, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
deployed: workspace Workspace team change is running in production deployed Change is completely running in production release-note-none size/XS team: workspace Issue belongs to the Workspace team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants