Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dashboard rwo pvc restart #3972

Merged
merged 4 commits into from
Mar 17, 2023

Conversation

torbendury
Copy link
Contributor

@torbendury torbendury commented Mar 11, 2023

What problem does this PR solve?

Close #3876

What's changed and how it works?

I have changed the dashboards updateStrategy. The PersistentVolumeClaim for the dashboard is created with the accessMode ReadWriteOnce, which does not allow the dashboard to do a rolling restart because the blue version of the dashboard won't be shutdown until the green version is up and running.

This lead to problems when doing a kubectl rollout-restart deployment chaos-dashboard because the new dashboard Pod could not mount the PVC.

Related changes

  • This change also requires further updates to the website (e.g. docs)
  • This change also requires further updates to the UI interface
  • Need to cheery-pick to release branches
    • release-2.5
    • release-2.4

Checklist

CHANGELOG

  • I have updated the CHANGELOG.md
  • I have labeled this PR with "no-need-update-changelog"

Tests

  • Unit test
  • E2E test
  • No code
  • Manual test (add steps below)

steps:

  • Install the ChaosMesh + dashboard using a helm install command
    • configure the dashboard to be enabled and use a PersistentVolumeClaim of arbitrary size
  • Change the update strategy as mentioned in this PR
  • Watch the dashboard rollout-restart instead of being stuck

Side effects

  • Breaking backward compatibility

DCO

If you find the DCO check fails, please run commands like below (Depends on the actual situations. For example, if the failed commit isn't the most recent) to fix it:

git commit --amend --signoff
git push --force

@ti-chi-bot
Copy link
Member

ti-chi-bot commented Mar 11, 2023

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • STRRL
  • g1eny0ung

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot
Copy link
Member

Welcome @torbendury!

It looks like this is your first PR to chaos-mesh/chaos-mesh 🎉.

I'm the bot to help you request reviewers, add labels and more, See available commands.

We want to make sure your contribution gets all the attention it needs!



Thank you, and welcome to chaos-mesh/chaos-mesh. 😃

Signed-off-by: Torben Dury <torbendury@googlemail.com>
Signed-off-by: Torben Dury <torbendury@googlemail.com>
@torbendury
Copy link
Contributor Author

Apparently it's not that easy to sign-off commits using GitHub Desktop, anyway I was able to force push over my mistake. Open for comments/review!

@g1eny0ung
Copy link
Member

Apparently it's not that easy to sign-off commits using GitHub Desktop, anyway I was able to force push over my mistake. Open for comments/review!

Ah, I'm also facing this problem before. The GitHub desktop lacks sign-off commits support. :(

@torbendury
Copy link
Contributor Author

Are those failing checks related to my PR? I took a quick look but they don't really seem to correlate imho

Copy link
Member

@g1eny0ung g1eny0ung left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm ok with this solution of setting the strategy to type: Recreate because real-time availability of the dashboard is not so critical.

We can also make a separate option for ReadWriteOnce PV or PVC, like:

strategy:
  {{- if eq .Values.dashboard.persistentVolume.accessModes "ReadWriteOnce" }}
  type: Recreate
  {{- else }}
  # ...

I think both are fine. What do you think? @STRRL

@g1eny0ung g1eny0ung requested review from STRRL and removed request for iguoyr March 12, 2023 12:35
@g1eny0ung
Copy link
Member

Are those failing checks related to my PR? I took a quick look but they don't really seem to correlate IMHO

@torbendury It can be said to be irrelevant. These errors are not about the charts themselves, they are just used to prevent some omissions (because of the need to synchronize the chart's changes to install.sh).

Run make install.sh in your local and commit again can pass the checks. 😃

@torbendury
Copy link
Contributor Author

I also thought of making it depending on the accessMode of the PVC itself, but it is (at least at the moment!) hardcoded here by the Helm Chart itself, if I didn't overlook anything

Signed-off-by: Torben Dury <torbendury@googlemail.com>
@g1eny0ung
Copy link
Member

I also thought of making it depending on the accessMode of the PVC itself, but it is (at least at the moment!) hardcoded here by the Helm Chart itself, if I didn't overlook anything

Good catch! Then I think about the accessMode we can consider in a separate PR.

Copy link
Member

@STRRL STRRL left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@STRRL
Copy link
Member

STRRL commented Mar 16, 2023

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: 83b4cb9

@ti-chi-bot
Copy link
Member

@torbendury: Your PR was out of date, I have automatically updated it for you.

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@ti-chi-bot ti-chi-bot merged commit 2d1cea6 into chaos-mesh:master Mar 17, 2023
@ti-chi-bot ti-chi-bot mentioned this pull request Mar 17, 2023
11 tasks
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created to branch release-2.5: #3975.

g1eny0ung added a commit that referenced this pull request Mar 17, 2023
* This is an automated cherry-pick of #3972

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

* Update CHANGELOG.md

---------

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
Co-authored-by: Torben Dury <94752317+torbendury@users.noreply.github.com>
Co-authored-by: Yue Yang <g1enyy0ung@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Dashboard not rolling-restarting
4 participants