Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pre-backup-lock/post-backup-unlock scripts are not idempotent #140

Closed
x6j8x opened this issue Apr 24, 2019 · 2 comments
Closed

pre-backup-lock/post-backup-unlock scripts are not idempotent #140

x6j8x opened this issue Apr 24, 2019 · 2 comments

Comments

@x6j8x
Copy link

x6j8x commented Apr 24, 2019

Issue

pre-backup-lock/post-backup-unlock scripts are not idempotent.

Context

If for whatever reason you have to run the scripts again if the instance is already locked/unlocked they will fail.

Steps to Reproduce

Run bbr backup and then bbr backup-cleanup - the latter will fail due to the api instance being already unlocked.

Error attempting to run post-backup-unlock for job cloud_controller_ng on api/...

Expected result

Running pre-backup-lock or post-backup-unlock should always succeed even if the respective state (instance being in maintenance mode or not) is already active.

Current result

It fails if the instance is already in the desired state.

Possible Fix

Use the already existing /var/vcap/data/cloud_controller_ng/tmp/proxy_temp/bbr_maintenance "semaphore" file as indicator whether or not the instance is in the desired state is already (or employ a more sophisticated algorithm checking for the existence of the nginx maintenance process).

@cf-gitbot
Copy link

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/165556252

The labels on this github issue will be updated when the story is started.

monamohebbi added a commit that referenced this issue May 30, 2019
* Verifying failed health checks in the unlock scripts will fail if ccng is already up and unlocked
* In response to issue #140 (#140)

[Finishes #165556252](https://www.pivotaltracker.com/story/show/165556252)

Co-authored-by: Mona Mohebbi <mmohebbi@pivotal.io>
Co-authored-by: Connor Braa <cbraa@pivotal.io>
@tcdowney
Copy link
Member

tcdowney commented Jun 4, 2019

Thanks for reporting this @x6j8x!

The following two commits should fix this issue and will be available in the next capi-release:

Basically it just checks with bpm list to see if the nginx_maintenance job is already in the stopped state. Not super sophisticated, but gets the job done since bpm already knows how to track the lifecycle of the process.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants