Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Clean up stale HA nodes on restore #396

Merged
merged 12 commits into from May 21, 2018
Merged

Clean up stale HA nodes on restore #396

Changes from 3 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
90b7666
Clean up stale HA nodes
lildude Apr 26, 2018
786af0f
Merge branch 'master' into lildude/clean-old-nodes-on-restore
lildude May 4, 2018
374ff58
Merge branch 'master' into lildude/clean-old-nodes-on-restore
lildude May 10, 2018
75f8209
Merge branch 'master' into lildude/clean-old-nodes-on-restore
lildude May 14, 2018
230d1c1
Remove all other nodes
lildude May 14, 2018
8fcb84c
Simplify and make more readible
lildude May 16, 2018
99a8ea3
Add dummy cmds for testing
lildude May 17, 2018
b96b87a
Test to ensure stale node cleanup takes place
lildude May 17, 2018
06279da
Merge branch 'master' into lildude/clean-old-nodes-on-restore
lildude May 17, 2018
e34aede
Install jq on Travis
lildude May 17, 2018
b74d601
Merge branch 'lildude/clean-old-nodes-on-restore' of github.com:githu…
lildude May 17, 2018
184156d
Merge branch 'master' into lildude/clean-old-nodes-on-restore
lildude May 21, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
10 changes: 10 additions & 0 deletions bin/ghe-restore
View file
Open in desktop
Expand Up @@ -341,6 +341,16 @@ else
fi
fi

# Clean up all stale replicas
if ! $CLUSTER; then
inactive_nodes=$(echo "ghe-spokes server show inactive --json | jq -r '.[] | select(.host | contains(\"git-server\")).host' | sed 's/^git-server-//g'" | ghe-ssh "$GHE_HOSTNAME" -- /bin/bash)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately this fails to cleanup the stale HA nodes if the target instance is in an unconfigured state (and the replica(s) were online at the time of the backup).

This occurs because when unconfigured, we don't automatically run ghe-config-apply near the end of the restore, which results in the nodes still appearing as online in the database. It takes a ghe-config-apply for these stale nodes to transition to a offline state.

As an alternative, could we drop the inactive from the ghe-spokes server show and instead just cleanup any nodes that don't match the UUID that we restored (as found in $GHE_RESTORE_SNAPSHOT_PATH/uuid)?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ooh, good point. I completely forgot about restoring to an unconfigured instance. Your suggestion makes sense too.

if [ -n "$inactive_nodes" ]; then
echo "Cleaning up stale nodes ..."
for uuid in $inactive_nodes; do
ghe-ssh "$GHE_HOSTNAME" -- "/usr/local/share/enterprise/ghe-cluster-cleanup-node $uuid" 1>&3
done
fi
fi

# Update the remote status to "complete". This has to happen before importing
# ssh host keys because subsequent commands will fail due to the host key
Expand Down