Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Call _nodes/shutdown from pre-stop hook #6544

Merged
merged 18 commits into from
Mar 22, 2023
Merged

Call _nodes/shutdown from pre-stop hook #6544

Changes from 1 commit
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
fb7726f
Call _nodes/shutdown from pre-stop hook
pebrc Mar 15, 2023
c7c9757
Remove design note
pebrc Mar 17, 2023
6110ca3
Document and add overrides via env vars
pebrc Mar 17, 2023
42b7f66
Exit code on error and templating
pebrc Mar 17, 2023
3fa9381
indentation
pebrc Mar 17, 2023
429dbfb
more indentation
pebrc Mar 17, 2023
428b883
left-over TODO removed
pebrc Mar 17, 2023
c626900
Add missing e2e secret check update
pebrc Mar 17, 2023
1bc0e91
review/shellcheck input
pebrc Mar 20, 2023
7f687a8
update ECK docs to mention restart requirement
pebrc Mar 20, 2023
229b14c
replace echo with log
pebrc Mar 20, 2023
81d7d16
Do not cancel pre-stop hook originated shutdowns
pebrc Mar 20, 2023
9b4054f
Do not set voting config exclusions in pre-stop hook
pebrc Mar 20, 2023
7ff1372
Update docs/orchestrating-elastic-stack-applications/elasticsearch/pr…
pebrc Mar 20, 2023
00ca6ed
include retry messages in log
pebrc Mar 21, 2023
2cd09e6
Do no automatically exit on error and do not hot loop retries while w…
pebrc Mar 21, 2023
bf17123
Always honor PRE_STOP_ADDITIONAL_WAIT_SECONDS, do not log $status we …
pebrc Mar 21, 2023
72569e8
Improve log message consistency
pebrc Mar 22, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
15 changes: 6 additions & 9 deletions pkg/controller/elasticsearch/nodespec/lifecycle_hook.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ const PreStopHookScriptConfigKey = "pre-stop-hook-script.sh"

var preStopHookScriptTemplate = template.Must(template.New("pre-stop").Parse(`#!/usr/bin/env bash

set -euo pipefail
set -uo pipefail

# This script will wait for up to $PRE_STOP_ADDITIONAL_WAIT_SECONDS before allowing termination of the Pod
# This slows down the process shutdown and allows to make changes to the pool gracefully, without blackholing traffic when DNS
Expand Down Expand Up @@ -109,11 +109,11 @@ function retry() {
function log() {
local timestamp
timestamp=$(date --iso-8601=seconds)
echo "{\"@timestamp\": \"${timestamp}\", \"message\": \"$@\", \"ecs.version\": \"1.2.0\", \"event.dataset\": \"elasticsearch.pre-stop-hook\"}" | tee /proc/1/fd/2 2> /dev/null
echo "{\"@timestamp\": \"${timestamp}\", \"message\": \"$*\", \"ecs.version\": \"1.2.0\", \"event.dataset\": \"elasticsearch.pre-stop-hook\"}" | tee /proc/1/fd/2 2> /dev/null
}

function error_exit() {
log "$1"
log "$@"
exit 1
}

Expand Down Expand Up @@ -165,7 +165,7 @@ ES_URL={{.ServiceURL}}

log "retrieving node ID"
retry 10 request -X GET "$ES_URL/_cat/nodes?full_id=true&h=id,name" $BASIC_AUTH
if [ "$?" -ne 0 ]; then
if [ "$?" -ne 0 ]; then
error_exit "$status"
thbkrkr marked this conversation as resolved.
Show resolved Hide resolved
fi

Expand All @@ -186,17 +186,14 @@ retry 10 request -X PUT $ES_URL/_nodes/"$NODE_ID"/shutdown $BASIC_AUTH -H 'Conte
}
"
if [ "$?" -ne 0 ]; then
error_exit "Failed to call node shutdown API" "$resp_body"
error_exit "Failed to call node shutdown API" "$status"
fi

while :
do
log "waiting for node shutdown to complete"
request -X GET $ES_URL/_nodes/"$NODE_ID"/shutdown $BASIC_AUTH
if [ "$?" -ne 0 ]; then
continue
fi
if grep -q -v 'IN_PROGRESS\|STALLED' "$resp_body"; then
if [ "$?" -eq 0 ] && grep -q -v 'IN_PROGRESS\|STALLED' "$resp_body"; then
break
fi
sleep 10
Expand Down