Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use results of kube-controller-manager leader election in addon manager #55466

Merged
merged 2 commits into from
Nov 14, 2017
Merged

Use results of kube-controller-manager leader election in addon manager #55466

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
5e4692f
Use results of kube-controller-manager leader election in addon manager
x13n Nov 10, 2017
ae6e506
Merge branch 'master' into addon-manager
x13n Nov 14, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 1 addition & 1 deletion cluster/addons/addon-manager/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
IMAGE=gcr.io/google-containers/kube-addon-manager
ARCH?=amd64
TEMP_DIR:=$(shell mktemp -d)
VERSION=v6.4-beta.2
VERSION=v6.5
KUBECTL_VERSION?=v1.6.4

ifeq ($(ARCH),amd64)
Expand Down
32 changes: 30 additions & 2 deletions cluster/addons/addon-manager/kube-addons.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,11 @@ ADDON_MANAGER_LABEL="addonmanager.kubernetes.io/mode"
# will be reconciled for now.
CLUSTER_SERVICE_LABEL="kubernetes.io/cluster-service"

# Whether only one addon manager should be running in a multi-master setup.
# Disabling this flag will force all addon managers to assume they are the
# leaders.
ADDON_MANAGER_LEADER_ELECTION=${ADDON_MANAGER_LEADER_ELECTION:-true}

# Remember that you can't log from functions that print some output (because
# logs are also printed on stdout).
# $1 level
Expand Down Expand Up @@ -140,6 +145,25 @@ function ensure_addons() {
log INFO "== Kubernetes addon ensure completed at $(date -Is) =="
}

function is_leader() {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This real pokes around at the internals. We want to start moving the addon manager to use the front door. Can you look at the controller-manager leader election result instead? cc @tallclair

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The leader election results for kube-controller-manager are stored on it's endpoints object:

kubectl -n kube-system get ep kube-controller-manager -o json

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for pointing this one! Done.

# In multi-master setup, only one addon manager should be running. We use
# existing leader election in kube-controller-manager instead of implementing
# a separate mechanism here.
if ! $ADDON_MANAGER_LEADER_ELECTION; then
log INFO "Leader election disabled."
return 0;
fi
KUBE_CONTROLLER_MANAGER_LEADER=`${KUBECTL} -n kube-system get ep kube-controller-manager \
-o go-template=$'{{index .metadata.annotations "control-plane.alpha.kubernetes.io/leader"}}' \
| sed 's/^.*"holderIdentity":"\([^"]*\)".*/\1/'`
# If there was any problem with getting the leader election results, var will
# be empty. Since it's better to have multiple addon managers than no addon
# managers at all, we're going to assume that we're the leader in such case.
log INFO "Leader is $KUBE_CONTROLLER_MANAGER_LEADER"
[[ "$KUBE_CONTROLLER_MANAGER_LEADER" == "" ||
"$HOSTNAME" == "$KUBE_CONTROLLER_MANAGER_LEADER" ]]
}

# The business logic for whether a given object should be created
# was already enforced by salt, and /etc/kubernetes/addons is the
# managed result is of that. Start everything below that directory.
Expand Down Expand Up @@ -175,8 +199,12 @@ done
log INFO "== Entering periodical apply loop at $(date -Is) =="
while true; do
start_sec=$(date +"%s")
ensure_addons
reconcile_addons
if is_leader; then
ensure_addons
reconcile_addons
else
log INFO "Not elected leader, going back to sleep."
fi
end_sec=$(date +"%s")
len_sec=$((${end_sec}-${start_sec}))
# subtract the time passed from the sleep time
Expand Down
5 changes: 5 additions & 0 deletions cluster/common.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -882,6 +882,11 @@ EOF
if [[ "${NODE_ACCELERATORS:-}" == *"type=nvidia"* ]]; then
cat >>$file <<EOF
ENABLE_NVIDIA_GPU_DEVICE_PLUGIN: $(yaml-quote "true")
EOF
fi
if [ -n "${ADDON_MANAGER_LEADER_ELECTION:-}" ]; then
cat >>$file <<EOF
ADDON_MANAGER_LEADER_ELECTION: $(yaml-quote ${ADDON_MANAGER_LEADER_ELECTION})
EOF
fi

Expand Down