Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gce: Prefer MASTER_ADVERTISE_ADDRESS in apiserver setup #63696

Merged
merged 1 commit into from May 16, 2018

Conversation

Projects
None yet
5 participants
@grosskur
Copy link
Contributor

grosskur commented May 11, 2018

MASTER_ADVERTISE_ADDRESS is used to set the --advertise-address flag
for the apiserver. It's useful for running the apiserver behind a load
balancer.

However, if PROJECT_ID, TOKEN_URL, TOKEN_BODY, and NODE_NETWORK are
all set, the GCE VM's external IP address will be fetched and used
instead and MASTER_ADVERTISE_ADDRESS will be ignored.

Change this behavior so that MASTER_ADVERTISE_ADDRESS takes precedence
because it's more specific. We still fall back to using the VM's
external IP address if the other variables are set.

Also: Move the setting of --ssh-user and --ssh-keyfile based on
PROXY_SSH_USER) to a top-level block because this is common to all
codepaths.

GCE: Fix to make the built-in `kubernetes` service properly point to the master's load balancer address in clusters that use multiple master VMs.
@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented May 11, 2018

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.


Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@grosskur

This comment has been minimized.

Copy link
Contributor Author

grosskur commented May 11, 2018

I've now signed the CNCF CLA.

@bowei

This comment has been minimized.

Copy link
Member

bowei commented May 13, 2018

/ok-to-test

@bowei

This comment has been minimized.

Copy link
Member

bowei commented May 13, 2018

/assign

@bowei

This comment has been minimized.

Copy link
Member

bowei commented May 13, 2018

/assign @dnardo

params+=" --advertise-address=${MASTER_ADVERTISE_ADDRESS}"
elif [[ -n "${PROJECT_ID:-}" && -n "${TOKEN_URL:-}" && -n "${TOKEN_BODY:-}" && -n "${NODE_NETWORK:-}" ]]; then
local -r vm_external_ip=$(curl --retry 5 --retry-delay 3 ${CURL_RETRY_CONNREFUSED} --fail --silent -H 'Metadata-Flavor: Google' "http://metadata/computeMetadata/v1/instance/network-interfaces/0/access-configs/0/external-ip")
params+=" --advertise-address=${vm_external_ip}"

This comment has been minimized.

@bowei

bowei May 13, 2018

Member

Seems like this was set only if PROXY_SSH_USER is set in the previous code. Was that wrong?

This comment has been minimized.

@grosskur

grosskur May 15, 2018

Author Contributor

Not sure. I looked at the git history but couldn't figure out why --advertise-address was only set if PROXY_SSH_USER was set. To be safe, I've updated the diff to leave that entire block as-is.

fi
elif [ -n "${MASTER_ADVERTISE_ADDRESS:-}" ]; then
params="${params} --advertise-address=${MASTER_ADVERTISE_ADDRESS}"
if [ -n "${MASTER_ADVERTISE_ADDRESS:-}" ]; then

This comment has been minimized.

@bowei

bowei May 13, 2018

Member

use the double bracket test to be consistent (e.g. "[[" and "]]")

This comment has been minimized.

@grosskur

grosskur May 15, 2018

Author Contributor

Done.

if [ -n "${MASTER_ADVERTISE_ADDRESS:-}" ]; then
params+=" --advertise-address=${MASTER_ADVERTISE_ADDRESS}"
elif [[ -n "${PROJECT_ID:-}" && -n "${TOKEN_URL:-}" && -n "${TOKEN_BODY:-}" && -n "${NODE_NETWORK:-}" ]]; then
local -r vm_external_ip=$(curl --retry 5 --retry-delay 3 ${CURL_RETRY_CONNREFUSED} --fail --silent -H 'Metadata-Flavor: Google' "http://metadata/computeMetadata/v1/instance/network-interfaces/0/access-configs/0/external-ip")

This comment has been minimized.

@bowei

bowei May 13, 2018

Member

why not use get-metadata-value?

This comment has been minimized.

@grosskur

grosskur May 15, 2018

Author Contributor

Whoops. I initially developed this patch on an older branch that didn't have get-metadata-value. Switched to use get-metadata-value now.

@dnardo

This comment has been minimized.

Copy link
Contributor

dnardo commented May 14, 2018

/lgtm other than Bowei's comments.

Was kubeclt logs/exec tested with this change (in all cases) ?

@grosskur grosskur force-pushed the grosskur:gce-advertise-addr branch from 56188ef to 1433d72 May 15, 2018

@grosskur

This comment has been minimized.

Copy link
Contributor Author

grosskur commented May 15, 2018

Thanks for the review!

@bowei: I've pushed a new diff that addresses your comments.

@dnardo: I've tested both kubectl exec and kubectl logs in all cases now, and verified they still work.

@grosskur

This comment has been minimized.

Copy link
Contributor Author

grosskur commented May 15, 2018

/retest

gce: Prefer MASTER_ADVERTISE_ADDRESS in apiserver setup
MASTER_ADVERTISE_ADDRESS is used to set the --advertise-address flag
for the apiserver. It's useful for running the apiserver behind a load
balancer.

However, if PROJECT_ID, TOKEN_URL, TOKEN_BODY, and NODE_NETWORK are
all set, the GCE VM's external IP address will be fetched and used
instead and MASTER_ADVERTISE_ADDRESS will be ignored.

Change this behavior so that MASTER_ADVERTISE_ADDRESS takes precedence
because it's more specific. We still fall back to using the VM's
external IP address if the other variables are set.

Also: Pass --ssh-user and --ssh-keyfile flags if both PROXY_SSH_USER
and MASTER_ADVERTISE_ADDRESS is set.

@grosskur grosskur force-pushed the grosskur:gce-advertise-addr branch from 1433d72 to 3541a93 May 16, 2018

@grosskur

This comment has been minimized.

Copy link
Contributor Author

grosskur commented May 16, 2018

/test pull-kubernetes-integration

@bowei

This comment has been minimized.

Copy link
Member

bowei commented May 16, 2018

/approve no-issue

@bowei

This comment has been minimized.

Copy link
Member

bowei commented May 16, 2018

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm label May 16, 2018

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented May 16, 2018

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bowei, dnardo, grosskur

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-github-robot

This comment has been minimized.

Copy link
Contributor

k8s-github-robot commented May 16, 2018

/test all [submit-queue is verifying that this PR is safe to merge]

@k8s-github-robot

This comment has been minimized.

Copy link
Contributor

k8s-github-robot commented May 16, 2018

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions here.

@k8s-github-robot k8s-github-robot merged commit e392f5b into kubernetes:master May 16, 2018

15 of 18 checks passed

Submit Queue Required Github CI test is not green: pull-kubernetes-e2e-gce
Details
pull-kubernetes-e2e-gce-100-performance Job triggered.
Details
pull-kubernetes-kubemark-e2e-gce-big Job triggered.
Details
cla/linuxfoundation grosskur authorized
Details
pull-kubernetes-bazel-build Job succeeded.
Details
pull-kubernetes-bazel-test Job succeeded.
Details
pull-kubernetes-cross Skipped
pull-kubernetes-e2e-gce Job succeeded.
Details
pull-kubernetes-e2e-gce-device-plugin-gpu Job succeeded.
Details
pull-kubernetes-e2e-gke Job succeeded.
Details
pull-kubernetes-e2e-kops-aws Job succeeded.
Details
pull-kubernetes-integration Job succeeded.
Details
pull-kubernetes-kubemark-e2e-gce Job succeeded.
Details
pull-kubernetes-local-e2e Skipped
pull-kubernetes-local-e2e-containerized Skipped
pull-kubernetes-node-e2e Job succeeded.
Details
pull-kubernetes-typecheck Job succeeded.
Details
pull-kubernetes-verify Job succeeded.
Details

k8s-github-robot pushed a commit that referenced this pull request May 22, 2018

Kubernetes Submit Queue
Merge pull request #63998 from grosskur/automated-cherry-pick-of-#636…
…96-upstream-release-1.8

Automatic merge from submit-queue.

Automated cherry pick of #63696: gce: Prefer MASTER_ADVERTISE_ADDRESS in apiserver setup

Cherry pick of #63696 on release-1.8.

#63696: gce: Prefer MASTER_ADVERTISE_ADDRESS in apiserver setup

```release-note
NONE
```

k8s-github-robot pushed a commit that referenced this pull request May 25, 2018

Kubernetes Submit Queue
Merge pull request #63996 from grosskur/automated-cherry-pick-of-#636…
…96-upstream-release-1.10

Automatic merge from submit-queue.

Automated cherry pick of #63696: gce: Prefer MASTER_ADVERTISE_ADDRESS in apiserver setup

Cherry pick of #63696 on release-1.10.

#63696: gce: Prefer MASTER_ADVERTISE_ADDRESS in apiserver setup

```release-note
NONE
```

k8s-github-robot pushed a commit that referenced this pull request May 30, 2018

Kubernetes Submit Queue
Merge pull request #63997 from grosskur/automated-cherry-pick-of-#636…
…96-upstream-release-1.9

Automatic merge from submit-queue.

Automated cherry pick of #63696: gce: Prefer MASTER_ADVERTISE_ADDRESS in apiserver setup

Cherry pick of #63696 on release-1.9.

#63696: gce: Prefer MASTER_ADVERTISE_ADDRESS in apiserver setup

```release-note
NONE
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.