kubeadm: add final fallback to constants.CurrentKubernetesVersion #72454

rojkov · 2018-12-31T10:14:15Z

What this PR does / why we need it:

It may happen that both the git version and the remote version
are broken/inaccessible. In this case the broken remote version
would be used.

To overcome this situation fall back to the constant CurrentKubernetesVersion.

The alternative could be os.Exit(1).

Also this change fixes bazel-based unit tests in air-gapped environment.

What type of PR is this?

/kind cleanup
/kind failing-test

Does this PR introduce a user-facing change?:

In case kubeadm can't access the current Kubernetes version remotely and fails to parse
the git-based version it falls back to a static predefined value of
k8s.io/kubernetes/cmd/kubeadm/app/constants.CurrentKubernetesVersion.

k8s-ci-robot · 2018-12-31T10:14:22Z

Hi @rojkov. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

rojkov · 2018-12-31T10:15:22Z

/cc @rosti
/cc @neolit123
/cc @bart0sh

rojkov · 2018-12-31T10:21:02Z

/kind cleanup
/kind failing-test

bart0sh · 2018-12-31T11:05:57Z

/ok-to-test
/priority important-longterm

liggitt · 2018-12-31T16:44:30Z

cmd/kubeadm/app/util/version.go

 			klog.Warningf("could not obtain client version; using remote version: %s", body)
 			return KubernetesReleaseVersion(body)
 		}

+		if err != nil && clientVersionErr != nil {
+			klog.Warningf("could not obtain neither client nor remote version; fall back to: %s", constants.CurrentKubernetesVersion)
+			return KubernetesReleaseVersion(constants.CurrentKubernetesVersion.String())


is there a unit test to ensure this constant is always correct? if not, it is very likely to drift and be forgotten

after the resent refactor these should be the only constants that we bump on each minor release and we must bump them anyway:
https://github.com/kubernetes/kubernetes/blob/b500a95af34a8babf74191a246b5dae0fd764e56/cmd/kubeadm/app/constants/constants.go#L379-L386

having a unit test puts this in a weird space - i.e. to which value to compare the constant to?
the constant is a fallback: remote version -> client version -> constant version.

we can potentially compare to the remote against https://dl.k8s.io/release/stable-1.txt (with patch=0), but arguably unit tests should not depend on remote connectivity....we do that already in a few places.

compare to the client version. problem here is that unit tests do not receive the version as ldflags.
i think this might be a problem only for the integration tests:
https://github.com/kubernetes/kubernetes/blob/master/hack/make-rules/test-kubeadm-cmd.sh

leaving the final call to @timothysc and @kad on this PR.
/assign @timothysc
/assign @kad

these should be the only constants that we bump on each minor release and we must bump them anyway

Doesn't the current release constant require bumping on every patch version as well?

CurrentKubernetesVersion parses to a MAJOR.MINOR.PATCH object, but the important elements are MAJOR.MINOR.

That seems like a misleading constant, then, and one that would be likely to be used incorrectly.

Honestly speaking this recursive function with side effects smells suboptimal to me since it clearly violates the Single Responsibility principle. I would rather try to constrain ClusterConfig.KubernetesVersion to be a canonical version only and always. Then at config parsing/cmdline option setting time we'd detect whether the provided value is

a version or

just a version label supposed to be resolved remotely into a version or

nil.

In the first case normalizedBuildVersion() would be used once. In the second case KubernetesReleaseVersion() would be limited to resolving the remote version only. And in the third case we could try to read the gitVersion and fail if it's not set.

Then the only authoritative source of the version value would be ClusterConfig.KubernetesVersion. With this approach the unit tests would not touch networking.

Honestly speaking this recursive function with side effects smells suboptimal to me since it clearly violates the Single Responsibility principle

i support your concern here and the recursion can be avoided.
@kad and @luxas probably have opinions here as they authored the original?

I would rather try to constrain ClusterConfig.KubernetesVersion to be a canonical version only and always

isn't this the case already? if the user does not feed a KubernetesVersion and a config, the version has to come from somewhere. internet endpoint, client version, constant?

With this approach the unit tests would not touch networking.

there are calls like this one:

kubernetes/cmd/kubeadm/app/cmd/token.go

Lines 213 to 215 in b502b99

// KubernetesVersion is not used, but we set it explicitly to avoid the lookup

// of the version from the internet when executing ConfigFileAndDefaultsToInternalConfig

phaseutil.SetKubernetesVersion(cfg)

that pre-set a version so that the version is later not fetched from the internet for commands that don't need internet. but this still needs a client that was built properly.

so potentially, unit tests can do the same.

isn't this the case already? if the user does not feed a KubernetesVersion and a config, the version has to come from somewhere. internet endpoint, client version, constant?

IIRC some unit tests don't (can't) consult ClusterConfig.KubernetesVersion relying solely on KubernetesReleaseVersion(). I'll try to come up with a proposal.

... in a separate PR that is.

neolit123 · 2019-01-01T10:05:45Z

cmd/kubeadm/app/util/version.go

 			klog.Warningf("could not obtain client version; using remote version: %s", body)
 			return KubernetesReleaseVersion(body)
 		}

+		if err != nil && clientVersionErr != nil {


why not unify the err != nil and err == nil branches under a clientVersionErr != nil branch?

I moved the branches under clientVersionErr != nil.

neolit123 · 2019-01-01T10:25:29Z

cmd/kubeadm/app/util/version.go

 			klog.Warningf("could not obtain client version; using remote version: %s", body)
 			return KubernetesReleaseVersion(body)
 		}

+		if err != nil && clientVersionErr != nil {
+			klog.Warningf("could not obtain neither client nor remote version; fall back to: %s", constants.CurrentKubernetesVersion)
+			return KubernetesReleaseVersion(constants.CurrentKubernetesVersion.String())


after the resent refactor these should be the only constants that we bump on each minor release and we must bump them anyway:
https://github.com/kubernetes/kubernetes/blob/b500a95af34a8babf74191a246b5dae0fd764e56/cmd/kubeadm/app/constants/constants.go#L379-L386

having a unit test puts this in a weird space - i.e. to which value to compare the constant to?
the constant is a fallback: remote version -> client version -> constant version.

we can potentially compare to the remote against https://dl.k8s.io/release/stable-1.txt (with patch=0), but arguably unit tests should not depend on remote connectivity....we do that already in a few places.

compare to the client version. problem here is that unit tests do not receive the version as ldflags.
i think this might be a problem only for the integration tests:
https://github.com/kubernetes/kubernetes/blob/master/hack/make-rules/test-kubeadm-cmd.sh

leaving the final call to @timothysc and @kad on this PR.
/assign @timothysc
/assign @kad

rojkov · 2019-01-02T16:00:40Z

/test pull-kubernetes-integration

kad · 2019-01-11T12:00:23Z

/lgtm
for this PR.
As for recursion: at the time this function was written, it had single responsibility of resolving label to exact validated version string, with support for label->label->..->version resolving (see tests for this function). Later additions with different types of fallbacks complicated it unfortunately.

kad · 2019-01-11T12:01:49Z

BTW, as this has user visible behavior change (static default fallback version), it would be good to have release note line.

rojkov · 2019-01-11T12:38:04Z

BTW, as this has user visible behavior change (static default fallback version), it would be good to have release note line.

Thanks! I've updated the release note.

rosti · 2019-01-17T09:53:00Z

For feedback and approval
/assign @neolit123 @fabriziopandini

rosti · 2019-01-17T10:05:46Z

cmd/kubeadm/app/util/version.go

-			return KubernetesReleaseVersion(clientVersion)
+			if clientVersionErr == nil {
+				// Handle air-gapped environments by falling back to the client version.
+				klog.Infof("could not fetch a Kubernetes version from the internet: %v", err)


Technically speaking, this one should be a warning.

Agree. Made them warnings now.

It may happen that both the git version and the remote version are broken/inaccessible. In this case the broken remote version would be used. To overcome this situation fall back to the constant CurrentKubernetesVersion. The alternative could be os.Exit(1). Also this change fixes Bazel-based unit tests in air-gapped environment.

fabriziopandini

@rojkov first of all sorry for the delay in reviewing this PR.

As per slack discussion, I understood that adding a third fallback rule for version discovery is a necessary evil in for following us cases:

git version is broken when building unit tests with Bazel behind proxy (Bazel drops http_proxy= vars)
running a build from the source tarball (not from git) in an air gaped environment

The proposed fallback method is not perfect, because it will always point at the .0 patch of each minor, but I think it is acceptable for the above use cases (running unit tests).

However, in my humble opinion, this function is already too complicated and I think that we cannot insert additional complexity without cleaning up some technical debt.
What do you think about the following proposal?

KubernetesReleaseVersion should be given the only the responsibility to coordinate the fallback logic, something similar to

getKubernetesReleaseVersion (from internet)
if ok return 

getGitVersion
if ok return

use constant

getKubernetesReleaseVersion should have the single responsibility of resolving label to exact validated version string, with support for label->label->..->version resolving (as per @kad comment)

The expected benefit of the above proposal is to spilt the fallback logic from the label resolving/recursive logic, thus hopefully improving code readability/maintenabily/testability.

I'm going also to open a separated issue for improving the name of CurrentKubernetesVersion as per @liggitt comment

rojkov · 2019-01-20T09:10:26Z

@fabriziopandini Thank you for the review! Actually I've already rewritten the function in this commit rojkov@d991754

    kubeadm: refactor Kubernetes release version calculations
    
    Move version label resolution logic to the dedicated
    function `resolveVersionLabel()`. This way we
    
    1. get rid of recursion in `KubernetesReleseVersion()`;
    2. guaranty that the function never falls into endless
       loops in case the server mistakenly resolves "stable"
       into "stable-1" and "stable-1" back into "stable".
    3. get simpler code in `KubermetesReleaseVersion`.
    
    Also it renames `normalizedBuildVersion()` to
    `normilizedBuildVersion()` to make it sound like a verb.
    Now the function clearly indicates errors instead of making
    a developer analize its output.
    
    Detection of air-gapped environment is done through examining
    http.Client's error rather than output of `fetchFromURL()` which
    depends the function's internal logic.

Basically it implements your suggestion already. I wanted to rebase it on top of this PR, but I can do the other way around.

rojkov · 2019-01-23T08:28:07Z

Submitted #73129 which is pending @kad 's review. Then I'll rebase this PR.

fabriziopandini · 2019-01-23T13:34:01Z

/hold
Waiting for #73129 to land before

kad · 2019-02-12T17:54:49Z

@fabriziopandini why to hold this one? I think it can be merged.

timothysc · 2019-03-27T16:46:06Z

/hold cancel

timothysc · 2019-03-27T16:46:16Z

/lgtm
/approve

k8s-ci-robot · 2019-03-27T16:46:56Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rojkov, timothysc

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~cmd/kubeadm/OWNERS~~ [timothysc]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Dec 31, 2018

k8s-ci-robot requested review from dixudx and luxas December 31, 2018 10:14

k8s-ci-robot added area/kubeadm sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Dec 31, 2018

k8s-ci-robot requested review from bart0sh, neolit123 and rosti December 31, 2018 10:15

liggitt reviewed Dec 31, 2018

View reviewed changes

neolit123 reviewed Jan 1, 2019

View reviewed changes

k8s-ci-robot assigned kad and timothysc Jan 1, 2019

neolit123 mentioned this pull request Jan 1, 2019

Simplify kubeadm minimum Kubernetes version bumps kubernetes/kubeadm#1260

Closed

rojkov force-pushed the kubeadm-fallback-version branch from b500a95 to 8ead090 Compare January 2, 2019 13:34

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 11, 2019

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed release-note-none Denotes a PR that doesn't merit a release note. labels Jan 11, 2019

k8s-ci-robot assigned fabriziopandini and neolit123 Jan 17, 2019

rosti reviewed Jan 17, 2019

View reviewed changes

rojkov force-pushed the kubeadm-fallback-version branch from 8ead090 to 9e25a00 Compare January 17, 2019 15:33

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 17, 2019

fabriziopandini suggested changes Jan 20, 2019

View reviewed changes

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 23, 2019

fabriziopandini mentioned this pull request Feb 6, 2019

kubeadm: refactor Kubernetes release version calculations #73129

Closed

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 27, 2019

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 27, 2019

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 27, 2019

k8s-ci-robot merged commit 5a6c66a into kubernetes:master Mar 27, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kubeadm: add final fallback to constants.CurrentKubernetesVersion #72454

kubeadm: add final fallback to constants.CurrentKubernetesVersion #72454

rojkov commented Dec 31, 2018 •

edited

k8s-ci-robot commented Dec 31, 2018

rojkov commented Dec 31, 2018

rojkov commented Dec 31, 2018

bart0sh commented Dec 31, 2018

liggitt Dec 31, 2018

neolit123 Jan 1, 2019

liggitt Jan 1, 2019

neolit123 Jan 1, 2019

liggitt Jan 1, 2019

rojkov Jan 2, 2019

neolit123 Jan 2, 2019

rojkov Jan 2, 2019

rojkov Jan 2, 2019

neolit123 Jan 1, 2019

rojkov Jan 2, 2019

neolit123 Jan 1, 2019

rojkov commented Jan 2, 2019

kad commented Jan 11, 2019

kad commented Jan 11, 2019

rojkov commented Jan 11, 2019

rosti commented Jan 17, 2019

rosti Jan 17, 2019 •

edited

rojkov Jan 17, 2019

fabriziopandini left a comment

rojkov commented Jan 20, 2019

rojkov commented Jan 23, 2019

fabriziopandini commented Jan 23, 2019

kad commented Feb 12, 2019

timothysc commented Mar 27, 2019

timothysc commented Mar 27, 2019

k8s-ci-robot commented Mar 27, 2019

	// KubernetesVersion is not used, but we set it explicitly to avoid the lookup
	// of the version from the internet when executing ConfigFileAndDefaultsToInternalConfig
	phaseutil.SetKubernetesVersion(cfg)

kubeadm: add final fallback to constants.CurrentKubernetesVersion #72454

kubeadm: add final fallback to constants.CurrentKubernetesVersion #72454

Conversation

rojkov commented Dec 31, 2018 • edited

k8s-ci-robot commented Dec 31, 2018

rojkov commented Dec 31, 2018

rojkov commented Dec 31, 2018

bart0sh commented Dec 31, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rojkov commented Jan 2, 2019

kad commented Jan 11, 2019

kad commented Jan 11, 2019

rojkov commented Jan 11, 2019

rosti commented Jan 17, 2019

rosti Jan 17, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fabriziopandini left a comment

Choose a reason for hiding this comment

rojkov commented Jan 20, 2019

rojkov commented Jan 23, 2019

fabriziopandini commented Jan 23, 2019

kad commented Feb 12, 2019

timothysc commented Mar 27, 2019

timothysc commented Mar 27, 2019

k8s-ci-robot commented Mar 27, 2019

rojkov commented Dec 31, 2018 •

edited

rosti Jan 17, 2019 •

edited