kubeadm: chroot to new --rootfs arg #54935

anguslees · 2017-11-01T12:30:17Z

What this PR does / why we need it:

This change adds a new --rootfs=path option to kubeadm, and (if
provided) chroot()s to this path before performing file operations.

This makes it possible to run the kubeadm binary from a container, but
perform remaining file operations against the host filesystem using
something like:

    docker run -v /:/rootfs --net=host --uts=host --pid=host \
       kubeadm:latest init ...

(Assuming something like the included examples/kubeadm/Dockerfile which sets CMD to kubeadm --rootfs=/rootfs - Edit: Dockerfile has been removed from this PR, but you get the idea)

Fixes kubernetes/kubeadm#503

Special notes for your reviewer:

I'm not sure where is best to put the Dockerfile, or hook it up to the build process. Advice sought.
The kubeadm command line arg handling was less unified than I was expecting to find. I've implemented this arg for init and join. I can add it to all the others too, if we're happy with the approach. An alternative would be to add the arg in the parent KubeadmCommand, possibly with a PersistantFlag - then it would automatically exist for all kubeadm subcommands.
It would be slightly preferable if we could order --rootfs before the subcommand so we could apply the arg automatically with ENTRYPOINT ["kubeadm", "--rootfs=/rootfs"]. This would be the only such flag in kubeadm however, so I have not implemented it that way atm. (Another alternative would be an env var)

Release note:

Adds a new EXPERIMENTAL `--rootfs` flag to kubeadm, which (if specified) causes kubeadm to chroot before performing any file operations.  This is expected to be useful when setting up kubernetes on a different filesystem, such as invoking kubeadm from docker.

kad

what about cmd/kubeadm/app/phases/* ?

kad · 2017-11-01T16:23:31Z

cmd/kubeadm/app/apis/kubeadm/v1alpha1/types.go

@@ -34,6 +34,7 @@ type MasterConfiguration struct {
 	CloudProvider      string     `json:"cloudProvider"`
 	NodeName           string     `json:"nodeName"`
 	AuthorizationModes []string   `json:"authorizationModes,omitempty"`
+	Rootfs             string     `json:"rootfs"`


do we really need to expose it here ?

Oh I see, I didn't realise how this differed from the other ../types.go. No, we don't need it exposed in this (persisted) version.

kad · 2017-11-01T16:24:01Z

cmd/kubeadm/app/apis/kubeadm/v1alpha1/types.go

@@ -103,6 +104,7 @@ type NodeConfiguration struct {
 	NodeName                 string   `json:"nodeName"`
 	TLSBootstrapToken        string   `json:"tlsBootstrapToken"`
 	Token                    string   `json:"token"`
+	Rootfs                   string   `json:"rootfs"`


same as above.

kad · 2017-11-01T16:30:31Z

For new flags, it would be good also to have release note.

anguslees · 2017-11-02T00:12:42Z

@kad: do you want me to duplicate this code for all these subcommands (and phases)? That seems to be the current pattern, but if I were writing kubeadm I would instead lift this (and perhaps other common options) up to a single PersistentFlags declaration and move the runtime logic up to the root cobra.Command's Run method (ie: don't just use cmdutil.SubCmdRunE by itself).

I'm happy to do the cut+paste-everywhere version, I just want to be sure that is the direction you want me to go before doing it.

kad · 2017-11-02T06:58:11Z

About moving to root cobra.Command I have no strong opinion. Most probably it will be better, but I haven't investigated that in details.

My point was purely about functionality: if we introduce functionality in some main commands (join/init), we should be able to do the same consistently in phases (if kubeadm is used as part of more complex solution which executes phases step-by-step) and in other main commands (reset, upgrade).

php-coder · 2017-11-06T15:25:13Z

cmd/kubeadm/app/cmd/init.go

+		if err := syscall.Chroot(cfg.Rootfs); err != nil {
+			return nil, fmt.Errorf("unable to chroot to %s: %v", cfg.Rootfs, err)
+		}
+		// NB: All file paths after here are relative to Rootfs


Note that chroot "does not change the current working directory, so that after the call '.' can be outside the tree rooted at '/'." See more info in man 2 chroot. It's advised to always explicitly invoke syscall.Chdir before or after the chroot.

Good call - I should not just rely on kubeadm using only absolute paths. I'll add an explicit chdir(/).

anguslees · 2017-11-13T05:55:11Z

PTAL.

Moved into a PersistentPreRun on the top level kubeadm command, so it now automatically applies to all current (and future) subcommands. It also makes the argument order more suitable for Dockerfile CMD (before the subcommand) - so now the example Dockerfile just automatically applies the --rootfs=/rootfs arg.

Note that the chroot happens before reading the config file - so the --config path is on the host rootfs, not inside the docker image (for the docker use-case example).

anguslees · 2017-11-23T04:47:52Z

(Given the release freeze, I'm going to wait until I get a review rather than just sit here rebasing this every day. Please don't misinterpret an outstanding conflict as an indication that I no longer care about this PR.)

kad · 2017-11-23T09:18:11Z

examples/kubeadm/Dockerfile

+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.


Maybe add here as a comment at least some "usage" for this image ? with example of docker run -it -v /:/rootfs kubeadm init ... /

Done. Let me know if it doesn't makes sense.

kad · 2017-11-23T11:30:29Z

In overall, looks good to me. @anguslees have you thought about looking at creating that kubeadm image as part of release artefacts ? If build scripts will be building this image and populating GCR at the same time as any other release images, it might be easier method for users to get kubeadm.

anguslees · 2017-11-23T23:17:52Z

If build scripts will be building this image and populating GCR at the same time as any other release images, it might be easier method for users to get kubeadm.

Agreed 100% and that is my desire. As I said in the original PR comment, I have no idea how to go about getting this added/moved to the release images. Any pointers? (to docs or people/channels where I can ask again?)

kad · 2017-11-23T23:59:27Z

I think first place to look at is: build/lib/release.sh, somewhere around function kube::release::create_docker_images_for_server(). But then there is bazel build, and this probably better to ask on #bazel channel in Slack.

errordeveloper · 2018-01-09T12:15:34Z

@anguslees is this still of interest? If yes, please rebase.

kad · 2018-08-08T09:09:20Z

@anguslees I agree, all commands that require reading/writing any file should have --rootfs flag to be consistent in expected behaviour. Otherwise this PR looks good for me.

neolit123 · 2018-08-08T12:08:32Z

@anguslees @kad
thank you for having a closer look at the commands that may or many not need --rootfs. i also think this PR is good to go now and it's not a big deal that we have commands that don't really benefit from the global flag right now.

thanks for the updates @anguslees and sorry for going back and forth with the reviews and also for this PR collecting dust for so long. i can do any further amends myself if you'd prefer.

/lgtm
pinging @timothysc @luxas @fabriziopandini for review/approval.

neolit123 · 2018-08-08T12:09:07Z

/test pull-kubernetes-e2e-kops-aws

anguslees · 2018-08-15T03:33:09Z

/test all

anguslees · 2018-08-15T05:38:39Z

/retest

neolit123 · 2018-08-16T00:03:27Z

/assign @fabriziopandini
please approve if you get a minute, we can take care of later adjustments to this, but this PR is since Nov 2017 and i think we should get it merged. :\

timothysc

This is useful, but there are weird side effects that may occur on chroots. I'm ok with this so long as --experiment-rootfs is given

/assign @timothysc

anguslees · 2018-08-23T01:16:22Z

/retest

anguslees · 2018-08-23T01:26:31Z

I'm ok with this so long as --experiment-rootfs is given

To be clear: Are you saying the change is ok as-is, or you want me to rename the flag to --experiment-rootfs?

(You github-approved the change, but without an "/approve" for the k8s-bots to see)

neolit123 · 2018-08-23T01:29:24Z

@anguslees @timothysc
i guess we can use --experimental-rootfs, but if decide to rename the flag this would break the user CLI.
maybe we can keep it --rootfs, but instead add the notes that this is experimental in both the release note and flag description:

Add the EXPERIMENTAL --rootfs flag to kubeadm, which (if specified) causes kubeadm to chroot before performing any file operations. This is expected to be useful when setting up kubernetes on a different filesystem, such as invoking kubeadm from docker.

"[EXPERIMENTAL] The path to the 'real' host root filesystem.",

WDYT?

anguslees · 2018-08-24T08:53:56Z

WDYT?

Happy to type whatever the latest approver wants to see, just need to know what that is.

timothysc · 2018-08-24T16:59:43Z

+1 to "[EXPERIMENTAL] The path to the 'real' host root filesystem.",

neolit123 · 2018-08-25T01:30:23Z

@anguslees ^^

also:

Add the EXPERIMENTAL --rootfs...

in the release note will be nice to have.
thanks

anguslees · 2018-08-27T02:37:28Z

I have prepended [EXPERIMENTAL] to the option help text.

neolit123 · 2018-08-27T12:04:37Z

@anguslees
thank you. please squash the commits.

timothysc

/lgtm
/approve

k8s-ci-robot · 2018-08-27T14:25:38Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: anguslees, neolit123, timothysc

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~cmd/kubeadm/OWNERS~~ [timothysc]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

timothysc · 2018-08-27T14:25:51Z

/test pull-kubernetes-e2e-kops-aws

k8s-github-robot · 2018-08-27T16:05:13Z

/test all [submit-queue is verifying that this PR is safe to merge]

k8s-github-robot · 2018-08-27T17:33:45Z

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions here.

k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Nov 1, 2017

k8s-github-robot assigned errordeveloper and justinsb Nov 1, 2017

kad reviewed Nov 1, 2017

View reviewed changes

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed release-note-none Denotes a PR that doesn't merit a release note. labels Nov 2, 2017

php-coder reviewed Nov 6, 2017

View reviewed changes

anguslees force-pushed the kubeadm-chroot branch from c32f91f to 373edad Compare November 13, 2017 05:50

anguslees force-pushed the kubeadm-chroot branch 2 times, most recently from 69056be to eb1fe7f Compare November 13, 2017 07:16

k8s-github-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 20, 2017

anguslees force-pushed the kubeadm-chroot branch from eb1fe7f to 93f2a81 Compare November 20, 2017 11:37

k8s-github-robot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Nov 20, 2017

kad reviewed Nov 23, 2017

View reviewed changes

anguslees force-pushed the kubeadm-chroot branch 2 times, most recently from f4bb01b to d134986 Compare November 23, 2017 11:11

k8s-github-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 23, 2017

k8s-github-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 30, 2017

k8s-ci-robot assigned neolit123 Aug 8, 2018

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 8, 2018

k8s-ci-robot assigned fabriziopandini Aug 16, 2018

k8s-ci-robot assigned timothysc Aug 23, 2018

timothysc approved these changes Aug 23, 2018

View reviewed changes

Add "EXPERIMENTAL" to the option description

7e77124

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 27, 2018

timothysc approved these changes Aug 27, 2018

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 27, 2018

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 27, 2018

k8s-github-robot merged commit 74d513f into kubernetes:master Aug 27, 2018

anguslees mentioned this pull request Feb 21, 2019

Deprecate "containerized kubelet" #74148

Closed

kubeadm: chroot to new --rootfs arg #54935

kubeadm: chroot to new --rootfs arg #54935

Conversation

anguslees commented Nov 1, 2017 • edited Loading

kad left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kad commented Nov 1, 2017

anguslees commented Nov 2, 2017 • edited Loading

kad commented Nov 2, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anguslees commented Nov 13, 2017 • edited Loading

anguslees commented Nov 23, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kad commented Nov 23, 2017

anguslees commented Nov 23, 2017

kad commented Nov 23, 2017

errordeveloper commented Jan 9, 2018

kad commented Aug 8, 2018

neolit123 commented Aug 8, 2018

neolit123 commented Aug 8, 2018

anguslees commented Aug 15, 2018

anguslees commented Aug 15, 2018

neolit123 commented Aug 16, 2018

timothysc left a comment

Choose a reason for hiding this comment

anguslees commented Aug 23, 2018

anguslees commented Aug 23, 2018

neolit123 commented Aug 23, 2018

anguslees commented Aug 24, 2018

timothysc commented Aug 24, 2018

neolit123 commented Aug 25, 2018

anguslees commented Aug 27, 2018

neolit123 commented Aug 27, 2018

timothysc left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Aug 27, 2018

timothysc commented Aug 27, 2018

k8s-github-robot commented Aug 27, 2018

k8s-github-robot commented Aug 27, 2018

anguslees commented Nov 1, 2017 •

edited

Loading

anguslees commented Nov 2, 2017 •

edited

Loading

anguslees commented Nov 13, 2017 •

edited

Loading

anguslees commented Nov 23, 2017 •

edited

Loading