add kubectl cp #34914

brendandburns · 2016-10-16T23:46:59Z

Implements kubectl cp (#13776)

Syntax examples:

# Copy from pod to local machine
$ kubectl cp [namespace/]pod:/some/file/or/dir ./some/local/file/or/dir

# Copy from local machine to pod
$ kubectl cp /some/local/file/or/dir [namespace/]pod:/some/remote/file/or/dir

@deads2k @smarterclayton @kubernetes/sig-cli

This change is

deads2k · 2016-10-17T11:55:04Z

@fabianofranz @csrwng fyi

pwittrock · 2016-10-17T16:27:44Z

@k8s-bot verify test this

0xmichalis · 2016-10-17T19:44:33Z

@kubernetes/kubectl

pwittrock · 2016-10-17T20:03:26Z

Review status: 0 of 3 files reviewed at latest revision, 8 unresolved discussions, some commit checks failed.

pkg/kubectl/cmd/cp.go, line 2 at r1 (raw file):

/*
Copyright 2014 The Kubernetes Authors.

super nit: 2016

pkg/kubectl/cmd/cp.go, line 37 at r1 (raw file):

  cp_example = dedent.Dedent(`
      # Copy /tmp/foo local file to /tmp/bar in a remote pod
      kubectl cp /tmp/foo <some-namespace>/<some-pod>:/tmp/bar

Is namespace optional?

pkg/kubectl/cmd/cp.go, line 40 at r1 (raw file):

      # Copy /tmp/foo from a remote pod to /tmp/bar locally
      kubectl cp <some-namespace>/<some-pod>:/tmp/foo /tmp/bar`)

Does this work on directories or just files?

pkg/kubectl/cmd/cp.go, line 53 at r1 (raw file):

  cmd := &cobra.Command{
      Use:     "cp <file-spec-src> <file-spec-dest>",
      Short:   "Copy files to and from containers.",

Should this be Copy a file to or from a container if it should only be used with a single file at a time? Otherwise maybe Copy files and directories to and from containers.

pkg/kubectl/cmd/cp.go, line 126 at r1 (raw file):

}

func recursiveTar(base, file string, tw *tar.Writer) error {

Does this compress the tar? Should it?

pkg/kubectl/cmd/cp.go, line 139 at r1 (raw file):

      }
      for _, f := range files {
          if err := recursiveTar(base, path.Join(file, f.Name()), tw); err != nil {

How does this approach to compare to using the filepath walk function as described in this stalk overflow question?

pkg/kubectl/cmd/cp_test.go, line 22 at r1 (raw file):

  "testing"
)

Please add tests for tar / untar. Is it possible to test copy to/from pod here as well?

pkg/kubectl/cmd/cp_test.go, line 23 at r1 (raw file):

)

func TestExtractFileSpec(t *testing.T) {

Would you also add an e2e test under https://github.com/kubernetes/kubernetes/blob/master/hack/make-rules/test-cmd.sh

Comments from Reviewable

pwittrock · 2016-10-17T20:04:29Z

FYI this keeps failing. Seems unrelated:

Unable to register third party resources: Get http://localhost:8080/api: dial tcp [::1]:8080: getsockopt: connection refuse

pwittrock · 2016-10-17T20:05:10Z

@brendandburns Thanks for work on this :) I look forward to using it.

brendandburns · 2016-10-18T05:23:08Z

Review status: 0 of 3 files reviewed at latest revision, 8 unresolved discussions.

pkg/kubectl/cmd/cp.go, line 2 at r1 (raw file):

Previously, pwittrock (Phillip Wittrock) wrote…

super nit: 2016

Done.

pkg/kubectl/cmd/cp.go, line 37 at r1 (raw file):

Previously, pwittrock (Phillip Wittrock) wrote…

Is namespace optional?

Yes, expanded examples..

pkg/kubectl/cmd/cp.go, line 40 at r1 (raw file):

Previously, pwittrock (Phillip Wittrock) wrote…

Does this work on directories or just files?

Both, expanded examples and description.

pkg/kubectl/cmd/cp.go, line 53 at r1 (raw file):

Previously, pwittrock (Phillip Wittrock) wrote…

Should this be Copy a file to or from a container if it should only be used with a single file at a time? Otherwise maybe Copy files and directories to and from containers.

expanded to include directories.

pkg/kubectl/cmd/cp.go, line 126 at r1 (raw file):

Previously, pwittrock (Phillip Wittrock) wrote…

Does this compress the tar? Should it?

No, it doesn't. Perhaps? Follow-on PR?

pkg/kubectl/cmd/cp.go, line 139 at r1 (raw file):

Previously, pwittrock (Phillip Wittrock) wrote…

How does this approach to compare to using the filepath walk function as described in this stalk overflow question?

It's more or less the same (here's the filepath code: https://golang.org/src/path/filepath/path.go

but walk doesn't follow symbolic links, so I think you actually want to us this which does.

pkg/kubectl/cmd/cp_test.go, line 22 at r1 (raw file):

Previously, pwittrock (Phillip Wittrock) wrote…

Please add tests for tar / untar. Is it possible to test copy to/from pod here as well?

done. I don't think it's possible in general. We'd need to shell out to tar, and I don't think we can guarantee it's present (e.g. Windows)

pkg/kubectl/cmd/cp_test.go, line 23 at r1 (raw file):

Previously, pwittrock (Phillip Wittrock) wrote…

Would you also add an e2e test under https://github.com/kubernetes/kubernetes/blob/master/hack/make-rules/test-cmd.sh

that won't work (for the same reasons as above)

We need a running pod in order to actually do the copy, but test-cmd.sh doesn't actually create any real containers afaik.

Comments from Reviewable

brendandburns · 2016-10-18T05:23:33Z

@pwittrock comments addressed, please take another look.

pwittrock · 2016-10-18T15:30:46Z

Reviewed 1 of 3 files at r1, 2 of 2 files at r2.
Review status: all files reviewed at latest revision, 4 unresolved discussions, some commit checks failed.

pkg/kubectl/cmd/cp.go, line 126 at r1 (raw file):

Previously, brendandburns (Brendan Burns) wrote…

No, it doesn't. Perhaps? Follow-on PR?

SGTM. If you don't have time to do a follow on, creating a TODO + issue is ok

pkg/kubectl/cmd/cp_test.go, line 22 at r1 (raw file):

Previously, brendandburns (Brendan Burns) wrote…

done. I don't think it's possible in general. We'd need to shell out to tar, and I don't think we can guarantee it's present (e.g. Windows)

Hm. In this case, do you think we should test for the present of tar and provide a helpful message to the user if it isn't found?

pkg/kubectl/cmd/cp_test.go, line 23 at r1 (raw file):

Previously, brendandburns (Brendan Burns) wrote…

that won't work (for the same reasons as above)

We need a running pod in order to actually do the copy, but test-cmd.sh doesn't actually create any real containers afaik.

Doesn't this test create a Pod, or is some piece being mocked out? https://github.com/kubernetes/kubernetes/blob/master/hack/make-rules/test-cmd.sh#L601

Comments from Reviewable

brendandburns · 2016-10-20T05:37:16Z

Review status: all files reviewed at latest revision, 3 unresolved discussions, some commit checks failed.

pkg/kubectl/cmd/cp.go, line 126 at r1 (raw file):

Previously, pwittrock (Phillip Wittrock) wrote…

SGTM. If you don't have time to do a follow on, creating a TODO + issue is ok

TODO'd

pkg/kubectl/cmd/cp_test.go, line 22 at r1 (raw file):

Previously, pwittrock (Phillip Wittrock) wrote…

Hm. In this case, do you think we should test for the present of tar and provide a helpful message to the user if it isn't found?

Well, the presence of 'tar' is inside the container. The point about testing is that we could mock out the complete command execution inside the container, but at that point the mock is so different than the real thing that I'm not sure it really tests anything more that what I already have...

We could test for 'tar' in the container with 'which tar' I suppose, but then what if 'which' isn't present? I don't think there are many containers where 'which' is present, but 'tar' isn't. This will mostly fail images from 'scratch'

pkg/kubectl/cmd/cp_test.go, line 23 at r1 (raw file):

Previously, pwittrock (Phillip Wittrock) wrote…

Doesn't this test create a Pod, or is some piece being mocked out? https://github.com/kubernetes/kubernetes/blob/master/hack/make-rules/test-cmd.sh#L601

The docker endpoint is mocked in those tests afaik:

https://github.com/kubernetes/kubernetes/blob/master/hack/make-rules/test-cmd.sh#L203

So no containers are actually created.

Comments from Reviewable

brendandburns · 2016-10-20T05:37:39Z

@pwittrock replied to comments, ptal.

thanks!
--brendan

k8s-ci-robot · 2016-10-20T05:52:56Z

Jenkins unit/integration failed for commit 42dfbe1. Full PR test history.

The magic incantation to run this job again is @k8s-bot unit test this. Please help us cut down flakes by linking to an open flake issue when you hit one in your PR.

k8s-ci-robot · 2016-10-20T06:00:57Z

Jenkins GCI GKE smoke e2e failed for commit 42dfbe1. Full PR test history.

The magic incantation to run this job again is @k8s-bot gci gke e2e test this. Please help us cut down flakes by linking to an open flake issue when you hit one in your PR.

pwittrock · 2016-10-20T15:52:39Z

Reviewed 1 of 1 files at r3.
Review status: all files reviewed at latest revision, 2 unresolved discussions, some commit checks failed.

pkg/kubectl/cmd/cp_test.go, line 22 at r1 (raw file):

Previously, brendandburns (Brendan Burns) wrote…

Well, the presence of 'tar' is inside the container. The point about testing is that we could mock out the complete command execution inside the container, but at that point the mock is so different than the real thing that I'm not sure it really tests anything more that what I already have...

We could test for 'tar' in the container with 'which tar' I suppose, but then what if 'which' isn't present? I don't think there are many containers where 'which' is present, but 'tar' isn't. This will mostly fail images from 'scratch'

Sorry, I was unclear, I was no longer referring to the tests. What I meant was this command will fail on any machine without tar right, but a user might not have tar installed causing this command to fail. Probably not a big deal, but it would be nice if we checked that the prereq commands were actually installed and provided a helpful message to the user if they are not. e.g. "kubectl cp requires the tar command to be installed but it was not found." I think this is a non-blocker even if you agree so I will mark this lgtm. If you agree with my comment you can either 1) add a todo and re-apply lgtm yourself or 2) add the message and I will TAL

pkg/kubectl/cmd/cp_test.go, line 23 at r1 (raw file):

Previously, brendandburns (Brendan Burns) wrote…

The docker endpoint is mocked in those tests afaik:

https://github.com/kubernetes/kubernetes/blob/master/hack/make-rules/test-cmd.sh#L203

So no containers are actually created.

Ah very interesting. Thx

Comments from Reviewable

pwittrock · 2016-10-20T15:53:50Z

FYI at least some of the test failures appear to be legit:

pkg/kubectl/cmd/cp_test.go:135:4: missing ',' before newline in composite literal

pwittrock · 2016-10-21T02:19:48Z

I am actually doing something where this would be really convienient :P

pwittrock · 2016-10-21T02:21:58Z

Looks like you gave edit permissions and I can fix that minor error for you if you are ok with me doing so.

brendandburns · 2016-10-24T16:01:20Z

self-LGTM on LGTM from @pwittrock

k8s-ci-robot · 2016-10-26T07:18:34Z

Jenkins verification failed for commit a198a57. Full PR test history.

The magic incantation to run this job again is @k8s-bot verify test this. Please help us cut down flakes by linking to an open flake issue when you hit one in your PR.

soltysh · 2016-10-26T13:07:42Z

pkg/kubectl/cmd/cp.go

+	}()
+
+	// TODO: Improve error messages by first testing if 'tar' is present in the container?
+	cmdArr := []string{"tar", "xf", "-"}


tar requirement on the pod should be explicitly specified in the description. Otherwise we'll get users complaining about it, imho.

pwittrock · 2016-10-27T20:25:09Z

error: Could not remove config section 'remote.bootstrap-upstream'

Ideas?

brendandburns · 2016-10-30T21:29:31Z

self-lgtm after fixing the bazel issue.

k8s-github-robot · 2016-10-30T21:37:55Z

@k8s-bot test this [submit-queue is verifying that this PR is safe to merge]

k8s-github-robot · 2016-10-30T22:12:41Z

Automatic merge from submit-queue

frazr · 2017-03-16T19:42:44Z

A nifty feature to kubectl cp would be that it would behave like "mkdir -p" when a parameter is added.

kubectl cp foo.txt POD:/tmp/foo/bar/bar.txt
tar: /tmp/foo/bar: Cannot open: No such file or directory
tar: Error is not recoverable: exiting now

Currently kubectl cp does not seem to support this.

googlebot added the cla: yes label Oct 16, 2016

k8s-github-robot assigned pwittrock Oct 16, 2016

k8s-github-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. release-note-label-needed labels Oct 16, 2016

brendandburns force-pushed the kubectl-cp branch from ac1d951 to 5266230 Compare October 17, 2016 00:03

juanvallejo mentioned this pull request Oct 17, 2016

Append oc copy command to copy a single file to container (like oc rsync) openshift/origin#10529

Closed

brendandburns added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed release-note-label-needed labels Oct 18, 2016

pwittrock added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 20, 2016

brendandburns force-pushed the kubectl-cp branch from 42dfbe1 to a198a57 Compare October 24, 2016 04:57

k8s-github-robot added kind/old-docs and removed lgtm "Looks good to me", indicates that a PR is ready to be merged. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Oct 24, 2016

k8s-github-robot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Oct 24, 2016

brendandburns added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 24, 2016

soltysh reviewed Oct 26, 2016

View reviewed changes

add kubectl cp

d65757f

brendandburns force-pushed the kubectl-cp branch from a198a57 to d65757f Compare October 30, 2016 03:25

k8s-github-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 30, 2016

brendandburns added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 30, 2016

k8s-github-robot merged commit f448f24 into kubernetes:master Oct 30, 2016

chentao1596 mentioned this pull request Dec 5, 2016

WIP:kubelet: support multi-headers when getting pod from HTTP source #38089

Closed

pweil- mentioned this pull request Mar 23, 2017

Dynamically provisioning ssh remote access for openshift containers (use case FileZilla sftp file transfers) openshift/origin#13521

Closed

KnVerey mentioned this pull request Nov 11, 2022

kubectl: remove unnecessary TODO annotations #111403

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add kubectl cp #34914

add kubectl cp #34914

brendandburns commented Oct 16, 2016 •

edited by k8s-oncall

Loading

deads2k commented Oct 17, 2016

pwittrock commented Oct 17, 2016

0xmichalis commented Oct 17, 2016

pwittrock commented Oct 17, 2016

pwittrock commented Oct 17, 2016

pwittrock commented Oct 17, 2016

brendandburns commented Oct 18, 2016

brendandburns commented Oct 18, 2016

pwittrock commented Oct 18, 2016

brendandburns commented Oct 20, 2016

brendandburns commented Oct 20, 2016

k8s-ci-robot commented Oct 20, 2016

k8s-ci-robot commented Oct 20, 2016

pwittrock commented Oct 20, 2016

pwittrock commented Oct 20, 2016

pwittrock commented Oct 21, 2016

pwittrock commented Oct 21, 2016

brendandburns commented Oct 24, 2016

k8s-ci-robot commented Oct 26, 2016

soltysh Oct 26, 2016

brendandburns Oct 30, 2016

pwittrock commented Oct 27, 2016

brendandburns commented Oct 30, 2016

k8s-github-robot commented Oct 30, 2016

k8s-github-robot commented Oct 30, 2016

frazr commented Mar 16, 2017 •

edited

Loading

add kubectl cp #34914

add kubectl cp #34914

Conversation

brendandburns commented Oct 16, 2016 • edited by k8s-oncall Loading

deads2k commented Oct 17, 2016

pwittrock commented Oct 17, 2016

0xmichalis commented Oct 17, 2016

pwittrock commented Oct 17, 2016

pwittrock commented Oct 17, 2016

pwittrock commented Oct 17, 2016

brendandburns commented Oct 18, 2016

brendandburns commented Oct 18, 2016

pwittrock commented Oct 18, 2016

brendandburns commented Oct 20, 2016

brendandburns commented Oct 20, 2016

k8s-ci-robot commented Oct 20, 2016

k8s-ci-robot commented Oct 20, 2016

pwittrock commented Oct 20, 2016

pwittrock commented Oct 20, 2016

pwittrock commented Oct 21, 2016

pwittrock commented Oct 21, 2016

brendandburns commented Oct 24, 2016

k8s-ci-robot commented Oct 26, 2016

soltysh Oct 26, 2016

Choose a reason for hiding this comment

brendandburns Oct 30, 2016

Choose a reason for hiding this comment

pwittrock commented Oct 27, 2016

brendandburns commented Oct 30, 2016

k8s-github-robot commented Oct 30, 2016

k8s-github-robot commented Oct 30, 2016

frazr commented Mar 16, 2017 • edited Loading

brendandburns commented Oct 16, 2016 •

edited by k8s-oncall

Loading

frazr commented Mar 16, 2017 •

edited

Loading