Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a kubectl cp ... command #13776

Closed
brendandburns opened this issue Sep 10, 2015 · 40 comments

Comments

@brendandburns
Copy link
Contributor

commented Sep 10, 2015

Containers are supposed to be immutable, I know, but sometimes copy is useful...

Syntax:

# copy from a container to local
kubectl cp container:/path/to/file /path/to/local/file

# copy from local to container
kubectl cp /path/to/local/file container:/path/to/file

#copy from container to container
kubectl cp container1:/path/to/file container2:/path/to/other/file

@smarterclayton @bgrant0607 @kubernetes/kubectl

@smarterclayton

This comment has been minimized.

Copy link
Contributor

commented Sep 10, 2015

Is that pod to pod copy without an intermediate, or passing back to the client? We'd need to identify pod -> container specificity

@bgrant0607

This comment has been minimized.

Copy link
Member

commented Sep 10, 2015

I agree this would be useful for debugging.

Please ensure pod/container syntax is consistent with exec, log, etc.

As @smarterclayton pointed out, you need to identify the pod.

How do you imagine implementing this?

cc @ncdc @vishh

@bgrant0607

This comment has been minimized.

Copy link
Member

commented Sep 10, 2015

And we should ensure it handles volumes properly. Not sure if docker's cp does yet. Almost certainly it won't work for terminated containers.

@smarterclayton

This comment has been minimized.

Copy link
Contributor

commented Sep 10, 2015

I should note we're going to do a client side command for rsync that does
increments transfer over exec, and rely on the client local rsync for bulk
transfers. This is a good general purpose tool to have just because you
don't need anything in the remote container (although eventually I'd like
to be able to inject a shell into any container via a busybox like http one
off server proc)

On Sep 9, 2015, at 8:28 PM, Brian Grant notifications@github.com wrote:

I agree this would be useful for debugging.

Please ensure pod/container syntax is consistent with exec, log, etc.

As @smarterclayton https://github.com/smarterclayton pointed out, you
need to identify the pod.

How do you imagine implementing this?

cc @ncdc https://github.com/ncdc @vishh https://github.com/vishh


Reply to this email directly or view it on GitHub
#13776 (comment)
.

@brendandburns

This comment has been minimized.

Copy link
Contributor Author

commented Sep 10, 2015

Yes, will def. make it match exec/attach/logs/...

@bgrant0607 I was thinking of implementing it in terms of exec, e.g.

kubectl cp $local-file $remote-file

just looks like:

cat $local-file | kubectl exec bash -c "cat > $remote-file"

And

kubectl cp $remote-file $local-file

is

kubectl exec cat $remote-file > $local-file

Possibly mix in some kubectl ls $remote-file to try to do maintain permissions / uid

And I envisioned the remote to remote transiting at least through the master, and likely through the client in the first pass (it's just easier, though I understand why you wouldn't want to do that)

@smarterclayton can you add more detail to what you are thinking wrt rsync, I had a little trouble parsing your description in particular ("that does increments transfer over exec")

Thanks
--brendan

@resouer

This comment has been minimized.

Copy link
Member

commented Sep 10, 2015

@bgrant0607 FYI, docker cp is implemented by directly copy files from host's file system, we do a fake Mount to generate real paths of container on the host, including volume dir. In a word, quite a docker specific way.

One of the concern is: I can see dockerd is so vulnerable that even larger amount log stdout may crash it. Not sure if cp big file or a lot of files will be too much burden for it?

@smarterclayton

This comment has been minimized.

Copy link
Contributor

commented Sep 10, 2015

The rsync path was exactly the same as what you described, rsync -av -e 'kubectl exec -i ...' localdir remotehost:/remotedir. We have to make one
change to allow it though, because when you run rsync like this it adds the
argument of the destination host first, then the command second. I haven't
looked at exec in the last few weeks but we probably need to have a way to
enforce that for exec to work.

The docker cp path is another option, and the best part is it doesn't
require anything specific in a container. However, right now (because I
got overruled by certain @thockin for the default path cough) exec also
goes through the dockerd. We have alterative solutions but they haven't
been prioritized yet. So docker cp is no worse than exec.

On Thu, Sep 10, 2015 at 9:13 AM, Harry Zhang notifications@github.com
wrote:

@bgrant0607 https://github.com/bgrant0607 FYI, docker cp is implemented
by directly copy files from host's file system, we'll do a fake Mount to
generate real paths of container on the host, including volume dir. In a
word, quite a docker specific way.

One of the concern is: I can see dockerd is so vulnerable that even larger
amount log stdout may crash it. Not sure if cp big file or a lot of files
will be too much burden for it?


Reply to this email directly or view it on GitHub
#13776 (comment)
.

Clayton Coleman | Lead Engineer, OpenShift

@brendandburns

This comment has been minimized.

Copy link
Contributor Author

commented Oct 21, 2015

@smarterclayton the rsync version requires the rsync daemon be present in the container, which is kind of an unnecessary requirement, don't you think? I don't really want to use docker cp since it requires plumbing a whole additional API call through the kubelet, apiserver etc.

I think I will just stick with cat.

@smarterclayton

This comment has been minimized.

Copy link
Contributor

commented Oct 21, 2015

Might be unnecessary, but present in the vast majority of containers.
We're doing it mostly to enable rapid dev scenarios on Windows - it's not
trivial to get an rsync on a windows system, but it's still a better
experience than pretty much any other story for dealing with O(1K) files
that need to be synced.

On Tue, Oct 20, 2015 at 11:32 PM, Brendan Burns notifications@github.com
wrote:

@smarterclayton https://github.com/smarterclayton the rsync version
requires the rsync daemon be present in the container, which is kind of an
unnecessary requirement, don't you think? I don't really want to use docker
cp since it requires plumbing a whole additional API call through the
kubelet, apiserver etc.

I think I will just stick with cat.


Reply to this email directly or view it on GitHub
#13776 (comment)
.

@xied75

This comment has been minimized.

Copy link

commented Jan 25, 2016

💯 Since kubectl aligns subcommands with docker, a 'cp' is definitely needed for everyday life, so that DevOps can still have some hair left. With 'cp' ability lots of problems can be solved with it, e.g. a gitrepo volume mount is no longer needed as many people use it to just get hold of some conf file, also the genius usage of k8s Secret for the same purpose.

Given we can do "kubectl exec " and this goes via TLS channel, can we do something similar to ssh/sshd that we can open a reverse proxy port somehow? If that's possible then in order to get a file into the running container I can do this:
"kubectl exec -- scp -P 22000 me@localhost:/home/me/k8s/nginx/default.conf /etc/nginx/conf.d/default.conf"

@johanhaleby

This comment has been minimized.

Copy link

commented Jan 25, 2016

I would also be interested in this. My use case is that I've created/edited a file locally I and want to copy it to a pod that has a Google Persistent Disk attached to it.

@bittrance

This comment has been minimized.

Copy link

commented Jan 26, 2016

@xied75 I have a python script in container that allows this:

tar cf - . | kubectl exec shinken -i -- catcher /configdir

See https://github.com/bittrance/docker-shinken/blob/master/catcher.py

@xied75

This comment has been minimized.

Copy link

commented Jan 27, 2016

Interesting enough I found similar solution yesterday, looks like "cp" can be done like this:

kubectl exec -i mypod -- /bin/bash -c 'cat > /the/path/to/your/pod/file.txt' < /path/to/your/local/file.txt
@joeatwork

This comment has been minimized.

Copy link

commented Feb 26, 2016

Some containers don't have equipment like bash or scp (or even cat) installed - one example of a container like this is gcr.io/google_containers/etcd:2.0.9, which is a pretty common beast in most Kubernetes clusters.

@xoss

This comment has been minimized.

Copy link

commented Mar 14, 2016

Thanks to @bittrance and @xied75, we found an even shorter solution which also includes entire directories:

tar cz . | kubectl exec -i POD -- tar xz .

Now, we have a node-dev (https://quay.io/amadeusmuc/node-dev) running in the developer's namespace and sync the entire folder to that plain vanilla node container. It is a rough solution, but it does work.
However, a decent implementation would help a lot.

Update:
We've added the following in order to prevent any unnecessary file/folder to be synced.

tar -X .gitignore -cpzf - . | kubectl exec -i $POD -- tar -xpzf - .;

We've wrapped that in a script called watchandpush:

#!/bin/bash
POD=$1;

echo "Pushing initally to pod $POD";
tar -X .gitignore -cpzf - . | kubectl exec -i $POD -- tar -xpzf - .;

while [[ true ]]
do
  if [[ `find . -type f -mtime -3s \( ! -iname ".*" \)` ]]; then
    echo "Changes detected. Pushing to pod $POD";
    tar -X .gitignore -cpzf - . | kubectl exec -i $POD -- tar -xpzf - .;
  fi
  sleep 2;
done;

So in the working directory, we do an initial push to $POD and watch for changes to push subsequent changes to $POD.

@jhoblitt

This comment has been minimized.

Copy link

commented Apr 13, 2016

👍 for built in cp support. This would be invaluable for debugging.

@singlenumber

This comment has been minimized.

Copy link

commented Apr 21, 2016

Support this feature, it would be helpful.

@dbbernstein

This comment has been minimized.

Copy link

commented Apr 28, 2016

+1 for working in a container without remote editing

@expz

This comment has been minimized.

Copy link

commented May 8, 2016

The Openshift Origin "distributon" of Kubernetes has already implemented an rsync subcommand of their oc command (in Go, of course). Perhaps it can be used without much modification:

https://github.com/openshift/origin/tree/388478c40e751c4295dcb9a44dd69e5ac65d0e3b/pkg/cmd/cli/cmd/rsync

@whereisaaron

This comment has been minimized.

Copy link

commented Jun 5, 2016

While some form of runtime container file transfer would be handy, the 'docker cp' command features obscure semantics with a range of overloaded behaviors triggers by judicious placement of slashes and dots. I'd prefer to see 'docker cp' semantics left behind in favor of one or two cleaner kubectl commands.

@joshperry

This comment has been minimized.

Copy link

commented Aug 23, 2016

This is an interesting req that I found after trying to do a volume mount, out of habit, with kubectl exec. I would definitely make use of this occasionally, but perhaps a volume mount feature more often.

When I need to copy things to containers it's usually to do some kind of debugging or hotfix, and I need some tools to facilitate my task. It would be nice to be able to attach a volume with tools or files along with the exec and then just use cp on the container system to copy from the volume if needed.

Mounting ~/.ssh for example would be particularly helpful.

@sijnc

This comment has been minimized.

Copy link

commented Aug 28, 2016

This would be very helpful, would love to be able to mount a pod/container and be able to live edit files for debugging and hotfix as suggested by @joshperry. We have a very large deployment that cannot run on a local dev machine so we have to develop in the kubernetes cluster.

The ability to mount via something like SSHFS would be great!

@brendandburns

This comment has been minimized.

Copy link
Contributor Author

commented Aug 29, 2016

I just realized that we could even use this to write a FUSE driver... Mount your pod's file system into the local filesystem.

What could possibly go wrong :)

@feniix

This comment has been minimized.

Copy link

commented Aug 29, 2016

You mean something similar to this? https://github.com/opencredo/kubefuse

@mqliang

This comment has been minimized.

Copy link
Member

commented Sep 19, 2016

kubectl exec POD -c Container -i 
apt-get insatll lrzsz
rz #receive file
sz FILE #send file

works for terms supporting zmodem/ymodem/xmodem , such as xshell, and iterm2(with zmodem integrating, see https://github.com/aurora/iterm2-zmodem). Unfortunately, If we want send/receive file using web term, it does't work(since web term does not support zmodem/ymodem/xmodem)

@mark9white

This comment has been minimized.

Copy link

commented Sep 20, 2016

@feniix that app gives you the kubernetes resources on the filesystem, not the pod filesystem.

Here's a little script I've got to mount the pod filesystem on OS X into ~/Downloads/fuse with $2 being the pod name:

kill $(ps aux  | grep "5100:22" | grep kubectl |  awk '{print $2}')
umount -f ~/Downloads/fuse

kubectl exec -i $2 -- /bin/bash -c 'cat > /sshfsci' < ~/bin/sshfsci
kubectl exec $2 -- /bin/bash  -c "chmod +x /sshfsci; /sshfsci"

kubectl port-forward $2 5100:22 &
sleep 2

mkdir -p ~/Downloads/fuse
sshfs root@localhost:/ ~/Downloads/fuse -p 5100 -o reconnect,ServerAliveInterval=5,ServerAliveCountMax=3
cd ~/Downloads/fuse

and the ~/bin/sshfsci script contains (replace *** with the key):

if [ $(dpkg-query -W -f='${Status}' openssh-server 2>/dev/null | grep -c "ok installed") -eq 0 ];
then
  apt-get update && apt-get install -y --no-install-recommends openssh-server;
fi
mkdir /var/run/sshd
sed -i 's/PermitRootLogin without-password/PermitRootLogin yes/' /etc/ssh/sshd_config
sed 's@session\s*required\s*pam_loginuid.so@session optional pam_loginuid.so@g' -i /etc/pam.d/sshd
if [ "$HOME" = "/" ]
then
  SSH_HOME=/root/.ssh
else
  SSH_HOME=$HOME/.ssh
fi
echo $SSH_HOME
mkdir -p $SSH_HOME
echo ssh-rsa ****** > $SSH_HOME/authorized_keys
chmod 600 $SSH_HOME/authorized_keys
service ssh start
@arterrey

This comment has been minimized.

Copy link

commented Sep 28, 2016

here is an rsync solution... super hacky but seems to work, assumes you have rsync in the container

rsync -av --blocking-io --rsync-path= --rsh='bin/kubectl exec wordpress-replication-bssaf -i -- ' rsync:/var/www/html ./html

(updated: added --blocking-io option which fixes the same issue in moby/moby#13660)

@smarterclayton

This comment has been minimized.

Copy link
Contributor

commented Sep 28, 2016

Yeah, that's what we use in openshift (for linux platforms). Works pretty
well when both client and server match up, can be more difficult on windows.

On Wed, Sep 28, 2016 at 4:36 AM, Adam Terrey notifications@github.com
wrote:

here is an rsync solution... super hacky but seems to work, assumes you
have rsync in the container

rsync -av -e 'kubectl exec POD -i -- env ' --rsync-path= rsync:/var/www/html ./html


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#13776 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABG_p3MmTxynetwBG71pBn2g9JdygTRmks5quicrgaJpZM4F6q6B
.

@timgivois

This comment has been minimized.

Copy link

commented Oct 27, 2016

+1

k8s-github-robot pushed a commit that referenced this issue Oct 30, 2016

Kubernetes Submit Queue
Merge pull request #34914 from brendandburns/kubectl-cp
Automatic merge from submit-queue

add kubectl cp

Implements `kubectl cp` (#13776)

Syntax examples:

```sh
# Copy from pod to local machine
$ kubectl cp [namespace/]pod:/some/file/or/dir ./some/local/file/or/dir

# Copy from local machine to pod
$ kubectl cp /some/local/file/or/dir [namespace/]pod:/some/remote/file/or/dir
```

@deads2k @smarterclayton @kubernetes/sig-cli
@brendandburns

This comment has been minimized.

Copy link
Contributor Author

commented Nov 1, 2016

This was added. I used tar, that is all.

@anguslees

This comment has been minimized.

Copy link
Member

commented Nov 24, 2016

Just because I didn't see it explicitly spelled out above, the rsync version is as easy as ensuring the rsync binary is installed in the container and setting RSYNC_RSH env var to point to the location of this script:

#!/bin/sh
name="$1"; shift
exec kubectl exec $name --stdin -- "$@"

Then you can just do rsync /local/path $name_of_pod:/remote/path and all the other rsync argument variations as usual.

@mrmcmuffinz

This comment has been minimized.

Copy link

commented Feb 17, 2017

I know I'm kinda late to the game here and this item was closed but what was the exact solution be able to copy a remote file over from my host to the pod? I'm on linux/unix so no windows issues here.

@ax-nathan

This comment has been minimized.

Copy link

commented Mar 1, 2017

@mrmcmuffinz

$ kubectl cp --help
Copy files and directories to and from containers.

Examples:
  # !!!Important Note!!!
  # Requires that the 'tar' binary is present in your container
  # image.  If 'tar' is not present, 'kubectl cp' will fail.
  
  # Copy /tmp/foo_dir local directory to /tmp/bar_dir in a remote pod in the default namespace
  kubectl cp /tmp/foo_dir <some-pod>:/tmp/bar_dir
  
  # Copy /tmp/foo local file to /tmp/bar in a remote pod in a specific container
  kubectl cp /tmp/foo <some-pod>:/tmp/bar -c <specific-container>
  
  # Copy /tmp/foo local file to /tmp/bar in a remote pod in namespace <some-namespace>
  kubectl cp /tmp/foo <some-namespace>/<some-pod>:/tmp/bar
  
  # Copy /tmp/foo from a remote pod to /tmp/bar locally
  kubectl cp <some-namespace>/<some-pod>:/tmp/foo /tmp/bar

Options:
  -c, --container='': Container name. If omitted, the first container in the pod will be chosen

Usage:
  kubectl cp <file-spec-src> <file-spec-dest> [options]

Use "kubectl options" for a list of global command-line options (applies to all commands).
@hex0cter

This comment has been minimized.

Copy link

commented May 17, 2017

+1!

@fncmedia

This comment has been minimized.

Copy link

commented Jul 13, 2017

Should it be possible to copy files of an completed pod? I failed on trying. Docker cp makes it possible when using on kubernetes node but this does not solve our issue.

@ghost

This comment has been minimized.

Copy link

commented Aug 14, 2017

+1

@AnthonyWC

This comment has been minimized.

Copy link

commented Aug 21, 2017

@fncmedia Did you check if tar is installed in the container?

@NiraliSupe

This comment has been minimized.

Copy link

commented Apr 21, 2018

Hello,
I need to take volume backup. I am trying to rsync pod volume folder to backup folder.
rsync -avv -e 'kubectl exec postgres-5554f4fdd4-nhs48 -n default -i -- env ' --rsync-path=/var/lib/postgresql/data/ ~/pod

but it is giving output

building file list ...
done
drwxr-xr-x          64 2018/04/21 13:50:10 podd
delta-transmission disabled for local transfer or --whole-file
total: matches=0  hash_hits=0  false_alarms=0 data=0

sent 98 bytes  received 20 bytes  236.00 bytes/sec
total size is 0  speedup is 0.00

and pod folder is empty

can someone please help with this?

@anguslees

This comment has been minimized.

Copy link
Member

commented Apr 25, 2018

@NiraliSupe on the end of a mostly-unrelated closed github issue is not the best place to hold a support discussion. Try #kubernetes-users on slack.kubernetes.com (http://slack.k8s.io/ if you don't already have an account)

@mckingho

This comment has been minimized.

Copy link

commented Jul 13, 2018

Is this command working on remote pod -> remote pod? I tried with kubectl cp podA:/app/a.tar.gz podB:/app/ and got error: open /app: permission denied.
Is it unsupported or permission issue that i set unproperly? I can cp from podA to local and then cp from local to podB inconveniently.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.