Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cp.push saved file crashed #6495

Closed
GeniusWiki opened this issue Aug 2, 2013 · 16 comments · Fixed by #7437
Closed

cp.push saved file crashed #6495

GeniusWiki opened this issue Aug 2, 2013 · 16 comments · Fixed by #7437

Comments

@GeniusWiki
Copy link

@GeniusWiki GeniusWiki commented Aug 2, 2013

Command looks this:
salt 'db-01' cp.push /var/backups/some.tar.gz

The file downloaded has same size, but the md4sum is different. And gunzip cannot unzip - the original file can be unzipped.

I test same command in other server but different gzip files, looks fine.
I also test this command in same server but a small text file, looks fine as well...

@GeniusWiki

This comment has been minimized.

Copy link
Author

@GeniusWiki GeniusWiki commented Aug 2, 2013

did further test, move same tar.gz to another server (which is success on another file cp.push), cp.push failed again.

@UtahDave

This comment has been minimized.

Copy link
Member

@UtahDave UtahDave commented Aug 2, 2013

So does this only affect gzipped files?

@GeniusWiki

This comment has been minimized.

Copy link
Author

@GeniusWiki GeniusWiki commented Aug 2, 2013

No , I test push the unzipped file, a plain SQL dump file, still crashed.

On 03/08/2013, at 3:35 AM, David Boucha notifications@github.com wrote:

So does this only affect gzipped files?


Reply to this email directly or view it on GitHub.

@terminalmage

This comment has been minimized.

Copy link
Contributor

@terminalmage terminalmage commented Aug 8, 2013

When you say "crashed", what do you mean by this? Is there a traceback? If so, please post it.

Also, please post the output of salt --versions-report on the minion.

@terminalmage

This comment has been minimized.

Copy link
Contributor

@terminalmage terminalmage commented Aug 8, 2013

I've confirmed this bug in develop

The original file:

erik@virtubuntu:~/Downloads% ls -l salt-0.16.0.tar.gz
-rw-r--r-- 1 erik erik 3523789 Aug  8 14:07 salt-0.16.0.tar.gz
erik@virtubuntu:~/Downloads% md5sum salt-0.16.0.tar.gz
4e4f382614b8c30818be0c2ebe2988e8  salt-0.16.0.tar.gz
erik@virtubuntu:~/Downloads% file salt-0.16.0.tar.gz
salt-0.16.0.tar.gz: gzip compressed data, was "dist/salt-0.16.0.tar", last modified: Mon Jul  1 17:53:03 2013, max compression

The file after being pushed to the master:

root@virtubuntu:/home/erik/saltdev/var/cache/salt/master/minions/virtubuntu/files/home/erik/Downloads# ls -l salt-0.16.0.tar.gz
-rw-r--r-- 1 root root 3523789 Aug  8 14:30 salt-0.16.0.tar.gz
root@virtubuntu:/home/erik/saltdev/var/cache/salt/master/minions/virtubuntu/files/home/erik/Downloads# md5sum salt-0.16.0.tar.gz
5cc431689f26a8eaa0bb59a9d620996f  salt-0.16.0.tar.gz
root@virtubuntu:/home/erik/saltdev/var/cache/salt/master/minions/virtubuntu/files/home/erik/Downloads# file salt-0.16.0.tar.gz
salt-0.16.0.tar.gz: data
root@virtubuntu:/home/erik/saltdev/var/cache/salt/master/minions/virtubuntu/files/home/erik/Downloads# tar tvf salt-0.16.0.tar.gz

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
@terminalmage

This comment has been minimized.

Copy link
Contributor

@terminalmage terminalmage commented Aug 8, 2013

Another problem I noticed here is that, if the recv_files file_recv option is not set to True, then cp.push still returns True even though no file was copied.

@terminalmage

This comment has been minimized.

Copy link
Contributor

@terminalmage terminalmage commented Aug 8, 2013

Additionally, it seems that when file_recv is not set, the minion still sends the file up to the master. Here's debug output from the master:

[INFO    ] Clear payload received with command publish
[INFO    ] User root Published command cp.push with jid 20130808163412295980
[DEBUG   ] Published command details {'tgt_type': 'glob', 'jid': '20130808163412295980', 'tgt': 'virtubuntu', 'ret': '', 'user': 'root', 'arg': ['/home/erik/Downloads/salt-0.16.0.tar.gz'], 'fun': 'cp.push'}
[INFO    ] Clear payload received with command _auth
[DEBUG   ] This salt-master instance has accepted 1 minion keys.
[INFO    ] Authentication request from virtubuntu
[INFO    ] Authentication accepted from virtubuntu
[INFO    ] AES payload received with command _file_recv
[INFO    ] AES payload received with command _file_recv
[INFO    ] AES payload received with command _file_recv
[INFO    ] AES payload received with command _file_recv
[INFO    ] AES payload received with command _file_recv
[INFO    ] AES payload received with command _file_recv
[INFO    ] AES payload received with command _file_recv
[INFO    ] AES payload received with command _file_recv
[INFO    ] AES payload received with command _file_recv
[INFO    ] AES payload received with command _file_recv
[INFO    ] AES payload received with command _file_recv
[INFO    ] AES payload received with command _file_recv
[INFO    ] AES payload received with command _file_recv
[INFO    ] AES payload received with command _file_recv
[INFO    ] AES payload received with command _return
[INFO    ] Got return from virtubuntu for job 20130808163412295980
@GeniusWiki

This comment has been minimized.

Copy link
Author

@GeniusWiki GeniusWiki commented Aug 8, 2013

Looks you reproduced the bug, anyway, put my version report and hope it help.
Salt: 0.16.0
Python: 2.7.3 (default, Apr 10 2013, 06:20:15)
Jinja2: 2.6
M2Crypto: 0.21.1
msgpack-python: 0.1.10
msgpack-pure: Not Installed
pycrypto: 2.4.1
PyYAML: 3.10
PyZMQ: 13.0.0
ZMQ: 3.2.2

As I used cp.push in 2 different server for daily backup, looks the problem related to file size. The smaller one always success. The large one >10M fail always.

@terminalmage

This comment has been minimized.

Copy link
Contributor

@terminalmage terminalmage commented Aug 8, 2013

@GeniusWiki Thanks! But can you clarify further what you mean by the cp.push "crashing"?

@GeniusWiki

This comment has been minimized.

Copy link
Author

@GeniusWiki GeniusWiki commented Aug 8, 2013

Just md5sum checking failed and gunzip failed, same as your test...

@terminalmage

This comment has been minimized.

Copy link
Contributor

@terminalmage terminalmage commented Aug 8, 2013

OK. I was just concerned that there were tracebacks I wasn't catching. I found a traceback when no file_recv option is set in the master config file, which I have fixed.

Thanks for the report, we'll look into this.

@GeniusWiki

This comment has been minimized.

Copy link
Author

@GeniusWiki GeniusWiki commented Aug 8, 2013

I confirmed my configuration:
file_recv: True

@terminalmage

This comment has been minimized.

Copy link
Contributor

@terminalmage terminalmage commented Aug 8, 2013

Yeah, the issue that caused the traceback I found earlier is that there was no default value when file_recv isn't included in the master config file.

@anitakrueger

This comment has been minimized.

Copy link
Contributor

@anitakrueger anitakrueger commented Sep 20, 2013

Wondering what the word is on this one?

I tried the same thing:
salt-call cp.push /tmp/base.tar.gz
The file makes it to the master's cache directory just fine, but is then garbled just like it was transferred as an ascii file instead of binary. The md5sums of the 2 files don't match anymore.
a4e1db13bef270c726cf30cfbabcbf19 /tmp/base.tar.gz
86dc81d6bbaf47c76366991f9b0466ae /var/cache/salt/master/minions/minion1/files/tmp/base.tar.gz

@basepi

This comment has been minimized.

Copy link
Collaborator

@basepi basepi commented Sep 23, 2013

I think this issue has just been lost in the shuffle. Thanks for bumping it, it's still on our radar, just so many issues currently! =)

@terminalmage

This comment has been minimized.

Copy link
Contributor

@terminalmage terminalmage commented Sep 24, 2013

I've found and fixed the problem. That was a tricky one, but once I tracked it down the solution was pretty easy.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.