traceback after upload of existing file due to dict/SortedDict difference #128

Closed
mdomsch opened this Issue Mar 5, 2013 · 2 comments

Projects

None yet

1 participant

@mdomsch
Member
mdomsch commented Mar 5, 2013

The upload of a single file fails if it already exists on the remote side. First, we sync upload a single file. Then we do it again, and get the traceback below.

Likely instead of using an ordinary dict() here we need to use a SortedDict().

'''
$ ./s3cmd sync --debug ./s3cmd s3://mdomsch.1000-s3cmd-test/s3cmd
DEBUG: ConfigParser: Reading file '/home/mdomsch/.s3cfg'
DEBUG: ConfigParser: access_key->AK...17_chars...A
DEBUG: ConfigParser: bucket_location->US
DEBUG: ConfigParser: cloudfront_host->cloudfront.amazonaws.com
DEBUG: ConfigParser: cloudfront_resource->/2010-07-15/distribution
DEBUG: ConfigParser: default_mime_type->binary/octet-stream
DEBUG: ConfigParser: delete_removed->False
DEBUG: ConfigParser: dry_run->False
DEBUG: ConfigParser: encoding->UTF-8
DEBUG: ConfigParser: encrypt->False
DEBUG: ConfigParser: follow_symlinks->False
DEBUG: ConfigParser: force->False
DEBUG: ConfigParser: get_continue->False
DEBUG: ConfigParser: gpg_command->/usr/bin/gpg
DEBUG: ConfigParser: gpg_decrypt->%(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
DEBUG: ConfigParser: gpg_encrypt->%(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
DEBUG: ConfigParser: gpg_passphrase->...-3_chars...
DEBUG: ConfigParser: guess_mime_type->True
DEBUG: ConfigParser: host_base->s3.amazonaws.com
DEBUG: ConfigParser: host_bucket->%(bucket)s.s3.amazonaws.com
DEBUG: ConfigParser: human_readable_sizes->False
DEBUG: ConfigParser: list_md5->False
DEBUG: ConfigParser: log_target_prefix->
DEBUG: ConfigParser: preserve_attrs->True
DEBUG: ConfigParser: progress_meter->True
DEBUG: ConfigParser: proxy_host->
DEBUG: ConfigParser: proxy_port->0
DEBUG: ConfigParser: recursive->False
DEBUG: ConfigParser: recv_chunk->4096
DEBUG: ConfigParser: reduced_redundancy->False
DEBUG: ConfigParser: secret_key->EQ...37_chars...w
DEBUG: ConfigParser: send_chunk->4096
DEBUG: ConfigParser: simpledb_host->sdb.amazonaws.com
DEBUG: ConfigParser: skip_existing->False
DEBUG: ConfigParser: socket_timeout->300
DEBUG: ConfigParser: urlencoding_mode->normal
DEBUG: ConfigParser: use_https->False
DEBUG: ConfigParser: verbosity->WARNING
DEBUG: Updating Config.Config cache_file ->
DEBUG: Updating Config.Config encoding -> UTF-8
DEBUG: Updating Config.Config follow_symlinks -> False
DEBUG: Updating Config.Config verbosity -> 10
DEBUG: Unicodising 'sync' using UTF-8
DEBUG: Unicodising './s3cmd' using UTF-8
DEBUG: Unicodising 's3://mdomsch.1000-s3cmd-test/s3cmd' using UTF-8
DEBUG: Command: sync
INFO: Compiling list of local files...
DEBUG: DeUnicodising u's3cmd' using UTF-8
DEBUG: DeUnicodising u'.' using UTF-8
DEBUG: DeUnicodising u's3cmd' using UTF-8
DEBUG: Unicodising 's3cmd' using UTF-8
DEBUG: Unicodising './s3cmd' using UTF-8
DEBUG: Unicodising 's3cmd' using UTF-8
DEBUG: Unicodising 's3cmd' using UTF-8
INFO: Retrieving list of remote files for s3://mdomsch.1000-s3cmd-test/s3cmd ...
DEBUG: get_hostname(mdomsch.1000-s3cmd-test): mdomsch.1000-s3cmd-test.s3.amazonaws.com
DEBUG: String 's3cmd' encoded to 's3cmd'
DEBUG: SignHeaders: 'GET\n\n\n\nx-amz-date:Tue, 05 Mar 2013 20:40:14 +0000\n/mdomsch.1000-s3cmd-test/'
DEBUG: CreateRequest: resource[uri]=/
DEBUG: SignHeaders: 'GET\n\n\n\nx-amz-date:Tue, 05 Mar 2013 20:40:14 +0000\n/mdomsch.1000-s3cmd-test/'
DEBUG: Processing request, please wait...
DEBUG: Using existing connection
DEBUG: format_uri(): /?prefix=s3cmd
DEBUG: Sending request method_string='GET', uri='/?prefix=s3cmd', headers={'content-length': '0', 'Authorization': 'AWS AKIAJI6BJFODVWGHKXHA:J9IfBad6KzrcEc8ky1YW59Pqe9Q=', 'x-amz-date': 'Tue, 05 Mar 2013 20:40:14 +0000'}, body=(0 bytes)
DEBUG: Response: {'status': 200, 'headers': {'x-amz-id-2': 'hCl+q4psMQDQ5D8H2bDRMVk80PYBEIpwdLWa+4/j0j7nLttnZk0gPxwT5ELogpek', 'server': 'AmazonS3', 'transfer-encoding': 'chunked', 'x-amz-request-id': 'A763EA136B4E1135', 'date': 'Tue, 05 Mar 2013 20:40:15 GMT', 'content-type': 'application/xml'}, 'reason': 'OK', 'data': '\nmdomsch.1000-s3cmd-tests3cmd1000false'}
INFO: Found 1 local files, 0 remote files
INFO: Applying --exclude/--include
DEBUG: CHECK: s3cmd
DEBUG: PASS: u's3cmd'
INFO: Verifying attributes...
DEBUG: Comparing filelists (direction: local -> remote)
DEBUG: CHECK: s3cmd
INFO: Summary: 1 local files to upload, 0 files to remote copy, 0 remote files to delete
DEBUG: Unicodising 's3://mdomsch.1000-s3cmd-test/s3cmd' using UTF-8
DEBUG: attr_header: {'x-amz-meta-s3cmd-attrs': 'uid:500/gname:mdomsch/uname:mdomsch/gid:500/mode:33277/mtime:1362230700/atime:1362230700/md5:ce244b9166611a9811b568c0b05426e4/ctime:1362230700'}
DEBUG: String 's3cmd' encoded to 's3cmd'
DEBUG: SignHeaders: 'PUT\n\ntext/x-python; charset=us-ascii\n\nx-amz-date:Tue, 05 Mar 2013 20:40:14 +0000\nx-amz-meta-s3cmd-attrs:uid:500/gname:mdomsch/uname:mdomsch/gid:500/mode:33277/mtime:1362230700/atime:1362230700/md5:ce244b9166611a9811b568c0b05426e4/ctime:1362230700\n/mdomsch.1000-s3cmd-test/s3cmd'
DEBUG: CreateRequest: resource[uri]=/s3cmd
DEBUG: Unicodising './s3cmd' using UTF-8
DEBUG: SignHeaders: 'PUT\n\ntext/x-python; charset=us-ascii\n\nx-amz-date:Tue, 05 Mar 2013 20:40:14 +0000\nx-amz-meta-s3cmd-attrs:uid:500/gname:mdomsch/uname:mdomsch/gid:500/mode:33277/mtime:1362230700/atime:1362230700/md5:ce244b9166611a9811b568c0b05426e4/ctime:1362230700\n/mdomsch.1000-s3cmd-test/s3cmd'
./s3cmd -> s3://mdomsch.1000-s3cmd-test/s3cmd [1 of 1]
DEBUG: get_hostname(mdomsch.1000-s3cmd-test): mdomsch.1000-s3cmd-test.s3.amazonaws.com
DEBUG: format_uri(): /s3cmd
4096 of 96382 4% in 0s 68.72 kB/sDEBUG: Response: {'status': 200, 'headers': {'content-length': '0', 'x-amz-id-2': '3qhan84YgUhvQLX13t2oL8x70xIAaSqKUW2ZLwMRkwghWXPE8xSCz1os4QhdcnTX', 'server': 'AmazonS3', 'x-amz-request-id': '44C87512FA7C0D84', 'etag': '"ce244b9166611a9811b568c0b05426e4"', 'date': 'Tue, 05 Mar 2013 20:40:15 GMT'}, 'reason': 'OK', 'data': '', 'size': 96382}
96382 of 96382 100% in 0s 142.13 kB/s done
DEBUG: MD5 sums: computed=ce244b9166611a9811b568c0b05426e4, received="ce244b9166611a9811b568c0b05426e4"
Done. Uploaded 96382 bytes in 0.7 seconds, 132.83 kB/s. Copied 0 files saving 0 bytes transfer.

$ ./s3cmd sync --debug ./s3cmd s3://mdomsch.1000-s3cmd-test/s3cmd
DEBUG: ConfigParser: Reading file '/home/mdomsch/.s3cfg'
DEBUG: ConfigParser: access_key->AK...17_chars...A
DEBUG: ConfigParser: bucket_location->US
DEBUG: ConfigParser: cloudfront_host->cloudfront.amazonaws.com
DEBUG: ConfigParser: cloudfront_resource->/2010-07-15/distribution
DEBUG: ConfigParser: default_mime_type->binary/octet-stream
DEBUG: ConfigParser: delete_removed->False
DEBUG: ConfigParser: dry_run->False
DEBUG: ConfigParser: encoding->UTF-8
DEBUG: ConfigParser: encrypt->False
DEBUG: ConfigParser: follow_symlinks->False
DEBUG: ConfigParser: force->False
DEBUG: ConfigParser: get_continue->False
DEBUG: ConfigParser: gpg_command->/usr/bin/gpg
DEBUG: ConfigParser: gpg_decrypt->%(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
DEBUG: ConfigParser: gpg_encrypt->%(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
DEBUG: ConfigParser: gpg_passphrase->...-3_chars...
DEBUG: ConfigParser: guess_mime_type->True
DEBUG: ConfigParser: host_base->s3.amazonaws.com
DEBUG: ConfigParser: host_bucket->%(bucket)s.s3.amazonaws.com
DEBUG: ConfigParser: human_readable_sizes->False
DEBUG: ConfigParser: list_md5->False
DEBUG: ConfigParser: log_target_prefix->
DEBUG: ConfigParser: preserve_attrs->True
DEBUG: ConfigParser: progress_meter->True
DEBUG: ConfigParser: proxy_host->
DEBUG: ConfigParser: proxy_port->0
DEBUG: ConfigParser: recursive->False
DEBUG: ConfigParser: recv_chunk->4096
DEBUG: ConfigParser: reduced_redundancy->False
DEBUG: ConfigParser: secret_key->EQ...37_chars...w
DEBUG: ConfigParser: send_chunk->4096
DEBUG: ConfigParser: simpledb_host->sdb.amazonaws.com
DEBUG: ConfigParser: skip_existing->False
DEBUG: ConfigParser: socket_timeout->300
DEBUG: ConfigParser: urlencoding_mode->normal
DEBUG: ConfigParser: use_https->False
DEBUG: ConfigParser: verbosity->WARNING
DEBUG: Updating Config.Config cache_file ->
DEBUG: Updating Config.Config encoding -> UTF-8
DEBUG: Updating Config.Config follow_symlinks -> False
DEBUG: Updating Config.Config verbosity -> 10
DEBUG: Unicodising 'sync' using UTF-8
DEBUG: Unicodising './s3cmd' using UTF-8
DEBUG: Unicodising 's3://mdomsch.1000-s3cmd-test/s3cmd' using UTF-8
DEBUG: Command: sync
INFO: Compiling list of local files...
DEBUG: DeUnicodising u's3cmd' using UTF-8
DEBUG: DeUnicodising u'.' using UTF-8
DEBUG: DeUnicodising u's3cmd' using UTF-8
DEBUG: Unicodising 's3cmd' using UTF-8
DEBUG: Unicodising './s3cmd' using UTF-8
DEBUG: Unicodising 's3cmd' using UTF-8
DEBUG: Unicodising 's3cmd' using UTF-8
INFO: Retrieving list of remote files for s3://mdomsch.1000-s3cmd-test/s3cmd ...
DEBUG: get_hostname(mdomsch.1000-s3cmd-test): mdomsch.1000-s3cmd-test.s3.amazonaws.com
DEBUG: String 's3cmd' encoded to 's3cmd'
DEBUG: SignHeaders: 'GET\n\n\n\nx-amz-date:Tue, 05 Mar 2013 20:40:17 +0000\n/mdomsch.1000-s3cmd-test/'
DEBUG: CreateRequest: resource[uri]=/
DEBUG: SignHeaders: 'GET\n\n\n\nx-amz-date:Tue, 05 Mar 2013 20:40:17 +0000\n/mdomsch.1000-s3cmd-test/'
DEBUG: Processing request, please wait...
DEBUG: Using existing connection
DEBUG: format_uri(): /?prefix=s3cmd
DEBUG: Sending request method_string='GET', uri='/?prefix=s3cmd', headers={'content-length': '0', 'Authorization': 'AWS AKIAJI6BJFODVWGHKXHA:FjLXv6l0gfN0wJWd4ceK5GpVYHY=', 'x-amz-date': 'Tue, 05 Mar 2013 20:40:17 +0000'}, body=(0 bytes)
DEBUG: Response: {'status': 200, 'headers': {'x-amz-id-2': 'CQDMuuZiXTRMRnFpXTB+47BoE7D3LJ1Q0yP6jNhFV25PBnarB4PycEKueogXCJms', 'server': 'AmazonS3', 'transfer-encoding': 'chunked', 'x-amz-request-id': 'F658805B522B418F', 'date': 'Tue, 05 Mar 2013 20:40:18 GMT', 'content-type': 'application/xml'}, 'reason': 'OK', 'data': '\nmdomsch.1000-s3cmd-tests3cmd1000falses3cmd2013-03-05T20:40:15.000Z"ce244b9166611a9811b568c0b05426e4"96382aeae39dcf6ebdee938eddfc9f15e450c1268e90bfe572482a08d720452ad38bcmattSTANDARD'}
INFO: Found 1 local files, 1 remote files
INFO: Applying --exclude/--include
DEBUG: CHECK: s3cmd
DEBUG: PASS: u's3cmd'
INFO: Verifying attributes...
DEBUG: Comparing filelists (direction: local -> remote)
DEBUG: CHECK: s3cmd

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
An unexpected error has occurred.
Please report the following lines to:
s3tools-bugs@lists.sourceforge.net
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Problem: AttributeError: 'dict' object has no attribute 'get_md5'
S3cmd: 1.5.0-alpha2

Traceback (most recent call last):
File "./s3cmd", line 2063, in
main()
File "./s3cmd", line 2004, in main
cmd_func(args)
File "./s3cmd", line 1170, in cmd_sync
return cmd_sync_local2remote(args)
File "./s3cmd", line 1154, in cmd_sync_local2remote
destination_base_uri = _single_process(local_list)
File "./s3cmd", line 977, in _single_process
_child(destination_base, local_list)
File "./s3cmd", line 1063, in _child
local_list, remote_list, update_list, copy_pairs = compare_filelists(local_list, remote_list, src_remote = False, dst_remote = True, delay_updates = cfg.delay_updates)
File "/home/mdomsch/git/s3cmd/S3/FileLists.py", line 457, in compare_filelists
same_file = _compare(src_list, dst_list, src_remote, dst_remote, relative_file)
File "/home/mdomsch/git/s3cmd/S3/FileLists.py", line 417, in _compare
dst_md5 = dst_list.get_md5(file)
AttributeError: 'dict' object has no attribute 'get_md5'

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
An unexpected error has occurred.
Please report the above lines to:
s3tools-bugs@lists.sourceforge.net
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
'''

@mdomsch
Member
mdomsch commented Mar 5, 2013
   if single_file_local and len(local_list) == 1 and len(remote_list) == 1:
        ## Make remote_key same as local_key for comparison if we're dealing with only one file
        remote_list_entry = remote_list[remote_list.keys()[0]]
        # Flush remote_list, by the way
        remote_list = { local_list.keys()[0] : remote_list_entry }

this s wrong - remote list is now a dict when it should be a SortedDict()

@mdomsch mdomsch added a commit to mdomsch/s3cmd that referenced this issue Mar 5, 2013
@mdomsch @mdomsch mdomsch + mdomsch fix single file sync local2remote traceback, bug #128
Fixes s3tools#128

In the single file transfer case, the remote_list was getting created
as a dict, rather than as a SortedDict as it should be.
5631c00
@mdomsch
Member
mdomsch commented Mar 7, 2013

merged.

@mdomsch mdomsch closed this Mar 7, 2013
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment