UnicodeDecodeError - glacier uploads fail using vault objects from Layer2()'s get vault

[tfgm-bud]

Layer2's get_vault() works by calling layer1.describe_vault() to get data about the vault in the form of response_data which is passed as a parameter to create a new Vault() object . The problem is that response_data contains unicode strings which are used to set attributes (like the vault name) for the new Vault() object.

Later, when calling create_archive_from_file() on this Vault() object the upload fails with a UnicodeDecodeError. The issue is, when forming the URI for the archive, the URI string becomes unicode due to the unicode vault name getting concatenated into the URI, then the message becomes unicode due to presence of the URI and then the whole thing fails when a message_body with binary data is concatenated with the msg header in HTTPConnection's _send_output() method.

Similar issues may occur in other places where the response containing unicode is used to create objects that are subsequently used to upload to Glacier.

Not sure of the best way to fix this but as a workaround:

Workaround:

cast the name field of the Vault object to a str after creating it using get_vault():

vault = connection.get_vault(self.args.vault)
vault.name = str(vault.name)

Reproducing

I observed this initially using https://github.com/basak/glacier-cli to upload binary (zip/tar) files

Installed boto (boto-2.38.0) from pip:
sudo -H pip install boto
cloned glacier-cli with:
git clone git://github.com/basak/glacier-cli.git

Commands to reproduce:

cd glacier-cli
wget https://github.com/basak/glacier-cli/archive/master.zip
python ./glacier.py  vault create test
python ./glacier.py archive upload test master.zip

Result:

$ python ./glacier.py archive upload test master.zip
Traceback (most recent call last):
  File "./glacier.py", line 730, in <module>
    App().main()
  File "./glacier.py", line 716, in main
    self.args.func()
  File "./glacier.py", line 498, in archive_upload
    file_obj=self.args.file, description=name)
  File "/usr/local/lib/python2.7/dist-packages/boto/glacier/vault.py", line 178, in create_archive_from_file
    writer.close()
  File "/usr/local/lib/python2.7/dist-packages/boto/glacier/writer.py", line 228, in close
    self.partitioner.flush()
  File "/usr/local/lib/python2.7/dist-packages/boto/glacier/writer.py", line 79, in flush
    self._send_part()
  File "/usr/local/lib/python2.7/dist-packages/boto/glacier/writer.py", line 75, in _send_part
    self.send_fn(part)
  File "/usr/local/lib/python2.7/dist-packages/boto/glacier/writer.py", line 222, in _upload_part
    self.uploader.upload_part(self.next_part_index, part_data)
  File "/usr/local/lib/python2.7/dist-packages/boto/glacier/writer.py", line 129, in upload_part
    content_range, part_data)
  File "/usr/local/lib/python2.7/dist-packages/boto/glacier/layer1.py", line 1279, in upload_part
    response_headers=response_headers)
  File "/usr/local/lib/python2.7/dist-packages/boto/glacier/layer1.py", line 114, in make_request
    data=data)
  File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 1071, in make_request
    retry_handler=retry_handler)
  File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 943, in _mexe
    request.body, request.headers)
  File "/usr/lib/python2.7/httplib.py", line 979, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 1013, in _send_request
    self.endheaders(body)
  File "/usr/lib/python2.7/httplib.py", line 975, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 833, in _send_output
    msg += message_body
UnicodeDecodeError: 'ascii' codec can't decode byte 0xfd in position 10: ordinal not in range(128)

[aries1980]

I can confirm the "vault.name = str(vault.name)" trick works for me.