Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Modifications for python3 support. #1345

Closed
wants to merge 53 commits into from

5 participants

Toby Burress Mike Schwartz Mitch Garnaat Oleg Anashkin Aaron Maxwell
Toby Burress

This creates a subdirectory under which the code can be converted to python3 compatible with 2to3.

It also makes several changes to the existing code base -- some substantial -- mostly to ensure that strings are in the correct format for python3.

redsymbol and others added some commits
Aaron Maxwell redsymbol Infrastructure and initial dev docs for python 3 porting work 1cebc38
Toby Burress kurin Fix rfc822 import errors in python3 3c1be09
Toby Burress kurin Intercept calls to the hmac module
This monkeypatches the hmac module so that calls to .new() and
.update() are done, in python3, with actual bytes objects.
c536708
Toby Burress kurin Modify sha256 always to pass bytes in python3
The same as the previous patch.

Also this patch monkeypatches hashlib.  This code should probably
be pulled and put elsewhere, but I don't know where else is good.
b2f5899
Toby Burress kurin Patch the sax parser for python3 strings. 82c3dde
Toby Burress kurin Ensure byte strings actually are byte strings 6e29ec0
Toby Burress kurin Change the rounding type
From what I can tell the rounding type doesn't matter, but who knows
e981e09
Toby Burress kurin Convert to bytes 0ebadc1
Toby Burress kurin cmp is deprecated, use cmp_to_key bea296b
Toby Burress kurin Change hex and string conversions.
The vault code at one point (line 59, here) makes a point of encoding
unicode strings into utf-8.  This breaks the unit tests in python3,
which aren't explicitly expecting byte strings.  They almost certainly
should be, but I didn't feel qualified to make such sweeping changes
to the unit tests, so for now I've put in a workaround that causes
python3 strings to remain as they are.

Also, python3 no longer has str.encode('hex')
ed79683
Toby Burress kurin Change the Binary class and _encode_s method
Bring these in line with the other python3 encoding changes.
1bffbd7
Toby Burress kurin Tag objects cannot also be None
This I think is a straight-up bug.
284b660
Toby Burress kurin Change the test for sets of strings.
The original code assumed that iterating through a set() would throw
up the contents in a predictable order.  This is not the case, and
in python3 apparently these specific strings are returned in the
opposite order.
d9cac2f
Toby Burress kurin Turn a string into a byte literal.
This is required to work in python3.  It should not affect the test.
40f170d
Toby Burress kurin Change the way the hex is expressed.
python3 doesn't have str.encode('hex'), and reaches those values
with a different expression.  I have replaced the longhand with the
specific values, which should not alter the test.
51f05b1
Toby Burress kurin Modify the test to also accept python3 exceptions.
In python3, IOError (which is apparently also OSError) does not
have a message attribute.
bf9732f
Toby Burress kurin Pass bytes to md5(), not a string, in python3. aef21ec
Toby Burress kurin Assist 2to3 in dealing with unittest.mock
Apparently "import mock" and "from mock import whatever" are used
all over the place.  As this is pretty experimental, I've just done
a simple find job to fix it.  This works for FreeBSD's sed(1) but
hasn't been checked against whatever else is out there.
1ea575e
Toby Burress kurin Adjust the output of base64.encodestring in p3k
Apparently a string of the form "%s:%s" % (x, y), when y is a byte
literal, will be formatted as "this-is-x:b'this-is-y'", which
obviously AWS isn't expecting.  So this patch is further insanity
to ensure that the output of base64.encodestring is a utf-8 encoded
string, not a byte literal.
e079c77
Toby Burress kurin Python3 needs BytesIO not StringIO
Doubtless I'll have to change this in a number of places.
9f53713
Toby Burress kurin *Don't* encode each unicode header.
This probably breaks stuff, which is why it's only commented out.
0e915fd
Toby Burress kurin Again, BytesIO and not StringIO 1f57139
Toby Burress kurin Re-dencode encoded and quoted header strings. b034808
Toby Burress kurin Only encode headers that have non-ascii characters.
So, we have to be even clever-er to avoid running certain headers
(namely the MD5 checksup) through urllib.quote_plus.

Honestly at this point we should probably just check for it explicitly.
0daf224
Toby Burress kurin Remove a hack.
So in 2.5 and below, apparently "host" had to be amended to "host:port"
for compatibility with httplib.  The code would check to see if we
were running 2.6 and 2.7, which in 3.3 we *aren't*, but I guess it
never got the memo.

2.5 and older aren't supported now anyway, so remove the check entirely.
f4b1452
Toby Burress kurin Properly handle the mock -> unittest.mock conversion. 8cf4f8e
Toby Burress kurin python 2.6 doesn't have functools.cmp_to_key 739261a
Toby Burress kurin Clean up the exception assertions. 964e4b9
Toby Burress kurin Merge branch 'develop' of github.com:boto/boto into develop 70ec23c
Toby Burress kurin Fix tests to conform to python3
httpretty returns byte literals, which need to be decoded
3c03080
Toby Burress kurin Fixes to handle byte literal returns 2915451
Toby Burress kurin now with httpretty
Which does support python3, thank goodness
d2962ca
Mike Schwartz
Collaborator

Hi Toby,

I notice the list of files where you made mods doesn't include any of the Google Cloud Storage code (boto/boto/gs/)
We'd like to also get the GCS code to work under Python 3. We could help with that work (unless you'd prefer to make all the changes in this CL, to keep them uniform).

Thanks,

Mike Schwartz

Toby Burress

Sure, I'd be happy to look at that. The commits here really only bring the unit tests and my personal use of S3 and EC2 into line; I hadn't looked at GCS (or any other module) yet.

Mike Schwartz
Collaborator
kurin added some commits
Toby Burress kurin Another instance of StringIO -> BytesIO
I should probably put a function in utils.py but this pattern shows
up rarely enough.
9bf91c4
Toby Burress kurin Merge remote-tracking branch 'upstream/develop' into develop
Keeping up with upstream.
fe97740
Toby Burress

I don't know how that merge made it into the pull request; I'll probably have to close this one and open a new one.

But I can now do a gsutil perfdiag without failures, which I don't know how much of GCS's functionality that uses, but it's at least some of it.

Toby Burress

The s3 integration tests now pass successfully in 2.6, 2.7, and 3.3.

Mitch Garnaat
Owner

I was just trying this out but ran into an issue with pyami/config.py. I don't think that's been updated yet. I'm wondering how you are running your integration tests without it, though.

Also, we should look at some of the existing directories and decide whether they need to be there anymore or not. For example, boto/roboto, boto/manage, boto/mashups, etc. that are really old and not really used. This would be a good time to break those out as separate packages or just get rid of them altogether. Certainly no reason to try to port them to Python 3.x.

Toby Burress

So far I've only done the unit tests and the s3 integration tests. I was going to tackle ec2's integration tests next; I haven't touched anything under pyami/.

Toby Burress

I also don't want to wholesale remove any parts of the library, because I don't think I'm qualified to make that decision. But I mean if they happened to disappear the next time I merged from develop, I wouldn't mourn them.

kurin added some commits
Toby Burress kurin Merge remote-tracking branch 'upstream/develop' into develop 9470a4f
Toby Burress kurin Add port to server_name method.
Apparently this is used to create HTTP objects, instead of passing
the port explicitly.
2dbd632
Toby Burress kurin Update test to expect bytes. 139f0f4
Toby Burress kurin Update the text/bytes handling in SQS messages.
SQS message are always unicode, but they have to be converted to/from
bytes to make it through the base64 encoder.
347031b
Toby Burress kurin Modify setup.py to build both python3 and python2
Now calls to `setup.py bdist`, in python3, will run 2to3 on the
source.  This requires a pretty recent install of Distribute (0.6.31
works)

This will allow a single source package to install in both python2
and python3.
85b9f90
Toby Burress

For the record, this now also passes the ec2 and sqs integration tests.

It also has a modification to setup.py so that when installed by python3, it automatically converts with 2to3. This way you can distribute a single source dist.

Mike Schwartz
Collaborator
Oleg Anashkin

+1 to get boto working on python 3.3.

Toby Burress

Hi extesy, there's actually a branch now for python3, py3kport. It's a few bugfixes behind develop but nothing major. Please check it out and let me know if you run into any issues. You can build it by running setup.py install (it will automatically run 2to3 when installing) and the 3.3 requirements are in py3kport/requirements.txt-3.3. Let me know if you hit any bugs!

Oleg Anashkin

@kurin Thanks for your work on this port. The problem with manually building from a separate branch is that it's not updated as often as a master. I would really prefer to simply use pip for install and get all new fixes as they are released.

@garnaat I see that you have just pulled the latest changes into py3kport branch. Are you going to merge everything into master any time soon?

Toby Burress

Yeah. It's pretty close to that now, but some patches coming in to develop need some massaging to get them to work in python3. I don't know when the python3 changes (which, in sum, aren't really trivial, although new modifications are) are going to make it into develop, but for my part I'll try merging from develop into py3kport every release or so.

Toby Burress

Also also, this pull request is no longer valid. Python3 support should be pulled in from the py3kport branch now.

Toby Burress kurin closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Jan 29, 2013
  1. Aaron Maxwell
Commits on Feb 10, 2013
  1. Toby Burress
  2. Toby Burress

    Intercept calls to the hmac module

    kurin authored
    This monkeypatches the hmac module so that calls to .new() and
    .update() are done, in python3, with actual bytes objects.
  3. Toby Burress

    Modify sha256 always to pass bytes in python3

    kurin authored
    The same as the previous patch.
    
    Also this patch monkeypatches hashlib.  This code should probably
    be pulled and put elsewhere, but I don't know where else is good.
  4. Toby Burress
  5. Toby Burress
  6. Toby Burress

    Change the rounding type

    kurin authored
    From what I can tell the rounding type doesn't matter, but who knows
  7. Toby Burress

    Convert to bytes

    kurin authored
  8. Toby Burress
  9. Toby Burress

    Change hex and string conversions.

    kurin authored
    The vault code at one point (line 59, here) makes a point of encoding
    unicode strings into utf-8.  This breaks the unit tests in python3,
    which aren't explicitly expecting byte strings.  They almost certainly
    should be, but I didn't feel qualified to make such sweeping changes
    to the unit tests, so for now I've put in a workaround that causes
    python3 strings to remain as they are.
    
    Also, python3 no longer has str.encode('hex')
  10. Toby Burress

    Change the Binary class and _encode_s method

    kurin authored
    Bring these in line with the other python3 encoding changes.
  11. Toby Burress

    Tag objects cannot also be None

    kurin authored
    This I think is a straight-up bug.
  12. Toby Burress

    Change the test for sets of strings.

    kurin authored
    The original code assumed that iterating through a set() would throw
    up the contents in a predictable order.  This is not the case, and
    in python3 apparently these specific strings are returned in the
    opposite order.
  13. Toby Burress

    Turn a string into a byte literal.

    kurin authored
    This is required to work in python3.  It should not affect the test.
  14. Toby Burress

    Change the way the hex is expressed.

    kurin authored
    python3 doesn't have str.encode('hex'), and reaches those values
    with a different expression.  I have replaced the longhand with the
    specific values, which should not alter the test.
  15. Toby Burress

    Modify the test to also accept python3 exceptions.

    kurin authored
    In python3, IOError (which is apparently also OSError) does not
    have a message attribute.
  16. Toby Burress
  17. Toby Burress

    Assist 2to3 in dealing with unittest.mock

    kurin authored
    Apparently "import mock" and "from mock import whatever" are used
    all over the place.  As this is pretty experimental, I've just done
    a simple find job to fix it.  This works for FreeBSD's sed(1) but
    hasn't been checked against whatever else is out there.
  18. Toby Burress

    Adjust the output of base64.encodestring in p3k

    kurin authored
    Apparently a string of the form "%s:%s" % (x, y), when y is a byte
    literal, will be formatted as "this-is-x:b'this-is-y'", which
    obviously AWS isn't expecting.  So this patch is further insanity
    to ensure that the output of base64.encodestring is a utf-8 encoded
    string, not a byte literal.
  19. Toby Burress

    Python3 needs BytesIO not StringIO

    kurin authored
    Doubtless I'll have to change this in a number of places.
  20. Toby Burress

    *Don't* encode each unicode header.

    kurin authored
    This probably breaks stuff, which is why it's only commented out.
Commits on Feb 14, 2013
  1. Toby Burress

    Again, BytesIO and not StringIO

    kurin authored
  2. Toby Burress
  3. Toby Burress

    Only encode headers that have non-ascii characters.

    kurin authored
    So, we have to be even clever-er to avoid running certain headers
    (namely the MD5 checksup) through urllib.quote_plus.
    
    Honestly at this point we should probably just check for it explicitly.
  4. Toby Burress

    Remove a hack.

    kurin authored
    So in 2.5 and below, apparently "host" had to be amended to "host:port"
    for compatibility with httplib.  The code would check to see if we
    were running 2.6 and 2.7, which in 3.3 we *aren't*, but I guess it
    never got the memo.
    
    2.5 and older aren't supported now anyway, so remove the check entirely.
Commits on Feb 21, 2013
  1. Toby Burress
  2. Toby Burress
  3. Toby Burress
  4. Toby Burress
  5. Toby Burress

    Fix tests to conform to python3

    kurin authored
    httpretty returns byte literals, which need to be decoded
  6. Toby Burress
  7. Toby Burress

    now with httpretty

    kurin authored
    Which does support python3, thank goodness
Commits on Feb 22, 2013
  1. Toby Burress

    Another instance of StringIO -> BytesIO

    kurin authored
    I should probably put a function in utils.py but this pattern shows
    up rarely enough.
  2. Toby Burress

    Merge remote-tracking branch 'upstream/develop' into develop

    kurin authored
    Keeping up with upstream.
  3. Toby Burress
Commits on Feb 23, 2013
  1. Toby Burress
Commits on Feb 24, 2013
  1. Toby Burress
  2. Toby Burress

    More encoding/decoding drama.

    kurin authored
  3. Toby Burress
  4. Toby Burress

    Workaround for python3

    kurin authored
    Apparently in python3 exception variables get unbound when leaving
    the except: scope.
  5. Toby Burress
  6. Toby Burress

    Ensure that bytes are bytes.

    kurin authored
  7. Toby Burress
  8. Toby Burress
Commits on Feb 27, 2013
  1. Toby Burress
  2. Toby Burress

    Add port to server_name method.

    kurin authored
    Apparently this is used to create HTTP objects, instead of passing
    the port explicitly.
  3. Toby Burress

    Update test to expect bytes.

    kurin authored
  4. Toby Burress

    Update the text/bytes handling in SQS messages.

    kurin authored
    SQS message are always unicode, but they have to be converted to/from
    bytes to make it through the base64 encoder.
Commits on Feb 28, 2013
  1. Toby Burress

    Modify setup.py to build both python3 and python2

    kurin authored
    Now calls to `setup.py bdist`, in python3, will run 2to3 on the
    source.  This requires a pretty recent install of Distribute (0.6.31
    works)
    
    This will allow a single source package to install in both python2
    and python3.
Commits on Mar 2, 2013
  1. Toby Burress
  2. Toby Burress
  3. Toby Burress

    Errors come back as bytes.

    kurin authored
Commits on Mar 10, 2013
  1. Toby Burress
This page is out of date. Refresh to see the latest.
Showing with 552 additions and 150 deletions.
  1. +2 −0  .gitignore
  2. +1 −0  MANIFEST.in
  3. +2 −0  README.rst
  4. +4 −1 boto/__init__.py
  5. +32 −0 boto/auth.py
  6. +6 −1 boto/cloudsearch/document.py
  7. +2 −1  boto/cloudsearch/search.py
  8. +9 −0 boto/compat.py
  9. +22 −18 boto/connection.py
  10. +8 −3 boto/dynamodb/types.py
  11. +6 −1 boto/glacier/utils.py
  12. +6 −2 boto/glacier/vault.py
  13. +2 −2 boto/gs/key.py
  14. +4 −0 boto/https_connection.py
  15. +11 −8 boto/s3/bucket.py
  16. +17 −5 boto/s3/key.py
  17. +2 −0  boto/s3/tagging.py
  18. +11 −10 boto/ses/connection.py
  19. +3 −2 boto/sqs/message.py
  20. +54 −5 boto/utils.py
  21. +1 −0  boto/version.txt
  22. +135 −0 py3kport/README.txt
  23. +46 −0 py3kport/rebuild.sh
  24. +6 −0 py3kport/requirements.txt-3.3
  25. +15 −6 setup.py
  26. +3 −3 tests/integration/s3/mock_storage_service.py
  27. +1 −1  tests/integration/s3/test_bucket.py
  28. +5 −5 tests/integration/s3/test_connection.py
  29. +2 −2 tests/integration/s3/test_encryption.py
  30. +14 −14 tests/integration/s3/test_key.py
  31. +1 −1  tests/integration/s3/test_mfa.py
  32. +5 −5 tests/integration/s3/test_multipart.py
  33. +0 −2  tests/integration/s3/test_pool.py
  34. +2 −2 tests/integration/s3/test_versioning.py
  35. +4 −1 tests/unit/__init__.py
  36. +4 −1 tests/unit/auth/test_sigv4.py
  37. +4 −1 tests/unit/cloudformation/test_connection.py
  38. +4 −1 tests/unit/cloudfront/test_invalidation_list.py
  39. +10 −7 tests/unit/cloudsearch/test_document.py
  40. +1 −1  tests/unit/cloudsearch/test_search.py
  41. +4 −1 tests/unit/dynamodb/test_layer2.py
  42. +3 −2 tests/unit/dynamodb/test_types.py
  43. +4 −1 tests/unit/ec2/test_address.py
  44. +4 −1 tests/unit/ec2/test_blockdevicemapping.py
  45. +4 −1 tests/unit/ec2/test_instance.py
  46. +4 −1 tests/unit/ec2/test_volume.py
  47. +4 −1 tests/unit/glacier/test_concurrent.py
  48. +4 −1 tests/unit/glacier/test_job.py
  49. +1 −1  tests/unit/glacier/test_layer1.py
  50. +5 −2 tests/unit/glacier/test_layer2.py
  51. +6 −2 tests/unit/glacier/test_vault.py
  52. +14 −6 tests/unit/glacier/test_writer.py
  53. +4 −1 tests/unit/provider/test_provider.py
  54. +4 −1 tests/unit/sns/test_connection.py
  55. +4 −1 tests/unit/sqs/test_queue.py
  56. +16 −15 tests/unit/test_connection.py
2  .gitignore
View
@@ -12,3 +12,5 @@ MANIFEST
.coverage
*flymake.py
venv
+# for python 3 port
+py3kport/build*
1  MANIFEST.in
View
@@ -1,3 +1,4 @@
+include boto/version.txt
include boto/cacerts/cacerts.txt
include README.rst
include Changelog.rst
2  README.rst
View
@@ -1,3 +1,5 @@
+** Py3k Porting: See py3kport/README.txt **
+
####
boto
####
5 boto/__init__.py
View
@@ -36,7 +36,10 @@
import urlparse
from boto.exception import InvalidUriError
-__version__ = '2.8.0-dev'
+vfile = os.path.join(os.path.dirname(os.path.abspath(boto.__file__ )), "version.txt")
+with open(vfile) as f:
+ __version__ = f.read().strip()
+
Version = __version__ # for backware compatibility
UserAgent = 'Boto/%s (%s)' % (__version__, sys.platform)
32 boto/auth.py
View
@@ -71,6 +71,38 @@ def new(self, *args, **kwargs):
import sha
sha256 = None
+# and this modifies sha and sha256 to accept the proper type
+sha256 = boto.utils.wrap_hash_function(sha256)
+try:
+ # this doesn't belong here
+ import hashlib
+ hashlib.sha256 = sha256
+except ImportError:
+ pass
+
+# this code intercepts calls to hmac.new and hmac.update
+# this fixes an issue in python3 where string objects get sent where
+# bytes are expected
+
+def fakenew(*args, **kwargs):
+ args = list(args)
+ args[0] = boto.utils.ensure_bytes(args[0])
+ return hmac._realnew(*args, **kwargs)
+hmac._realnew = hmac.new
+hmac.new = fakenew
+
+def fakeupdate(*args, **kwargs):
+ args = list(args)
+ args[1] = boto.utils.ensure_bytes(args[1])
+ return args[0]._realupdate(*args[1:], **kwargs)
+hmac.HMAC._realupdate = hmac.HMAC.update
+hmac.HMAC.update = fakeupdate
+
+# ...and patch base64 too
+def fakeencodestring(*args, **kwargs):
+ return boto.utils.ensure_string(base64._realencodestring(*args, **kwargs))
+base64._realencodestring = base64.encodestring
+base64.encodestring = fakeencodestring
class HmacKeys(object):
"""Key based Auth handler helper."""
7 boto/cloudsearch/document.py
View
@@ -221,7 +221,12 @@ def __init__(self, response, doc_service, sdf):
self.sdf = sdf
try:
- self.content = json.loads(response.content)
+ if hasattr(response.content, 'decode') and not hasattr(response.content, 'encode'):
+ # decode the python3 byte literal
+ rc = response.content.decode('utf-8')
+ else:
+ rc = response.content
+ self.content = json.loads(rc)
except:
boto.log.error('Error indexing documents.\nResponse Content:\n{0}\n\n'
'SDF:\n{1}'.format(response.content, self.sdf))
3  boto/cloudsearch/search.py
View
@@ -288,7 +288,8 @@ def __call__(self, query):
params = query.to_params()
r = requests.get(url, params=params)
- data = json.loads(r.content)
+ content = r.content.decode('utf-8')
+ data = json.loads(content)
data['query'] = query
data['search_service'] = self
9 boto/compat.py
View
@@ -26,3 +26,12 @@
import simplejson as json
except ImportError:
import json
+
+json._loads = json.loads
+def loads(*args, **kwargs):
+ if args:
+ args = list(args)
+ if hasattr(args[0], 'decode') and not hasattr(args[0], 'encode'):
+ args[0] = args[0].decode('utf-8')
+ return json._loads(*args, **kwargs)
+json.loads = loads
40 boto/connection.py
View
@@ -95,6 +95,14 @@
DEFAULT_CA_CERTS_FILE = os.path.join(os.path.dirname(os.path.abspath(boto.cacerts.__file__ )), "cacerts.txt")
+# sigh
+# monkeypatch xml.sax.parseString
+def ps(*args, **kwargs):
+ args = list(args)
+ args[0] = boto.utils.ensure_bytes(args[0])
+ xml.sax._parseString(*args, **kwargs)
+xml.sax._parseString = xml.sax.parseString
+xml.sax.parseString = ps
class HostConnectionPool(object):
@@ -369,8 +377,14 @@ def __str__(self):
def authorize(self, connection, **kwargs):
for key in self.headers:
val = self.headers[key]
- if isinstance(val, unicode):
- self.headers[key] = urllib.quote_plus(val.encode('utf-8'))
+ if len([x for x in val if ord(x) > 127]):
+ # this is a unicode string with non-ascii characters
+ v = val.encode("utf-8")
+ v = urllib.quote_plus(v)
+ if not hasattr(v, 'encode'):
+ # v is a python3 byte literal; we need to deocde it into unicode again
+ v = v.decode("utf-8")
+ self.headers[key] = v
connection._auth_handler.add_auth(self, **kwargs)
@@ -618,21 +632,10 @@ def get_path(self, path='/'):
def server_name(self, port=None):
if not port:
port = self.port
- if port == 80:
+ if port == 80 or port == 443:
signature_host = self.host
else:
- # This unfortunate little hack can be attributed to
- # a difference in the 2.6 version of httplib. In old
- # versions, it would append ":443" to the hostname sent
- # in the Host header and so we needed to make sure we
- # did the same when calculating the V2 signature. In 2.6
- # (and higher!)
- # it no longer does that. Hence, this kludge.
- if ((ON_APP_ENGINE and sys.version[:3] == '2.5') or
- sys.version[:3] in ('2.6', '2.7')) and port == 443:
- signature_host = self.host
- else:
- signature_host = '%s:%d' % (self.host, port)
+ signature_host = "%s:%d"%(self.host, port)
return signature_host
def handle_proxy(self, proxy, proxy_port, proxy_user, proxy_pass):
@@ -815,7 +818,7 @@ def _mexe(self, request, sender=None, override_num_retries=None,
boto.log.debug('Host: %s' % request.host)
response = None
body = None
- e = None
+ ex = None
if override_num_retries is None:
num_retries = config.getint('Boto', 'num_retries', self.num_retries)
else:
@@ -874,6 +877,7 @@ def _mexe(self, request, sender=None, override_num_retries=None,
response = None
continue
except self.http_exceptions, e:
+ ex = e # e will be unset after leaving the except: block
for unretryable in self.http_unretryable_exceptions:
if isinstance(e, unretryable):
boto.log.debug(
@@ -892,8 +896,8 @@ def _mexe(self, request, sender=None, override_num_retries=None,
# Otherwise, raise the exception that must have already h#appened.
if response:
raise BotoServerError(response.status, response.reason, body)
- elif e:
- raise e
+ elif ex:
+ raise ex
else:
msg = 'Please report this exception as a Boto Issue!'
raise BotoClientError(msg)
11 boto/dynamodb/types.py
View
@@ -27,11 +27,12 @@
import base64
from decimal import (Decimal, DecimalException, Context,
Clamped, Overflow, Inexact, Underflow, Rounded)
+import decimal
from exceptions import DynamoDBNumberError
DYNAMODB_CONTEXT = Context(
- Emin=-128, Emax=126, rounding=None, prec=38,
+ Emin=-128, Emax=126, rounding=decimal.ROUND_HALF_EVEN, prec=38,
traps=[Clamped, Overflow, Inexact, Rounded, Underflow])
@@ -136,10 +137,13 @@ def dynamize_value(val):
class Binary(object):
def __init__(self, value):
+ if isinstance(value, str) and not hasattr(value, 'decode'):
+ # python3 strings ONLY
+ value = bytes([ord(x) for x in value])
self.value = value
def encode(self):
- return base64.b64encode(self.value)
+ return base64.b64encode(self.value).decode('utf-8')
def __eq__(self, other):
if isinstance(other, Binary):
@@ -250,7 +254,8 @@ def _encode_n(self, attr):
raise DynamoDBNumberError(msg)
def _encode_s(self, attr):
- if isinstance(attr, unicode):
+ if isinstance(attr, unicode) and hasattr(attr, 'decode'):
+ # is a python2 unicode string ONLY
attr = attr.encode('utf-8')
elif not isinstance(attr, str):
attr = str(attr)
7 boto/glacier/utils.py
View
@@ -22,6 +22,8 @@
import hashlib
import math
+# is this kosher?
+import boto.utils
_MEGABYTE = 1024 * 1024
DEFAULT_PART_SIZE = 4 * _MEGABYTE
@@ -69,6 +71,7 @@ def minimum_part_size(size_in_bytes, default_part_size=DEFAULT_PART_SIZE):
def chunk_hashes(bytestring, chunk_size=_MEGABYTE):
+ bytestring = boto.utils.ensure_bytes(bytestring)
chunk_count = int(math.ceil(len(bytestring) / float(chunk_size)))
hashes = []
for i in xrange(chunk_count):
@@ -125,7 +128,7 @@ def compute_hashes_from_fileobj(fileobj, chunk_size=1024 * 1024):
chunks = []
chunk = fileobj.read(chunk_size)
while chunk:
- linear_hash.update(chunk)
+ linear_hash.update(boto.utils.ensure_bytes(chunk))
chunks.append(hashlib.sha256(chunk).digest())
chunk = fileobj.read(chunk_size)
if not chunks:
@@ -134,6 +137,8 @@ def compute_hashes_from_fileobj(fileobj, chunk_size=1024 * 1024):
def bytes_to_hex(str_as_bytes):
+ if isinstance(str_as_bytes[0], int):
+ return ''.join(["%02x" % x for x in str_as_bytes]).strip()
return ''.join(["%02x" % ord(x) for x in str_as_bytes]).strip()
8 boto/glacier/vault.py
View
@@ -54,7 +54,8 @@ def __init__(self, layer1, response_data=None):
if response_data:
for response_name, attr_name, default in self.ResponseDataElements:
value = response_data[response_name]
- if isinstance(value, unicode):
+ if isinstance(value, unicode) and hasattr(value, 'decode'):
+ # python2 unicode strings only
value = value.encode('utf8')
setattr(self, attr_name, value)
else:
@@ -228,7 +229,10 @@ def resume_archive_from_file(self, upload_id, filename=None,
for part_desc in part_list_response['Parts']:
part_index = self._range_string_to_part_index(
part_desc['RangeInBytes'], part_size)
- part_tree_hash = part_desc['SHA256TreeHash'].decode('hex')
+ if hasattr(part_desc['SHA256TreeHash'], 'decode'):
+ part_tree_hash = part_desc['SHA256TreeHash'].decode('hex')
+ else:
+ part_tree_hash = bytes.fromhex(part_desc['SHA256TreeHash'])
part_hash_map[part_index] = part_tree_hash
if not file_obj:
4 boto/gs/key.py
View
@@ -23,7 +23,7 @@
import binascii
import os
import re
-import StringIO
+import io
from boto.exception import BotoClientError
from boto.s3.key import Key as S3Key
from boto.s3.keyfile import KeyFile
@@ -500,7 +500,7 @@ def set_contents_from_string(self, s, headers=None, replace=True,
if isinstance(s, unicode):
s = s.encode("utf-8")
- fp = StringIO.StringIO(s)
+ fp = io.BytesIO(s)
r = self.set_contents_from_file(fp, headers, replace, cb, num_cb,
policy, md5,
if_generation=if_generation)
4 boto/https_connection.py
View
@@ -25,6 +25,7 @@
import ssl
import boto
+from boto.utils import ensure_bytes
class InvalidCertificateException(httplib.HTTPException):
"""Raised when a certificate is provided with an invalid hostname."""
@@ -103,6 +104,9 @@ def __init__(self, host, port=default_port, key_file=None, cert_file=None,
self.cert_file = cert_file
self.ca_certs = ca_certs
+ def send(self, data):
+ return httplib.HTTPConnection.send(self, ensure_bytes(data))
+
def connect(self):
"Connect to a host on a given (SSL) port."
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
19 boto/s3/bucket.py
View
@@ -45,7 +45,7 @@
import boto.utils
import xml.sax
import xml.sax.saxutils
-import StringIO
+import io
import urllib
import re
import base64
@@ -315,7 +315,8 @@ def _get_all(self, element_map, initial_query_string='',
k = k.replace('_', '-')
if k == 'maxkeys':
k = 'max-keys'
- if isinstance(v, unicode):
+ if isinstance(v, unicode) and hasattr(v, 'decode'):
+ # for python2 strings only
v = v.encode('utf-8')
if v is not None and v != '':
l.append('%s=%s' % (urllib.quote(k), urllib.quote(str(v))))
@@ -544,7 +545,7 @@ def delete_keys2(hdrs):
if count <= 0:
return False # no more
data = data.encode('utf-8')
- fp = StringIO.StringIO(data)
+ fp = io.BytesIO(data)
md5 = boto.utils.compute_md5(fp)
hdrs['Content-MD5'] = md5[1]
hdrs['Content-Type'] = 'text/xml'
@@ -1167,7 +1168,7 @@ def configure_lifecycle(self, lifecycle_config, headers=None):
"""
xml = lifecycle_config.to_xml()
xml = xml.encode('utf-8')
- fp = StringIO.StringIO(xml)
+ fp = io.BytesIO(xml)
md5 = boto.utils.compute_md5(fp)
if headers is None:
headers = {}
@@ -1394,7 +1395,8 @@ def set_cors_xml(self, cors_xml, headers=None):
CORS configuration. See the S3 documentation for details
of the exact syntax required.
"""
- fp = StringIO.StringIO(cors_xml)
+ cors_xml = boto.utils.ensure_bytes(cors_xml)
+ fp = io.BytesIO(cors_xml)
md5 = boto.utils.compute_md5(fp)
if headers is None:
headers = {}
@@ -1552,7 +1554,7 @@ def complete_multipart_upload(self, key_name, upload_id,
# Some errors will be reported in the body of the response
# even though the HTTP response code is 200. This check
# does a quick and dirty peek in the body for an error element.
- if body.find('<Error>') > 0:
+ if body.find(b'<Error>') > 0:
contains_error = True
boto.log.debug(body)
if response.status == 200 and not contains_error:
@@ -1607,11 +1609,12 @@ def get_xml_tags(self):
def set_xml_tags(self, tag_str, headers=None, query_args='tagging'):
if headers is None:
headers = {}
- md5 = boto.utils.compute_md5(StringIO.StringIO(tag_str))
+ tag_str = boto.utils.ensure_bytes(tag_str)
+ md5 = boto.utils.compute_md5(io.BytesIO(tag_str))
headers['Content-MD5'] = md5[1]
headers['Content-Type'] = 'text/xml'
response = self.connection.make_request('PUT', self.name,
- data=tag_str.encode('utf-8'),
+ data=tag_str,
query_args=query_args,
headers=headers)
body = response.read()
22 boto/s3/key.py
View
@@ -24,7 +24,10 @@
import mimetypes
import os
import re
-import rfc822
+try:
+ import rfc822
+except ImportError:
+ import email.utils as rfc822
import StringIO
import base64
import binascii
@@ -36,12 +39,12 @@
from boto.s3.keyfile import KeyFile
from boto.s3.user import User
from boto import UserAgent
-from boto.utils import compute_md5
+from boto.utils import compute_md5, wrap_hash_function
try:
from hashlib import md5
except ImportError:
from md5 import md5
-
+md5 = wrap_hash_function(md5)
class Key(object):
"""
@@ -1271,7 +1274,12 @@ class of the new Key to be REDUCED_REDUNDANCY. The Reduced
"""
if isinstance(s, unicode):
s = s.encode("utf-8")
- fp = StringIO.StringIO(s)
+ if hasattr(s, 'decode') and not hasattr(s, 'encode'):
+ # python3
+ import io
+ fp = io.BytesIO(s)
+ else:
+ fp = StringIO.StringIO(s)
r = self.set_contents_from_file(fp, headers, replace, cb, num_cb,
policy, md5, reduced_redundancy,
encrypt_key=encrypt_key)
@@ -1581,7 +1589,11 @@ def get_contents_as_string(self, headers=None,
:rtype: string
:returns: The contents of the file as a string
"""
- fp = StringIO.StringIO()
+ if hasattr("", 'decode'):
+ fp = StringIO.StringIO()
+ else:
+ import io
+ fp = io.BytesIO()
self.get_contents_to_file(fp, headers, cb, num_cb, torrent=torrent,
version_id=version_id,
response_headers=response_headers)
2  boto/s3/tagging.py
View
@@ -21,6 +21,8 @@ def to_xml(self):
self.key, self.value)
def __eq__(self, other):
+ if other is None:
+ return False
return (self.key == other.key and self.value == other.value)
21 boto/ses/connection.py
View
@@ -28,6 +28,7 @@
from boto.regioninfo import RegionInfo
import boto
import boto.jsonresponse
+import boto.utils
from boto.ses import exceptions as ses_exceptions
@@ -125,52 +126,52 @@ def _handle_error(self, response, body):
boto.log.error('%s %s' % (response.status, response.reason))
boto.log.error('%s' % body)
- if "Address blacklisted." in body:
+ if b"Address blacklisted." in body:
# Delivery failures happened frequently enough with the recipient's
# email address for Amazon to blacklist it. After a day or three,
# they'll be automatically removed, and delivery can be attempted
# again (if you write the code to do so in your application).
ExceptionToRaise = ses_exceptions.SESAddressBlacklistedError
exc_reason = "Address blacklisted."
- elif "Email address is not verified." in body:
+ elif b"Email address is not verified." in body:
# This error happens when the "Reply-To" value passed to
# send_email() hasn't been verified yet.
ExceptionToRaise = ses_exceptions.SESAddressNotVerifiedError
exc_reason = "Email address is not verified."
- elif "Daily message quota exceeded." in body:
+ elif b"Daily message quota exceeded." in body:
# Encountered when your account exceeds the maximum total number
# of emails per 24 hours.
ExceptionToRaise = ses_exceptions.SESDailyQuotaExceededError
exc_reason = "Daily message quota exceeded."
- elif "Maximum sending rate exceeded." in body:
+ elif b"Maximum sending rate exceeded." in body:
# Your account has sent above its allowed requests a second rate.
ExceptionToRaise = ses_exceptions.SESMaxSendingRateExceededError
exc_reason = "Maximum sending rate exceeded."
- elif "Domain ends with dot." in body:
+ elif b"Domain ends with dot." in body:
# Recipient address ends with a dot/period. This is invalid.
ExceptionToRaise = ses_exceptions.SESDomainEndsWithDotError
exc_reason = "Domain ends with dot."
- elif "Local address contains control or whitespace" in body:
+ elif b"Local address contains control or whitespace" in body:
# I think this pertains to the recipient address.
ExceptionToRaise = ses_exceptions.SESLocalAddressCharacterError
exc_reason = "Local address contains control or whitespace."
- elif "Illegal address" in body:
+ elif b"Illegal address" in body:
# A clearly mal-formed address.
ExceptionToRaise = ses_exceptions.SESIllegalAddressError
exc_reason = "Illegal address"
# The re.search is to distinguish from the
# SESAddressNotVerifiedError error above.
- elif re.search('Identity.*is not verified', body):
+ elif re.search(b'Identity.*is not verified', body):
ExceptionToRaise = ses_exceptions.SESIdentityNotVerifiedError
exc_reason = "Identity is not verified."
- elif "ownership not confirmed" in body:
+ elif b"ownership not confirmed" in body:
ExceptionToRaise = ses_exceptions.SESDomainNotConfirmedError
exc_reason = "Domain ownership is not confirmed."
else:
# This is either a common AWS error, or one that we don't devote
# its own exception to.
ExceptionToRaise = self.ResponseError
- exc_reason = response.reason
+ exc_reason = boto.utils.ensure_string(response.reason)
raise ExceptionToRaise(response.status, exc_reason, body)
5 boto/sqs/message.py
View
@@ -68,6 +68,7 @@
from boto.sqs.attributes import Attributes
from boto.exception import SQSDecodeError
import boto
+from boto.utils import ensure_string, ensure_bytes
class RawMessage:
"""
@@ -151,11 +152,11 @@ class Message(RawMessage):
"""
def encode(self, value):
- return base64.b64encode(value)
+ return ensure_string(base64.b64encode(ensure_bytes(value)))
def decode(self, value):
try:
- value = base64.b64decode(value)
+ value = ensure_string(base64.b64decode(ensure_bytes(value)))
except:
boto.log.warning('Unable to decode message')
return value
59 boto/utils.py
View
@@ -60,6 +60,7 @@
import email.encoders
import gzip
import base64
+import functools
try:
from hashlib import md5
except ImportError:
@@ -154,7 +155,12 @@ def canonical_string(method, path, headers, expires=None,
qsa = [a.split('=', 1) for a in qsa]
qsa = [unquote_v(a) for a in qsa if a[0] in qsa_of_interest]
if len(qsa) > 0:
- qsa.sort(cmp=lambda x, y:cmp(x[0], y[0]))
+ def cmp(a, b):
+ return (a > b) - (a < b)
+ if hasattr(functools, 'cmp_to_key'):
+ qsa.sort(key=functools.cmp_to_key(lambda x, y:cmp(x[0], y[0])))
+ else:
+ qsa.sort(cmp=lambda x, y:cmp(x[0], y[0]))
qsa = ['='.join(a) for a in qsa]
buf += '?'
buf += '&'.join(qsa)
@@ -186,10 +192,13 @@ def get_aws_metadata(headers, provider=None):
for hkey in headers.keys():
if hkey.lower().startswith(metadata_prefix):
val = urllib.unquote_plus(headers[hkey])
- try:
- metadata[hkey[len(metadata_prefix):]] = unicode(val, 'utf-8')
- except UnicodeDecodeError:
- metadata[hkey[len(metadata_prefix):]] = val
+ if hasattr(val, 'decode'):
+ try:
+ metadata[hkey[len(metadata_prefix):]] = val.decode('utf-8')
+ except UnicodeDecodeError:
+ metadata[hkey[len(metadata_prefix):]] = val
+ else:
+ metadata[hkey[len(metadata_prefix):]] = ensure_string(val)
del headers[hkey]
return metadata
@@ -925,3 +934,43 @@ def compute_hash(fp, buf_size=8192, size=None, hash_algorithm=md5):
data_size = fp.tell() - spos
fp.seek(spos)
return (hex_digest, base64_digest, data_size)
+
+def ensure_bytes(string):
+ # in python2 convert unicode strings to regular strings and in
+ # python3 convert strings to bytes, but don't waste time converting
+ # regular python2 strings to themselves
+ if not isinstance(string, str) and hasattr(string, 'encode'):
+ # python2 unicode string
+ string = string.encode('utf-8')
+ elif isinstance(string, str) and not hasattr(string, 'decode'):
+ # python3 string -> bytes
+ string = string.encode('utf-8')
+ return string
+
+def ensure_string(string):
+ if hasattr(string, 'decode') and not hasattr(string, 'encode'):
+ # it's a python3 byte literal
+ string = string.decode('utf-8')
+ return string
+
+class FakeHashObj(object):
+ """Intercept calls to hashlib objects and ensure
+ only bytes get by in python3
+ """
+ def __init__(self, obj):
+ self.obj = obj
+
+ def __getattr__(self, attr):
+ return getattr(self.obj, attr)
+
+ def update(self, str):
+ return self.obj.update(boto.utils.ensure_bytes(str))
+
+def wrap_hash_function(hf):
+ def get_hash(*args, **kwargs):
+ if args:
+ args = list(args)
+ args[0] = boto.utils.ensure_bytes(args[0])
+ obj = FakeHashObj(hf(*args, **kwargs))
+ return obj
+ return get_hash
1  boto/version.txt
View
@@ -0,0 +1 @@
+2.8.0-dev
135 py3kport/README.txt
View
@@ -0,0 +1,135 @@
+This fork of the official boto project is for adding python 3
+compatibility. The goal is to have a single code base that by default
+runs on python2.x, and with a simple application of 2to3, also runs on
+python3.x. We measure progress by how many of the extensive unit
+tests pass.
+
+To contribute, you will need to:
+
+1) Verify you can run and get all the tests to pass on python2.7.
+
+2) Set up to run tests (which at this point won't all pass) on python 3.3.
+
+3) Refer to the github issues and wiki to organize our team effort.
+
+PRELUDE: INSTALL PYTHON ENVIRONMENTS
+
+You are going to set up virtual environments for both 2.7 and
+3.3. Let's start with the former. (I'm going to assume you will work
+in a bash-like shell prompt.)
+
+*** Python 2.7.3
+
+It's important to first verify you can get all tests to run and pass
+in Python 2, for three reasons:
+
+ 1) This is a baseline: the fastest way to ensure you know how to set
+ up your development environment.
+
+ 2) You will be able to jump back to this as a reference if you need
+ it while troubleshooting.
+
+ 3) Most importantly, our changes to the boto source *must* preserve
+ all functionality in python 2. Otherwise upstream will not accept
+ merge requests. So you will regularly want to verify the python 2
+ tests all still pass.
+
+First, install python 2.7.3, pip, and virtualenv. Search online or ask
+for help if needed.
+
+This done, create a virtual environment that is clearly labeled. For
+example, suppose you decide to collect all your Python 2 virtual
+environments in a folder named $HOME/.pyvenv2 , and you want to
+name this one "boto-2.7-dev". Then you could invoke:
+
+virtualenv $HOME/.pyvenv2/boto-2.7-dev
+
+(Make sure you are using the virtualenv associated with Python 2.7.3.)
+
+That done, open a terminal (window) with a command line you will
+dedicate to run the Python 2.7 tests. Go into the boto source
+checkout, and then run:
+
+ source $HOME/.pyvenv2/boto-2.7-dev/bin/activate
+
+If you invoke:
+
+ which python pip
+
+... you should see something like:
+$HOME/.pyvenv2/boto-2.7-dev/bin/python
+$HOME/.pyvenv2/boto-2.7-dev/bin/pip
+
+Also, python -V should print "2.7.3".
+
+Then you can install the boto requirements. Do this with:
+
+ pip install -r requirements.txt
+
+(where requirements.txt is in the top-level boto source folder). This
+will take some minutes depending on the speed of your network
+connection. Once it is done, run the tests:
+
+ python tests/test.py tests/unit
+
+They should all pass. If not, troubleshoot until they do.
+
+*** Python 3.3.0
+
+The first steps are similar. Install the latest Python 3.3 and pip.
+3.3 includes virtualenv, so you don't need to install that separately;
+it's an executable named "pyvenv" in the same folder as the Python3.3
+executable.
+
+You probably want to have a separate directory for storing Python3
+virtual environments. Let's be forward thinking and call this
+$HOME/.pyvenv (but you can tack a "3" on the end and make it .pyvenv3
+if you really want to). Then if you want to label the boto venv
+something like "boto-3.3-dev", you can say:
+
+ pyvenv $HOME/.pyvenv/boto-3.3-dev
+
+Then go into a NEW shell window, and run:
+
+ source $HOME/.pyvenv/boto-3.3-dev/bin/activate
+ pip install -r py3kport/requirements.txt-3.3
+
+Note you are installing a different set of requirements. If you look
+at the diff between requirements.txt and requirements.txt-3.3, you'll
+see that the Python 2 version of boto uses many packages that have
+been integrated into Python 3's standard library (including, most
+recently in 3.3, mock).
+
+To actually run the tests requires a new approach. We want to apply
+to keep the actual source in python 2 form. For Python 3, we want to
+copy these sources to a build directory; apply 2to3; and run the tests
+on *that*.
+
+I've written a bash script that automates this, which I encourage you
+to read and understand. It's in py3kport/rebuild.sh, and you use it
+like this:
+
+ source $HOME/.pyvenv/boto-3.3-dev/bin/activate
+ ./py3kport/rebuild.sh
+
+This executes creates the Python 3 sources in ./py3kport/build. You
+only need to source bin/activate once. Each time you remake the
+sources (by re-running rebuild.sh), it will delete the py3kport/build
+folder, recreate it, and re-run the tests.
+
+If you want to run specific tests, you can do so selectively via:
+
+ cd py3kport/build
+ python tests/test.py tests/unit
+
+Sometimes this will be useful when troubleshooting, because you'll
+make changes to the generated python3 code directly, in order to just
+understand how to get a test to pass, before modifying the python 2
+source. (It becomes a fun puzzle: you need to modify the python2
+source so that the python 2 tests pass, AND so that the version
+generated by 2to3 will also pass under Python 3.)
+
+(NOTE WELL! Remember the build directory is deleted every time, so be
+sure to save private copies of anything you don't want to lose next
+time you run rebuild. Everything matching py3kport/build* is
+gitignored, so you can do "cp -a build build.42", etc.)
46 py3kport/rebuild.sh
View
@@ -0,0 +1,46 @@
+#!/bin/bash
+# Rebuild the python 3 code from the python 2 sources, and run tests.
+
+set -eu
+IFS=$'\n\t'
+
+# These are items in the boto source we want to copy into the build directory, and apply with 2to3.
+items=(
+ boto
+ tests
+)
+
+BASE=$(dirname "${BASH_SOURCE[0]}")
+BUILD_DIR="${BASE}/build"
+SRC_DIR="${BASE}/.."
+
+# Prep build directory
+rm -rf "$BUILD_DIR"
+mkdir -p "$BUILD_DIR"
+
+# If /usr/bin/realpath is installed, we can make the path names more clear.
+which realpath >/dev/null && {
+ BUILD_DIR=$(realpath "$BUILD_DIR")
+ SRC_DIR=$(realpath "$SRC_DIR")
+}
+
+echo "Using source directory: $SRC_DIR"
+echo "Re-created build directory: $BUILD_DIR"
+
+for item in ${items[@]}; do
+ cp -a "${SRC_DIR}/${item}" "${BUILD_DIR}/${item}"
+done
+
+echo "Applying 2to3..."
+cd "${BUILD_DIR}"
+
+2to3 -n -w . > 2to3.out 2> 2to3.err
+# Note that if 2to3 has an error (i.e. exits with a nonzero exit
+# code), this script will exit because it uses uses the set -e option
+# above. man bash for details.
+
+echo "2to3 successful! Running tests... (with verbose output)"
+
+set -x
+python tests/test.py tests/unit
+
6 py3kport/requirements.txt-3.3
View
@@ -0,0 +1,6 @@
+nose==1.1.2
+requests==0.13.1
+rsa==3.1.1
+tox==1.4
+Sphinx==1.1.3
+httpretty==0.5.10
21 setup.py
View
@@ -34,7 +34,10 @@
import sys
-from boto import __version__
+try: # Python 3
+ from distutils.command.build_py import build_py_2to3 as build_py
+except ImportError: # Python 2
+ from distutils.command.build_py import build_py
if sys.version_info <= (2, 4):
error = "ERROR: boto requires Python Version 2.5 or above...exiting."
@@ -45,8 +48,13 @@ def readme():
with open("README.rst") as f:
return f.read()
-setup(name = "boto",
- version = __version__,
+def version():
+ with open("boto/version.txt") as f:
+ return f.read().strip()
+
+setup(cmdclass = {'build_py': build_py},
+ name = "boto",
+ version = version(),
description = "Amazon Web Services Library",
long_description = readme(),
author = "Mitch Garnaat",
@@ -74,7 +82,7 @@ def readme():
"boto.swf", "boto.mws", "boto.cloudsearch", "boto.glacier",
"boto.beanstalk", "boto.datapipeline", "boto.elasticache",
"boto.elastictranscoder", "boto.opsworks"],
- package_data = {"boto.cacerts": ["cacerts.txt"]},
+ package_data = {"boto.cacerts": ["cacerts.txt"], "boto": ["version.txt"]},
license = "MIT",
platforms = "Posix; MacOS X; Windows",
classifiers = ["Development Status :: 5 - Production/Stable",
@@ -83,8 +91,9 @@ def readme():
"Operating System :: OS Independent",
"Topic :: Internet",
"Programming Language :: Python :: 2",
- "Programming Language :: Python :: 2.5",
"Programming Language :: Python :: 2.6",
- "Programming Language :: Python :: 2.7"],
+ "Programming Language :: Python :: 2.7",
+ "Programming Language :: Python :: 3",
+ "Programming Language :: Python :: 3.3"],
**extra
)
6 tests/integration/s3/mock_storage_service.py
View
@@ -31,7 +31,7 @@
import base64
import re
-from boto.utils import compute_md5
+from boto.utils import compute_md5, ensure_bytes
from boto.s3.prefix import Prefix
try:
@@ -89,7 +89,7 @@ def get_contents_to_file(self, fp, headers=NOT_IMPL,
torrent=NOT_IMPL,
version_id=NOT_IMPL,
res_download_handler=NOT_IMPL):
- fp.write(self.data)
+ fp.write(ensure_bytes(self.data))
def get_file(self, fp, headers=NOT_IMPL, cb=NOT_IMPL, num_cb=NOT_IMPL,
torrent=NOT_IMPL, version_id=NOT_IMPL,
@@ -191,7 +191,7 @@ def set_etag(self):
contents of mock key.
"""
m = md5()
- m.update(self.data)
+ m.update(ensure_bytes(self.data))
hex_md5 = m.hexdigest()
self.etag = hex_md5
2  tests/integration/s3/test_bucket.py
View
@@ -162,7 +162,7 @@ def test_website_configuration(self):
{'IndexDocument': {'Suffix': 'index.html'}}})
config2, xml = self.bucket.get_website_configuration_with_xml()
self.assertEqual(config, config2)
- self.assertTrue('<Suffix>index.html</Suffix>' in xml, xml)
+ self.assertTrue(b'<Suffix>index.html</Suffix>' in xml, xml)
def test_website_redirect_all_requests(self):
response = self.bucket.configure_website(
10 tests/integration/s3/test_connection.py
View
@@ -53,14 +53,14 @@ def test_1_basic(self):
bucket.disable_logging()
c.delete_bucket(logging_bucket)
k = bucket.new_key('foobar')
- s1 = 'This is a test of file upload and download'
- s2 = 'This is a second string to test file upload and download'
+ s1 = b'This is a test of file upload and download'
+ s2 = b'This is a second string to test file upload and download'
k.set_contents_from_string(s1)
fp = open('foobar', 'wb')
# now get the contents from s3 to a local file
k.get_contents_to_file(fp)
fp.close()
- fp = open('foobar')
+ fp = open('foobar', 'rb')
# check to make sure content read from s3 is identical to original
assert s1 == fp.read(), 'corrupted file'
fp.close()
@@ -90,7 +90,7 @@ def test_1_basic(self):
con.request("PUT", up.path + '?' + up.query, body="hello there")
resp = con.getresponse()
assert 200 == resp.status
- assert "hello there" == k.get_contents_as_string()
+ assert "hello there" == k.get_contents_as_string().decode('utf-8')
bucket.delete_key(k)
# test a few variations on get_all_keys - first load some data
# for the first one, let's override the content type
@@ -239,7 +239,7 @@ def test_basic_anon(self):
def test_error_code_populated(self):
c = S3Connection()
try:
- c.create_bucket('bad$bucket$name')
+ c.create_bucket('bb')
except S3ResponseError, e:
self.assertEqual(e.error_code, 'InvalidBucketName')
else:
4 tests/integration/s3/test_encryption.py
View
@@ -65,8 +65,8 @@ def test_1_versions(self):
# create an unencrypted key
k = bucket.new_key('foobar')
- s1 = 'This is unencrypted data'
- s2 = 'This is encrypted data'
+ s1 = b'This is unencrypted data'
+ s2 = b'This is encrypted data'
k.set_contents_from_string(s1)
time.sleep(5)
28 tests/integration/s3/test_key.py
View
@@ -26,7 +26,7 @@
from tests.unit import unittest
import time
-import StringIO
+import io
from boto.s3.connection import S3Connection
from boto.s3.key import Key
from boto.exception import S3ResponseError
@@ -47,8 +47,8 @@ def tearDown(self):
def test_set_contents_from_file_dataloss(self):
# Create an empty stringio and write to it.
- content = "abcde"
- sfp = StringIO.StringIO()
+ content = b"abcde"
+ sfp = io.BytesIO()
sfp.write(content)
# Try set_contents_from_file() without rewinding sfp
k = self.bucket.new_key("k")
@@ -66,18 +66,18 @@ def test_set_contents_from_file_dataloss(self):
self.assertEqual(ks, content)
# finally, try with a 0 length string
- sfp = StringIO.StringIO()
+ sfp = io.BytesIO()
k = self.bucket.new_key("k")
k.set_contents_from_file(sfp)
self.assertEqual(k.size, 0)
# check actual contents by getting it.
kn = self.bucket.new_key("k")
ks = kn.get_contents_as_string()
- self.assertEqual(ks, "")
+ self.assertEqual(ks, b"")
def test_set_contents_as_file(self):
- content="01234567890123456789"
- sfp = StringIO.StringIO(content)
+ content=b"01234567890123456789"
+ sfp = io.BytesIO(content)
# fp is set at 0 for just opened (for read) files.
# set_contents should write full content to key.
@@ -110,8 +110,8 @@ def test_set_contents_as_file(self):
self.assertEqual(ks, content[5:10])
def test_set_contents_with_md5(self):
- content="01234567890123456789"
- sfp = StringIO.StringIO(content)
+ content=b"01234567890123456789"
+ sfp = io.BytesIO(content)
# fp is set at 0 for just opened (for read) files.
# set_contents should write full content to key.
@@ -145,8 +145,8 @@ def test_set_contents_with_md5(self):
pass
def test_get_contents_with_md5(self):
- content="01234567890123456789"
- sfp = StringIO.StringIO(content)
+ content=b"01234567890123456789"
+ sfp = io.BytesIO(content)
k = self.bucket.new_key("k")
k.set_contents_from_file(sfp)
@@ -166,7 +166,7 @@ def callback(wrote, total):
self.my_cb_last = None
k = self.bucket.new_key("k")
k.BufferSize = 2
- sfp = StringIO.StringIO("")
+ sfp = io.BytesIO(b"")
k.set_contents_from_file(sfp, cb=callback, num_cb=10)
self.assertEqual(self.my_cb_cnt, 1)
self.assertEqual(self.my_cb_last, 0)
@@ -179,8 +179,8 @@ def callback(wrote, total):
self.assertEqual(self.my_cb_cnt, 1)
self.assertEqual(self.my_cb_last, 0)
- content="01234567890123456789"
- sfp = StringIO.StringIO(content)
+ content=b"01234567890123456789"
+ sfp = io.BytesIO(content)
# expect 2 calls due start/finish
self.my_cb_cnt = 0
2  tests/integration/s3/test_mfa.py
View
@@ -65,7 +65,7 @@ def test_mfadel(self):
# Add a key to the bucket
k = self.bucket.new_key('foobar')
- s1 = 'This is v1'
+ s1 = b'This is v1'
k.set_contents_from_string(s1)
v1 = k.version_id
10 tests/integration/s3/test_multipart.py
View
@@ -34,7 +34,7 @@
import unittest
import time
-import StringIO
+import io
from boto.s3.connection import S3Connection
@@ -59,7 +59,7 @@ def test_abort(self):
def test_complete_ascii(self):
key_name = "test"
mpu = self.bucket.initiate_multipart_upload(key_name)
- fp = StringIO.StringIO("small file")
+ fp = io.BytesIO(b"small file")
mpu.upload_part_from_file(fp, part_num=1)
fp.close()
cmpu = mpu.complete_upload()
@@ -69,7 +69,7 @@ def test_complete_ascii(self):
def test_complete_japanese(self):
key_name = u"テスト"
mpu = self.bucket.initiate_multipart_upload(key_name)
- fp = StringIO.StringIO("small file")
+ fp = io.BytesIO(b"small file")
mpu.upload_part_from_file(fp, part_num=1)
fp.close()
cmpu = mpu.complete_upload()
@@ -108,8 +108,8 @@ def test_list_multipart_uploads(self):
def test_four_part_file(self):
key_name = "k"
- contents = "01234567890123456789"
- sfp = StringIO.StringIO(contents)
+ contents = b"01234567890123456789"
+ sfp = io.BytesIO(contents)
# upload 20 bytes in 4 parts of 5 bytes each
mpu = self.bucket.initiate_multipart_upload(key_name)
2  tests/integration/s3/test_pool.py
View
@@ -28,8 +28,6 @@
import time
import uuid
-from StringIO import StringIO
-
from threading import Thread
def spawn(function, *args, **kwargs):
4 tests/integration/s3/test_versioning.py
View
@@ -55,7 +55,7 @@ def test_1_versions(self):
# create a new key in the versioned bucket
k = self.bucket.new_key("foobar")
- s1 = 'This is v1'
+ s1 = b'This is v1'
k.set_contents_from_string(s1)
# remember the version id of this object
@@ -68,7 +68,7 @@ def test_1_versions(self):
self.assertEqual(s1, o1)
# now overwrite that same key with new data
- s2 = 'This is v2'
+ s2 = b'This is v2'
k.set_contents_from_string(s2)
v2 = k.version_id
5 tests/unit/__init__.py
View
@@ -4,7 +4,10 @@
import unittest
import httplib
-from mock import Mock
+try:
+ from unittest.mock import Mock
+except ImportError:
+ from mock import Mock
class AWSMockServiceTestCase(unittest.TestCase):
5 tests/unit/auth/test_sigv4.py
View
@@ -19,7 +19,10 @@
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
# IN THE SOFTWARE.
#
-from mock import Mock
+try:
+ from unittest.mock import Mock
+except ImportError:
+ from mock import Mock
from tests.unit import unittest
from boto.auth import HmacAuthV4Handler
5 tests/unit/cloudformation/test_connection.py
View
@@ -7,7 +7,10 @@
except ImportError:
import simplejson as json
-from mock import Mock
+try:
+ from unittest.mock import Mock
+except ImportError:
+ from mock import Mock
from tests.unit import AWSMockServiceTestCase
from boto.cloudformation.connection import CloudFormationConnection
5 tests/unit/cloudfront/test_invalidation_list.py
View
@@ -3,7 +3,10 @@
import string
from tests.unit import unittest
-import mock
+try:
+ import unittest.mock as mock
+except ImportError:
+ import mock
import boto
17 tests/unit/cloudsearch/test_document.py
View
@@ -2,7 +2,10 @@
from tests.unit import unittest
from httpretty import HTTPretty
-from mock import MagicMock
+try:
+ from unittest.mock import MagicMock
+except ImportError:
+ from mock import MagicMock
import urlparse
import json
@@ -45,7 +48,7 @@ def test_cloudsearch_add_basics(self):
"category": ["cat_a", "cat_b", "cat_c"]})
document.commit()
- args = json.loads(HTTPretty.last_request.body)[0]
+ args = json.loads(HTTPretty.last_request.body.decode('utf-8'))[0]
self.assertEqual(args['lang'], 'en')
self.assertEqual(args['type'], 'add')
@@ -61,7 +64,7 @@ def test_cloudsearch_add_single_basic(self):
"category": ["cat_a", "cat_b", "cat_c"]})
document.commit()
- args = json.loads(HTTPretty.last_request.body)[0]
+ args = json.loads(HTTPretty.last_request.body.decode('utf-8'))[0]
self.assertEqual(args['id'], '1234')
self.assertEqual(args['version'], 10)
@@ -77,7 +80,7 @@ def test_cloudsearch_add_single_fields(self):
"category": ["cat_a", "cat_b", "cat_c"]})
document.commit()
- args = json.loads(HTTPretty.last_request.body)[0]
+ args = json.loads(HTTPretty.last_request.body.decode('utf-8'))[0]
self.assertEqual(args['fields']['category'], ['cat_a', 'cat_b',
'cat_c'])
@@ -133,7 +136,7 @@ def test_cloudsearch_add_basics(self):
document.add(key, obj['version'], obj['fields'])
document.commit()
- args = json.loads(HTTPretty.last_request.body)
+ args = json.loads(HTTPretty.last_request.body.decode('utf-8'))
for arg in args:
self.assertTrue(arg['id'] in self.objs)
@@ -177,7 +180,7 @@ def test_cloudsearch_delete(self):
endpoint="doc-demo-userdomain.us-east-1.cloudsearch.amazonaws.com")
document.delete("5", "10")
document.commit()
- args = json.loads(HTTPretty.last_request.body)[0]
+ args = json.loads(HTTPretty.last_request.body.decode('utf-8'))[0]
self.assertEqual(args['version'], '10')
self.assertEqual(args['type'], 'delete')
@@ -210,7 +213,7 @@ def test_cloudsearch_delete_multiples(self):
document.delete("5", "10")
document.delete("6", "11")
document.commit()
- args = json.loads(HTTPretty.last_request.body)
+ args = json.loads(HTTPretty.last_request.body.decode('utf-8'))
self.assertEqual(len(args), 2)
for arg in args:
2  tests/unit/cloudsearch/test_search.py
View
@@ -62,7 +62,7 @@ class CloudSearchSearchTest(unittest.TestCase):
}
def get_args(self, requestline):
- (_, request, _) = requestline.split(" ")
+ (_, request, _) = requestline.decode('utf-8').split(" ")
(_, request) = request.split("?", 1)
args = urlparse.parse_qs(request)
return args
5 tests/unit/dynamodb/test_layer2.py
View
@@ -22,7 +22,10 @@
#
from tests.unit import unittest
-from mock import Mock
+try:
+ from unittest.mock import Mock
+except ImportError:
+ from mock import Mock
from boto.dynamodb.layer2 import Layer2
from boto.dynamodb.table import Table, Schema
5 tests/unit/dynamodb/test_types.py
View
@@ -38,8 +38,9 @@ def test_encoding_to_dynamodb(self):
self.assertEqual(dynamizer.encode(Decimal('1.1')), {'N': '1.1'})
self.assertEqual(dynamizer.encode(set([1, 2, 3])),
{'NS': ['1', '2', '3']})
- self.assertEqual(dynamizer.encode(set(['foo', 'bar'])),
- {'SS': ['foo', 'bar']})
+ set_encode = dynamizer.encode(set(['foo', 'bar']))
+ set_encode['SS'] = sorted(set_encode['SS'])
+ self.assertEqual(set_encode, {'SS': sorted(['foo', 'bar'])})
self.assertEqual(dynamizer.encode(types.Binary('\x01')),
{'B': 'AQ=='})
self.assertEqual(dynamizer.encode(set([types.Binary('\x01')])),
5 tests/unit/ec2/test_address.py
View
@@ -1,4 +1,7 @@
-import mock
+try:
+ import unittest.mock as mock
+except ImportError:
+ import mock
import unittest
from boto.ec2.address import Address
5 tests/unit/ec2/test_blockdevicemapping.py
View
@@ -1,4 +1,7 @@
-import mock
+try:
+ import unittest.mock as mock
+except ImportError:
+ import mock
import unittest
from boto.ec2.blockdevicemapping import BlockDeviceType, BlockDeviceMapping
5 tests/unit/ec2/test_instance.py
View
@@ -3,7 +3,10 @@
from tests.unit import unittest
from tests.unit import AWSMockServiceTestCase
-import mock
+try:
+ import unittest.mock as mock
+except ImportError:
+ import mock
from boto.ec2.connection import EC2Connection
5 tests/unit/ec2/test_volume.py
View
@@ -1,4 +1,7 @@
-import mock
+try:
+ import unittest.mock as mock
+except ImportError:
+ import mock
from tests.unit import unittest
from boto.ec2.snapshot import Snapshot
5 tests/unit/glacier/test_concurrent.py
View
@@ -23,7 +23,10 @@
import tempfile
from Queue import Queue
-import mock
+try:
+ import unittest.mock as mock
+except ImportError:
+ import mock
from tests.unit import unittest
from tests.unit import AWSMockServiceTestCase
5 tests/unit/glacier/test_job.py
View
@@ -20,7 +20,10 @@
# IN THE SOFTWARE.
#
from tests.unit import unittest
-import mock
+try:
+ import unittest.mock as mock
+except ImportError:
+ import mock
from boto.glacier.job import Job
from boto.glacier.layer1 import Layer1
2  tests/unit/glacier/test_layer1.py
View
@@ -83,7 +83,7 @@ def test_get_archive_output(self):
class GlacierUploadArchiveResets(GlacierLayer1ConnectionBase):
def test_upload_archive(self):
fake_data = tempfile.NamedTemporaryFile()
- fake_data.write('foobarbaz')
+ fake_data.write(b'foobarbaz')
# First seek to a non zero offset.
fake_data.seek(2)
self.set_http_response(status_code=201)
7 tests/unit/glacier/test_layer2.py
View
@@ -23,7 +23,10 @@
from tests.unit import unittest
-from mock import call, Mock, patch, sentinel
+try:
+ from unittest.mock import call, Mock, patch, sentinel
+except ImportError:
+ from mock import call, Mock, patch, sentinel
from boto.glacier.layer1 import Layer1
from boto.glacier.layer2 import Layer2
@@ -222,7 +225,7 @@ def test_resume_archive_from_file(self, mock_resume_file_upload):
sentinel.upload_id, file_obj=sentinel.file_obj)
mock_resume_file_upload.assert_called_once_with(
self.vault, sentinel.upload_id, part_size, sentinel.file_obj,
- {0: '12'.decode('hex'), 1: '34'.decode('hex')})
+ {0: b'\x12', 1: b'4'})
class TestJob(GlacierLayer2Base):
8 tests/unit/glacier/test_vault.py
View
@@ -23,8 +23,12 @@
import unittest
from cStringIO import StringIO
-import mock
-from mock import ANY
+try:
+ import unittest.mock as mock
+ from unittest.mock import ANY
+except ImportError:
+ import mock
+ from mock import ANY
from boto.glacier import vault
20 tests/unit/glacier/test_writer.py
View
@@ -24,12 +24,20 @@
from StringIO import StringIO
from tests.unit import unittest
-from mock import (
- call,
- Mock,
- patch,
- sentinel,
-)
+try:
+ from unittest.mock import (
+ call,
+ Mock,
+ patch,
+ sentinel,
+ )
+except ImportError:
+ from mock import (
+ call,
+ Mock,
+ patch,
+ sentinel,
+ )
from nose.tools import assert_equal
from boto.glacier.layer1 import Layer1
5 tests/unit/provider/test_provider.py
View
@@ -2,7 +2,10 @@
from datetime import datetime, timedelta
from tests.unit import unittest
-import mock
+try:
+ import unittest.mock as mock
+except ImportError:
+ import mock
from boto import provider