Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Make boto.ec2 work with Python 3.2 and Python 3.3 #1156

Closed
wants to merge 24 commits into from

3 participants

@msabramo

A bunch of changes to make boto.ec2 tests pass on Python 3.2 and Python 3.3 while all boto tests continue to pass on Python 2. There has been some manual testing as well to verify that basic functionality works (e.g.: printing all available images) when connecting to Eucalyptus Community Cloud (see here) and EC2 (see here).

Passing Travis CI build: https://travis-ci.org/msabramo/boto/builds/3465072

This is the work that I alluded to in #1127.

msabramo added some commits
@msabramo msabramo Update boto/pyami/config.py for Python 3 c3d1f67
@msabramo msabramo Update boto/exception.py for Python 3 6f43e29
@msabramo msabramo Add stuff to tox.ini for testing py32 and py33 with boto.ec2 alone 8fe81b3
@msabramo msabramo Remove simplejson from requirements.txt as it causes a failure with p…
…y33 (and I think it's not needed with py26 and py27?)
23bb84e
@msabramo msabramo Make tests/unit/__init__.py Python 3 friendly by abstracting out that…
… stuff moved from httplib to http.client in Python 3
17f11d7
@msabramo msabramo Make boto/ec2/connection.py Python 3 friendly by changing "except com…
…ma" syntax to "except as" syntax
9a7ad50
@msabramo msabramo Make boto/connection.py more Python 3 friendly d363a67
@msabramo msabramo Make boto/utils.py more Python 3 friendly e26f487
@msabramo msabramo Make boto/auth_handler.py Python 3 friendly 2ad1ff1
@msabramo msabramo Make test commands in tox.ini consistent 779736d
@msabramo msabramo Use newer o0 notation for octal number in boto/ec2/keypair.py 3d12d0c
@msabramo msabramo Put parentheses around print calls in boto/ec2/reservedinstance.py 0e6823c
@msabramo msabramo Tweak HTTPS detection to work in Python 3 in boto/connection.py and b…
…oto/http_connection.py
9dfc91e
@msabramo msabramo Paper over lack of `long` type in Python 3 in boto/connection.py 2bd3784
@msabramo msabramo Update boto/__init__.py for Python 3 4a0debc
@msabramo msabramo A bunch of changes to support Python 3. Tests passing in Python 3 now…
… for boto.ec2!
5c91d91
@msabramo msabramo Use same default testenv command as on `develop` branch. Add a testen…
…v for testing boto.ec2 only with py26
364c980
@msabramo msabramo .travis.yml: Test Python 3.2 but only run tests for boto.ec2 since th…
…at's all that I've ported to Python 3 so far.
93f54ed
@msabramo

Has anyone gotten a chance to try this out? I don't actually have a use case for boto at the moment, so I haven't gotten a chance to use this code for real.

@skiold

py27 and py33_ec2 test passing with msabramo@0223797

But that code tree breaks when trying to open a connection to the api (see https://gist.github.com/skiold/5006231). I get the same error with py33 and py27.

@msabramo

Thanks, @skiold, I think I see what the problem might be. Hoping to have a fix soon.

@msabramo msabramo Fix infinite recursion error reported by @skiold on GitHub at
boto#1156 (comment)

I reverted back to qualifying stuff from `httplib` using `httplib.`;
because otherwise `HTTPResponse` is ambiguous and may refer to itself
instead of the version in `httplib`.
d2654f0
@msabramo

@skiold et al, I added commit d2654f0 which should fix the issue reported by @skiold.

@msabramo

Tox passing:

marca@marca-mac:~/dev/git-repos/boto$ tox
...
  py26: commands succeeded
  py27: commands succeeded
  py32_ec2: commands succeeded
  py33_ec2: commands succeeded
  congratulations :)

Passing Travis CI build: https://travis-ci.org/msabramo/boto/builds/4988307

Bug reported by @skiold seems to be fixed -- now it prints a proper HTTP 401 authorization failure message; (I don't have a real EC2 key and don't feel like registering for 1 year of free tier just for this):

marca@marca-mac:~/dev/git-repos/boto$ AWS_ACCESS_KEY_ID="xxx" AWS_SECRET_ACCESS_KEY="yyy" .tox/py27/bin/python -c 'import boto.ec2; boto.ec2.regions()'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "boto/ec2/__init__.py", line 40, in regions
    return c.get_all_regions()
  File "boto/ec2/connection.py", line 2665, in get_all_regions
    [('item', RegionInfo)], verb='POST')
  File "boto/connection.py", line 989, in get_list
    raise self.ResponseError(response.status, response.reason, body)
boto.exception.EC2ResponseError: EC2ResponseError: 401 Unauthorized
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>AuthFailure</Code><Message>AWS was not able to validate the provided access credentials</Message></Error></Errors><RequestID>0e8a7e6e-d4f9-49cb-834e-9e63e306f355</RequestID></Response>

marca@marca-mac:~/dev/git-repos/boto$ AWS_ACCESS_KEY_ID="xxx" AWS_SECRET_ACCESS_KEY="yyy" .tox/py32/bin/python -c 'import boto.ec2; boto.ec2.regions()'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "boto/ec2/__init__.py", line 40, in regions
    return c.get_all_regions()
  File "boto/ec2/connection.py", line 2665, in get_all_regions
    [('item', RegionInfo)], verb='POST')
  File "boto/connection.py", line 989, in get_list
    raise self.ResponseError(response.status, response.reason, body)
boto.exception.EC2ResponseError: EC2ResponseError: 401 Unauthorized
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>AuthFailure</Code><Message>AWS was not able to validate the provided access credentials</Message></Error></Errors><RequestID>56687567-1e72-474a-aad0-ff48645e61ef</RequestID></Response>

marca@marca-mac:~/dev/git-repos/boto$ AWS_ACCESS_KEY_ID="xxx" AWS_SECRET_ACCESS_KEY="yyy" .tox/py33/bin/python -c 'import boto.ec2; boto.ec2.regions()'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "./boto/ec2/__init__.py", line 40, in regions
    return c.get_all_regions()
  File "./boto/ec2/connection.py", line 2665, in get_all_regions
    [('item', RegionInfo)], verb='POST')
  File "./boto/connection.py", line 989, in get_list
    raise self.ResponseError(response.status, response.reason, body)
boto.exception.EC2ResponseError: EC2ResponseError: 401 Unauthorized
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>AuthFailure</Code><Message>AWS was not able to validate the provided access credentials</Message></Error></Errors><RequestID>fe1fb9d1-c107-4d7c-a067-96bb74bb0495</RequestID></Response>

If folks could try this out with real AWS credentials, that would be great.

@msabramo

OK, I've managed to get this to connect to the Eucalyptus Community Cloud. I signed up for an account and then used the instructions at this blog post to configure it.

marca@marca-mac:~/dev/git-repos/boto$ cat print_euca_images.py 
import boto

euca = boto.connect_euca()

print(euca.get_all_images())

marca@marca-mac:~/dev/git-repos/boto$ .tox/py26/bin/python print_euca_images.py 
[Image:emi-F86C3FE3, Image:emi-51DD3B86, ...]

marca@marca-mac:~/dev/git-repos/boto$ .tox/py27/bin/python print_euca_images.py 
[Image:emi-F86C3FE3, Image:emi-51DD3B86, ...]

marca@marca-mac:~/dev/git-repos/boto$ .tox/py32/bin/python print_euca_images.py 
[Image:emi-F86C3FE3, Image:emi-51DD3B86, ...]

marca@marca-mac:~/dev/git-repos/boto$ .tox/py33/bin/python print_euca_images.py 
[Image:emi-F86C3FE3, Image:emi-51DD3B86, ...]

Note that I can connect to euca and print a list of images with Python 2.6, 2.7, 3.2, and 3.3.

@skiold

Working against ec2 also
:+1:

@msabramo

For adventurous folks who are interesting in using boto.s3 on Python 3, have a look at https://github.com/msabramo/boto/tree/pr_python3_boto_s3_2013-02-26 -- in that tree I have 8 out of 9 of the boto.s3 unit tests passing. I have not tried to connect to Eucalyptus Community Cloud or anything though.

@kurin
@msabramo

Interesting. I wonder if our changes are similar. It does seem like a good idea to have a central repo where folks can collaborate.

I wonder if @garnaat is open to merging this stuff -- I just recalled that he had been toying with the idea of doing a rewrite of boto as boto3. Not sure if that is still the thought.

I'll be at PyCon US in March -- I wonder if this is something that folks might want to sprint on.

@kurin
@msabramo

That's great news that @garnaat is open to merging our changes. Yeah, a fork was tried before (the "neo" branch of this repo) and the problem was that it just got out of sync with the develop and master branches). Getting it merged into the mainline I think is the way to go and glad that @garnaat will consider it.

Yeah, most of my changes deal with bytes vs. text. We probably made a lot of the same changes.

If your fork passes integration tests as well, then maybe your fork is the better one to start with? I didn't try integration tests although there was a bit of manual testing that you can see above.

@kurin
@msabramo

The major change between your fork and mine is that you used 2to3 and I opted not to use it, because I wanted to see if I could make it work without the preprocessing. It does work, though it requires a lot more changes of mundane stuff that 2to3 could handle.

I wonder if @garnaat et al want to use 2to3 or not? If they do, then your fork is probably a better base.

@msabramo

@garnaat: Do you have any interest in this PR? The reason that I ask is I think that with the new version of six, which now has abstractions for various urllib and urlparse functions, it might be possible to revise this and simplify a bit. But I don't want to invest further time in this if it's not something that you're interested in merging.

@msabramo

@garnaat: Are you open to using something like this?

@msabramo

ping

@msabramo

OpenStack wants boto for Python 3. They are currently sprinting on Python 3 at Pycon and they have a dependency on boto. Do they have to wait for boto3? Maybe chat with them if you're at Pycon or ping @haypo who is spearheading the Python 3 effort...

@kurin

I'm not sure if you're pinging me or @garnaat, but for my part unfortunately I don't have any time to work on this.

@msabramo

Maybe @garnaat and @haypo might want to chat about what's the best thing for OpenStack to do. Perhaps OpenStack should be moving to something else like boto3 (currently has "experimental" and "not production-ready" stamped all over it) or botocore or aws-cli? @garnaat (or someone else) might have good advice here.

@msabramo msabramo closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Dec 2, 2012
  1. @msabramo
  2. @msabramo
  3. @msabramo
  4. @msabramo

    Remove simplejson from requirements.txt as it causes a failure with p…

    msabramo authored
    …y33 (and I think it's not needed with py26 and py27?)
  5. @msabramo

    Make tests/unit/__init__.py Python 3 friendly by abstracting out that…

    msabramo authored
    … stuff moved from httplib to http.client in Python 3
  6. @msabramo

    Make boto/ec2/connection.py Python 3 friendly by changing "except com…

    msabramo authored
    …ma" syntax to "except as" syntax
  7. @msabramo
  8. @msabramo
  9. @msabramo
  10. @msabramo
  11. @msabramo
  12. @msabramo
  13. @msabramo
  14. @msabramo
  15. @msabramo
  16. @msabramo
  17. @msabramo

    Use same default testenv command as on `develop` branch. Add a testen…

    msabramo authored
    …v for testing boto.ec2 only with py26
  18. @msabramo

    .travis.yml: Test Python 3.2 but only run tests for boto.ec2 since th…

    msabramo authored
    …at's all that I've ported to Python 3 so far.
Commits on Feb 12, 2013
  1. @msabramo

    Add py33 to .travis.yml

    msabramo authored
Commits on Feb 22, 2013
  1. @msabramo

    Fix infinite recursion error reported by @skiold on GitHub at

    msabramo authored
    boto#1156 (comment)
    
    I reverted back to qualifying stuff from `httplib` using `httplib.`;
    because otherwise `HTTPResponse` is ambiguous and may refer to itself
    instead of the version in `httplib`.
  2. @msabramo
  3. @msabramo

    Fix signature calculation bug by adding 3.2 and 3.3 to list of Python

    msabramo authored
    versions for which we should not append port to signature_host
  4. @msabramo
  5. @msabramo
This page is out of date. Refresh to see the latest.
View
7 .travis.yml
@@ -2,7 +2,10 @@ language: python
python:
- "2.6"
- "2.7"
+ - "3.2"
+ - "3.3"
before_install:
- sudo apt-get install swig
-install: pip install --use-mirrors -r requirements.txt
-script: python tests/test.py unit
+install: if [[ $(python --version 2>&1) == Python\ 3* ]]; then pip install -r requirements-py3.txt --use-mirrors; else pip install -r requirements.txt --use-mirrors; fi
+script: if [[ $(python --version 2>&1) == Python\ 3* ]]; then nosetests -v tests/unit/ec2; else python tests/test.py unit; fi
+
View
6 boto/__init__.py
@@ -32,7 +32,7 @@
import sys
import logging
import logging.config
-import urlparse
+from boto.compat23 import urlparse
from boto.exception import InvalidUriError
__version__ = '2.6.0-dev'
@@ -449,7 +449,7 @@ def connect_ec2_endpoint(url, aws_access_key_id=None,
"""
from boto.ec2.regioninfo import RegionInfo
- purl = urlparse.urlparse(url)
+ purl = urlparse(url)
kwargs['port'] = purl.port
kwargs['host'] = purl.hostname
kwargs['path'] = purl.path
@@ -673,7 +673,7 @@ def storage_uri(uri_str, default_scheme='file', debug=0, validate=True,
The last example uses the default scheme ('file', unless overridden)
"""
- # Manually parse URI components instead of using urlparse.urlparse because
+ # Manually parse URI components instead of using urlparse because
# what we're calling URIs don't really fit the standard syntax for URIs
# (the latter includes an optional host/net location part).
end_scheme_idx = uri_str.find('://')
View
28 boto/auth.py
@@ -71,6 +71,8 @@ def new(self, *args, **kwargs):
import sha
sha256 = None
+from boto.compat23 import ensure_bytes, quote, quote_plus
+
class HmacKeys(object):
"""Key based Auth handler helper."""
@@ -332,8 +334,8 @@ def query_string(self, http_request):
pairs = []
for pname in parameter_names:
pval = str(http_request.params[pname]).encode('utf-8')
- pairs.append(urllib.quote(pname, safe='') + '=' +
- urllib.quote(pval, safe='-_~'))
+ pairs.append(quote(pname, safe='') + '=' +
+ quote(pval, safe='-_~'))
return '&'.join(pairs)
def canonical_query_string(self, http_request):
@@ -344,8 +346,8 @@ def canonical_query_string(self, http_request):
l = []
for param in sorted(http_request.params):
value = str(http_request.params[param])
- l.append('%s=%s' % (urllib.quote(param, safe='-_.~'),
- urllib.quote(value, safe='-_.~')))
+ l.append('%s=%s' % (quote(param, safe='-_.~'),
+ quote(value, safe='-_.~')))
return '&'.join(l)
def canonical_headers(self, headers_to_sign):
@@ -416,6 +418,7 @@ def string_to_sign(self, http_request, canonical_request):
containing the original version of all headers that
were included in the StringToSign.
"""
+ canonical_request = ensure_bytes(canonical_request)
sts = ['AWS4-HMAC-SHA256']
sts.append(http_request.headers['X-Amz-Date'])
sts.append(self.credential_scope(http_request))
@@ -424,7 +427,7 @@ def string_to_sign(self, http_request, canonical_request):
def signature(self, http_request, string_to_sign):
key = self._provider.secret_key
- k_date = self._sign(('AWS4' + key).encode('utf-8'),
+ k_date = self._sign(('AWS4' + str(key)).encode('utf-8'),
http_request.timestamp)
k_region = self._sign(k_date, http_request.region_name)
k_service = self._sign(k_region, http_request.service_name)
@@ -490,7 +493,7 @@ def add_auth(self, http_request, **kwargs):
boto.log.debug('query_string: %s Signature: %s' % (qs, signature))
if http_request.method == 'POST':
headers['Content-Type'] = 'application/x-www-form-urlencoded; charset=UTF-8'
- http_request.body = qs + '&Signature=' + urllib.quote_plus(signature)
+ http_request.body = qs + '&Signature=' + quote_plus(signature)
http_request.headers['Content-Length'] = str(len(http_request.body))
else:
http_request.body = ''
@@ -498,7 +501,7 @@ def add_auth(self, http_request, **kwargs):
# already be there, we need to get rid of that and rebuild it
http_request.path = http_request.path.split('?')[0]
http_request.path = (http_request.path + '?' + qs +
- '&Signature=' + urllib.quote_plus(signature))
+ '&Signature=' + quote_plus(signature))
class QuerySignatureV0AuthHandler(QuerySignatureHelper, AuthHandler):
@@ -517,7 +520,7 @@ def _calc_signature(self, params, *args):
pairs = []
for key in keys:
val = boto.utils.get_utf8_value(params[key])
- pairs.append(key + '=' + urllib.quote(val))
+ pairs.append(key + '=' + quote(val))
qs = '&'.join(pairs)
return (qs, base64.b64encode(hmac.digest()))
@@ -545,7 +548,7 @@ def _calc_signature(self, params, *args):
hmac.update(key)
val = boto.utils.get_utf8_value(params[key])
hmac.update(val)
- pairs.append(key + '=' + urllib.quote(val))
+ pairs.append(key + '=' + quote(val))
qs = '&'.join(pairs)
return (qs, base64.b64encode(hmac.digest()))
@@ -568,11 +571,12 @@ def _calc_signature(self, params, verb, path, server_name):
pairs = []
for key in keys:
val = boto.utils.get_utf8_value(params[key])
- pairs.append(urllib.quote(key, safe='') + '=' +
- urllib.quote(val, safe='-_~'))
+ pairs.append(quote(key, safe='') + '=' +
+ quote(val, safe='-_~'))
qs = '&'.join(pairs)
boto.log.debug('query string: %s' % qs)
string_to_sign += qs
+ string_to_sign = ensure_bytes(string_to_sign)
boto.log.debug('string_to_sign: %s' % string_to_sign)
hmac.update(string_to_sign)
b64 = base64.b64encode(hmac.digest())
@@ -606,7 +610,7 @@ def add_auth(self, req, **kwargs):
# already be there, we need to get rid of that and rebuild it
req.path = req.path.split('?')[0]
req.path = (req.path + '?' + qs +
- '&Signature=' + urllib.quote_plus(signature))
+ '&Signature=' + quote_plus(signature))
def get_auth_handler(host, config, provider, requested_capability=None):
View
2  boto/auth_handler.py
@@ -23,7 +23,7 @@
Defines an interface which all Auth handlers need to implement.
"""
-from plugin import Plugin
+from boto.plugin import Plugin
class NotReadyToAuthenticate(Exception):
pass
View
42 boto/compat23.py
@@ -0,0 +1,42 @@
+import sys
+
+# True if we are running on Python 3.
+PY3 = sys.version_info[0] == 3
+
+if PY3:
+ integer_types = int,
+else:
+ integer_types = (int, long)
+
+try:
+ # Python 2
+ xrange = xrange
+except NameError:
+ xrange = range
+
+try:
+ # Python 2
+ basestring = basestring
+except NameError:
+ basestring = str
+
+try:
+ # Python 2
+ from urllib import quote, quote_plus
+except ImportError:
+ # Python 3
+ from urllib.parse import quote, quote_plus
+
+try:
+ # Python 2
+ from urlparse import urlparse
+except ImportError:
+ # Python 3
+ from urllib.parse import urlparse
+
+def ensure_bytes(bytes_or_str):
+ if isinstance(bytes_or_str, bytes) or bytes_or_str is None:
+ return bytes_or_str
+ else:
+ return bytes_or_str.encode('latin-1')
+
View
48 boto/connection.py
@@ -46,25 +46,33 @@
from __future__ import with_statement
import base64
import errno
-import httplib
+
+try:
+ # Python 2
+ import httplib
+except ImportError:
+ # Python 3
+ import http.client as httplib
+
+from boto.compat23 import ensure_bytes, integer_types, urlparse
+
import os
-import Queue
import random
import re
import socket
import sys
import time
import urllib
-import urlparse
+
import xml.sax
import copy
-import auth
-import auth_handler
import boto
import boto.utils
import boto.handler
import boto.cacerts
+import boto.auth
+import boto.auth_handler
from boto import config, UserAgent
from boto.exception import AWSConnectionError, BotoClientError
@@ -75,7 +83,7 @@
HAVE_HTTPS_CONNECTION = False
try:
import ssl
- from boto import https_connection
+ import boto.https_connection as https_connection
# Google App Engine runs on Python 2.5 so doesn't have ssl.SSLError.
if hasattr(ssl, 'SSLError'):
HAVE_HTTPS_CONNECTION = True
@@ -358,7 +366,7 @@ def __init__(self, method, protocol, host, port, path, auth_path,
del self.headers['Transfer-Encoding']
else:
self.headers = headers
- self.body = body
+ self.body = ensure_bytes(body)
def __str__(self):
return (('method:(%s) protocol:(%s) host(%s) port(%s) path(%s) '
@@ -369,8 +377,7 @@ def __str__(self):
def authorize(self, connection, **kwargs):
for key in self.headers:
val = self.headers[key]
- if isinstance(val, unicode):
- self.headers[key] = urllib.quote_plus(val.encode('utf-8'))
+ self.headers[key] = urllib.quote_plus(val)
connection._auth_handler.add_auth(self, **kwargs)
@@ -393,7 +400,7 @@ def read(self, amt=None):
"""Read the response.
This method does not have the same behavior as
- httplib.HTTPResponse.read. Instead, if this method is called with
+ HTTPResponse.read. Instead, if this method is called with
no ``amt`` arg, then the response body will be cached. Subsequent
calls to ``read()`` with no args **will return the cached response**.
@@ -485,7 +492,7 @@ def __init__(self, host, aws_access_key_id=None,
self.ca_certificates_file = config.get_value(
'Boto', 'ca_certificates_file', DEFAULT_CA_CERTS_FILE)
self.handle_proxy(proxy, proxy_port, proxy_user, proxy_pass)
- # define exceptions from httplib that we want to catch and retry
+ # define HTTP-related exceptions that we want to catch and retry
self.http_exceptions = (httplib.HTTPException, socket.error,
socket.gaierror, httplib.BadStatusLine)
# define subclasses of the above that are not retryable.
@@ -508,7 +515,7 @@ def __init__(self, host, aws_access_key_id=None,
self.host = host
self.path = path
# if the value passed in for debug
- if not isinstance(debug, (int, long)):
+ if not isinstance(debug, integer_types):
debug = 0
self.debug = config.getint('Boto', 'debug', debug)
if port:
@@ -544,7 +551,7 @@ def __init__(self, host, aws_access_key_id=None,
self._pool = ConnectionPool()
self._connection = (self.server_name(), self.is_secure)
self._last_rs = None
- self._auth_handler = auth.get_auth_handler(
+ self._auth_handler = boto.auth.get_auth_handler(
host, config, self.provider, self._required_auth_capability())
def __repr__(self):
@@ -610,7 +617,7 @@ def server_name(self, port=None):
# (and higher!)
# it no longer does that. Hence, this kludge.
if ((ON_APP_ENGINE and sys.version[:3] == '2.5') or
- sys.version[:3] in ('2.6', '2.7')) and port == 443:
+ sys.version[:3] in ('2.6', '2.7', '3.2', '3.3')) and port == 443:
signature_host = self.host
else:
signature_host = '%s:%d' % (self.host, port)
@@ -645,8 +652,8 @@ def handle_proxy(self, proxy, proxy_port, proxy_user, proxy_pass):
self.proxy_pass = config.get_value('Boto', 'proxy_pass', None)
if not self.proxy_port and self.proxy:
- print "http_proxy environment variable does not specify " \
- "a port, using default"
+ print("http_proxy environment variable does not specify " \
+ "a port, using default")
self.proxy_port = self.port
self.use_proxy = (self.proxy != None)
@@ -755,6 +762,7 @@ def proxy_ssl(self):
hostname, cert, 'hostname mismatch')
else:
# Fallback for old Python without ssl.wrap_socket
+ import httplib
if hasattr(httplib, 'ssl'):
sslSock = httplib.ssl.SSLSocket(sock)
else:
@@ -839,7 +847,7 @@ def _mexe(self, request, sender=None, override_num_retries=None,
return response
else:
scheme, request.host, request.path, \
- params, query, fragment = urlparse.urlparse(location)
+ params, query, fragment = urlparse(location)
if query:
request.path += '?' + query
msg = 'Redirecting: %s' % scheme + '://'
@@ -849,7 +857,7 @@ def _mexe(self, request, sender=None, override_num_retries=None,
scheme == 'https')
response = None
continue
- except self.http_exceptions, e:
+ except self.http_exceptions as e:
for unretryable in self.http_unretryable_exceptions:
if isinstance(e, unretryable):
boto.log.debug(
@@ -952,7 +960,7 @@ def make_request(self, action, params=None, path='/', verb='GET'):
return self._mexe(http_request)
def build_list_params(self, params, items, label):
- if isinstance(items, basestring):
+ if hasattr(items, 'startswith'):
items = [items]
for i in range(1, len(items) + 1):
params['%s.%d' % (label, i)] = items[i - 1]
@@ -965,6 +973,7 @@ def get_list(self, action, params, markers, path='/',
parent = self
response = self.make_request(action, params, path, verb)
body = response.read()
+ body = ensure_bytes(body)
boto.log.debug(body)
if not body:
boto.log.error('Null body %s' % body)
@@ -985,6 +994,7 @@ def get_object(self, action, params, cls, path='/',
parent = self
response = self.make_request(action, params, path, verb)
body = response.read()
+ body = ensure_bytes(body)
boto.log.debug(body)
if not body:
boto.log.error('Null body %s' % body)
View
2  boto/ec2/autoscale/__init__.py
@@ -45,6 +45,8 @@
from boto.ec2.autoscale.scheduled import ScheduledUpdateGroupAction
from boto.ec2.autoscale.tag import Tag
+from boto.compat23 import basestring, xrange
+
RegionData = {
'us-east-1': 'autoscaling.us-east-1.amazonaws.com',
'us-west-1': 'autoscaling.us-west-1.amazonaws.com',
View
2  boto/ec2/connection.py
@@ -2096,7 +2096,7 @@ def get_key_pair(self, keyname):
"""
try:
return self.get_all_key_pairs(keynames=[keyname])[0]
- except self.ResponseError, e:
+ except self.ResponseError as e:
if e.code == 'InvalidKeyPair.NotFound':
return None
else:
View
2  boto/ec2/keypair.py
@@ -83,7 +83,7 @@ def save(self, directory_path):
fp = open(file_path, 'wb')
fp.write(self.material)
fp.close()
- os.chmod(file_path, 0600)
+ os.chmod(file_path, 0o600)
return True
else:
raise BotoClientError('KeyPair contains no material')
View
14 boto/ec2/reservedinstance.py
@@ -81,13 +81,13 @@ def endElement(self, name, value, connection):
self.marketplace = True if value == 'true' else False
def describe(self):
- print 'ID=%s' % self.id
- print '\tInstance Type=%s' % self.instance_type
- print '\tZone=%s' % self.availability_zone
- print '\tDuration=%s' % self.duration
- print '\tFixed Price=%s' % self.fixed_price
- print '\tUsage Price=%s' % self.usage_price
- print '\tDescription=%s' % self.description
+ print('ID=%s' % self.id)
+ print('\tInstance Type=%s' % self.instance_type)
+ print('\tZone=%s' % self.availability_zone)
+ print('\tDuration=%s' % self.duration)
+ print('\tFixed Price=%s' % self.fixed_price)
+ print('\tUsage Price=%s' % self.usage_price)
+ print('\tDescription=%s' % self.description)
def purchase(self, instance_count=1):
return self.connection.purchase_reserved_instance_offering(self.id, instance_count)
View
14 boto/exception.py
@@ -30,13 +30,13 @@
from boto.resultset import ResultSet
-class BotoClientError(StandardError):
+class BotoClientError(Exception):
"""
General Boto Client error (error accessing AWS)
"""
def __init__(self, reason, *args):
- StandardError.__init__(self, reason, *args)
+ Exception.__init__(self, reason, *args)
self.reason = reason
def __repr__(self):
@@ -45,7 +45,7 @@ def __repr__(self):
def __str__(self):
return 'BotoClientError: %s' % self.reason
-class SDBPersistenceError(StandardError):
+class SDBPersistenceError(Exception):
pass
@@ -67,10 +67,10 @@ class GSPermissionsError(StoragePermissionsError):
"""
pass
-class BotoServerError(StandardError):
+class BotoServerError(Exception):
def __init__(self, status, reason, body=None, *args):
- StandardError.__init__(self, status, reason, body, *args)
+ Exception.__init__(self, status, reason, body, *args)
self.status = status
self.reason = reason
self.body = body or ''
@@ -85,7 +85,7 @@ def __init__(self, status, reason, body=None, *args):
try:
h = handler.XmlHandler(self, self)
xml.sax.parseString(self.body, h)
- except (TypeError, xml.sax.SAXParseException), pe:
+ except (TypeError, xml.sax.SAXParseException) as pe:
# Remove unparsable message body so we don't include garbage
# in exception. But first, save self.body in self.error_message
# because occasionally we get error messages from Eucalyptus
@@ -106,7 +106,7 @@ def __repr__(self):
def __str__(self):
return '%s: %s %s\n%s' % (self.__class__.__name__,
- self.status, self.reason, self.body)
+ self.status, self.reason, self.body.decode('utf-8'))
def startElement(self, name, attrs, connection):
pass
View
8 boto/https_connection.py
@@ -19,7 +19,13 @@
"""Extensions to allow HTTPS requests with SSL certificate validation."""
-import httplib
+try:
+ # Python 2
+ import httplib
+except ImportError:
+ # Python 3
+ import http.client as httplib
+
import re
import socket
import ssl
View
7 boto/provider.py
@@ -35,6 +35,7 @@
from boto.gs.acl import CannedACLStrings as CannedGSACLStrings
from boto.s3.acl import CannedACLStrings as CannedS3ACLStrings
from boto.s3.acl import Policy
+from boto.compat23 import ensure_bytes
HEADER_PREFIX_KEY = 'header_prefix'
@@ -272,11 +273,7 @@ def _populate_keys_from_metadata_server(self):
self._credential_expiry_time - datetime.now(), expires_at)
def _convert_key_to_str(self, key):
- if isinstance(key, unicode):
- # the secret key must be bytes and not unicode to work
- # properly with hmac.new (see http://bugs.python.org/issue5285)
- return str(key)
- return key
+ return ensure_bytes(key)
def configure_headers(self):
header_info_map = self.HeaderInfoMap[self.name]
View
39 boto/pyami/config.py
@@ -20,9 +20,24 @@
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
# IN THE SOFTWARE.
#
-import StringIO, os, re
+
+import os, re
import warnings
-import ConfigParser
+
+try:
+ # Python 2
+ from StringIO import StringIO
+except ImportError:
+ # Python 3
+ from io import StringIO
+
+try:
+ # Python 2
+ from ConfigParser import SafeConfigParser
+except ImportError:
+ # Python 3
+ from configparser import SafeConfigParser
+
import boto
# If running in Google App Engine there is no "user" and
@@ -55,10 +70,10 @@
BotoConfigLocations.append(expanduser(path))
-class Config(ConfigParser.SafeConfigParser):
+class Config(SafeConfigParser):
def __init__(self, path=None, fp=None, do_load=True):
- ConfigParser.SafeConfigParser.__init__(self, {'working_dir' : '/mnt/pyami',
+ SafeConfigParser.__init__(self, {'working_dir' : '/mnt/pyami',
'debug' : '0'})
if do_load:
if path:
@@ -76,7 +91,7 @@ def __init__(self, path=None, fp=None, do_load=True):
def load_credential_file(self, path):
"""Load a credential file as is setup like the Java utilities"""
- c_data = StringIO.StringIO()
+ c_data = StringIO()
c_data.write("[Credentials]\n")
for line in open(path, "r").readlines():
c_data.write(line.replace("AWSAccessKeyId", "aws_access_key_id").replace("AWSSecretKey", "aws_secret_access_key"))
@@ -99,7 +114,7 @@ def save_option(self, path, section, option, value):
Replace any previous value. If the path doesn't exist, create it.
Also add the option the the in-memory config.
"""
- config = ConfigParser.SafeConfigParser()
+ config = SafeConfigParser()
config.read(path)
if not config.has_section(section):
config.add_section(section)
@@ -143,21 +158,21 @@ def get_value(self, section, name, default=None):
def get(self, section, name, default=None):
try:
- val = ConfigParser.SafeConfigParser.get(self, section, name)
+ val = SafeConfigParser.get(self, section, name)
except:
val = default
return val
def getint(self, section, name, default=0):
try:
- val = ConfigParser.SafeConfigParser.getint(self, section, name)
+ val = SafeConfigParser.getint(self, section, name)
except:
val = int(default)
return val
def getfloat(self, section, name, default=0.0):
try:
- val = ConfigParser.SafeConfigParser.getfloat(self, section, name)
+ val = SafeConfigParser.getfloat(self, section, name)
except:
val = float(default)
return val
@@ -180,13 +195,13 @@ def setbool(self, section, name, value):
self.set(section, name, 'false')
def dump(self):
- s = StringIO.StringIO()
+ s = StringIO()
self.write(s)
- print s.getvalue()
+ print(s.getvalue())
def dump_safe(self, fp=None):
if not fp:
- fp = StringIO.StringIO()
+ fp = StringIO()
for section in self.sections():
fp.write('[%s]\n' % section)
for option in self.options(section):
View
73 boto/utils.py
@@ -41,10 +41,25 @@
import socket
import urllib
-import urllib2
+
+try:
+ # Python 2
+ from urllib2 import HTTPBasicAuthHandler, HTTPError, HTTPPasswordMgrWithDefaultRealm, Request, URLError, build_opener, install_opener, urlopen
+except ImportError:
+ # Python 3
+ from urllib.request import HTTPBasicAuthHandler, HTTPPasswordMgrWithDefaultRealm, Request, build_opener, install_opener, urlopen
+ from urllib.error import HTTPError, URLError
+
import imp
import subprocess
-import StringIO
+
+try:
+ # Python 2
+ from StringIO import StringIO
+except ImportError:
+ # Python 3
+ from io import BytesIO as StringIO
+
import time
import logging.handlers
import boto
@@ -53,11 +68,18 @@
import smtplib
import datetime
import re
-from email.MIMEMultipart import MIMEMultipart
-from email.MIMEBase import MIMEBase
-from email.MIMEText import MIMEText
-from email.Utils import formatdate
-from email import Encoders
+try:
+ # Python 2
+ from email.MIMEMultipart import MIMEMultipart
+ from email.MIMEBase import MIMEBase
+ from email.MIMEText import MIMEText
+except ImportError:
+ # Python 3
+ from email.mime.multipart import MIMEMultipart
+ from email.mime.base import MIMEBase
+ from email.mime.text import MIMEText
+from email.utils import formatdate
+from email import encoders
import gzip
import base64
try:
@@ -189,10 +211,10 @@ def get_aws_metadata(headers, provider=None):
def retry_url(url, retry_on_404=True, num_retries=10):
for i in range(0, num_retries):
try:
- req = urllib2.Request(url)
- resp = urllib2.urlopen(req)
+ req = Request(url)
+ resp = urlopen(req)
return resp.read()
- except urllib2.HTTPError, e:
+ except HTTPError as e:
# in 2.6 you use getcode(), in 2.5 and earlier you use code
if hasattr(e, 'getcode'):
code = e.getcode()
@@ -200,9 +222,9 @@ def retry_url(url, retry_on_404=True, num_retries=10):
code = e.code
if code == 404 and not retry_on_404:
return ''
- except urllib2.URLError, e:
+ except URLError as e:
raise e
- except Exception, e:
+ except Exception as e:
pass
boto.log.exception('Caught exception reading instance data')
time.sleep(2**i)
@@ -308,7 +330,7 @@ def get_instance_metadata(version='latest', url='http://169.254.169.254',
try:
return _get_instance_metadata('%s/%s/meta-data/' % (url, version),
num_retries=num_retries)
- except urllib2.URLError, e:
+ except URLError as e:
return None
finally:
if timeout is not None:
@@ -334,7 +356,7 @@ def get_instance_identity(version='latest', url='http://169.254.169.254',
if field:
iid[field] = val
return iid
- except urllib2.URLError, e:
+ except URLError as e:
return None
finally:
if timeout is not None:
@@ -392,7 +414,7 @@ def update_dme(username, password, dme_id, ip_address):
"""
dme_url = 'https://www.dnsmadeeasy.com/servlet/updateip'
dme_url += '?username=%s&password=%s&id=%s&ip=%s'
- s = urllib2.urlopen(dme_url % (username, password, dme_id, ip_address))
+ s = urlopen(dme_url % (username, password, dme_id, ip_address))
return s.read()
def fetch_file(uri, file=None, username=None, password=None):
@@ -414,12 +436,12 @@ def fetch_file(uri, file=None, username=None, password=None):
key.get_contents_to_file(file)
else:
if username and password:
- passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
+ passman = HTTPPasswordMgrWithDefaultRealm()
passman.add_password(None, uri, username, password)
- authhandler = urllib2.HTTPBasicAuthHandler(passman)
- opener = urllib2.build_opener(authhandler)
- urllib2.install_opener(opener)
- s = urllib2.urlopen(uri)
+ authhandler = HTTPBasicAuthHandler(passman)
+ opener = build_opener(authhandler)
+ install_opener(opener)
+ s = urlopen(uri)
file.write(s.read())
file.seek(0)
except:
@@ -433,7 +455,7 @@ class ShellCommand(object):
def __init__(self, command, wait=True, fail_fast=False, cwd = None):
self.exit_code = 0
self.command = command
- self.log_fp = StringIO.StringIO()
+ self.log_fp = StringIO()
self.wait = wait
self.fail_fast = fail_fast
self.run(cwd = cwd)
@@ -723,11 +745,12 @@ def notify(subject, body=None, html_body=None, to_string=None, attachments=None,
boto.log.exception('notify failed')
def get_utf8_value(value):
- if not isinstance(value, str) and not isinstance(value, unicode):
+ if not hasattr(value, 'startswith'):
value = str(value)
- if isinstance(value, unicode):
+
+ try:
return value.encode('utf-8')
- else:
+ except AttributeError:
return value
def mklist(value):
@@ -790,7 +813,7 @@ def write_mime_multipart(content, compress=False, deftype='text/plain', delimite
rcontent = wrapper.as_string()
if compress:
- buf = StringIO.StringIO()
+ buf = StringIO()
gz = gzip.GzipFile(mode='wb', fileobj=buf)
try:
gz.write(rcontent)
View
7 requirements-py3.txt
@@ -0,0 +1,7 @@
+mock==0.8.0
+nose==1.1.2
+M2Crypto==0.21.1
+requests==0.13.1
+tox==1.4
+Sphinx==1.1.3
+argparse==1.2.1
View
1  requirements.txt
@@ -4,6 +4,5 @@ M2Crypto==0.21.1
requests==0.13.1
tox==1.4
Sphinx==1.1.3
-simplejson==2.5.2
argparse==1.2.1
unittest2==0.5.1
View
14 tests/unit/__init__.py
@@ -2,7 +2,13 @@
import unittest2 as unittest
except ImportError:
import unittest
-import httplib
+
+try:
+ # Python 2
+ from httplib import HTTPSConnection, HTTPResponse
+except ImportError:
+ # Python 3
+ from http.client import HTTPSConnection, HTTPResponse
from mock import Mock
@@ -15,7 +21,7 @@ class AWSMockServiceTestCase(unittest.TestCase):
connection_class = None
def setUp(self):
- self.https_connection = Mock(spec=httplib.HTTPSConnection)
+ self.https_connection = Mock(spec=HTTPSConnection)
self.https_connection_factory = (
Mock(return_value=self.https_connection), ())
self.service_connection = self.create_service_connection(
@@ -39,7 +45,7 @@ def _mexe_spy(self, request, *args, **kwargs):
def create_response(self, status_code, reason='', header=[], body=None):
if body is None:
body = self.default_body()
- response = Mock(spec=httplib.HTTPResponse)
+ response = Mock(spec=HTTPResponse)
response.status = status_code
response.read.return_value = body
response.reason = reason
@@ -47,7 +53,7 @@ def create_response(self, status_code, reason='', header=[], body=None):
response.getheaders.return_value = header
def overwrite_header(arg, default=None):
header_dict = dict(header)
- if header_dict.has_key(arg):
+ if arg in header_dict:
return header_dict[arg]
else:
return default
View
39 tox.ini
@@ -1,8 +1,45 @@
[tox]
-envlist = py26,py27
+envlist = py26,py27,py32_ec2,py33_ec2
[testenv]
commands =
pip install -qr requirements.txt
+ # nosetests -v tests/test.py tests/unit
python tests/test.py tests/unit
+
+[testenv:py26_ec2]
+basepython = python2.6
+commands =
+ python --version
+ nosetests -v tests/unit/ec2
+deps =
+ mock==0.8.0
+ nose==1.1.2
+
+[testenv:py27_ec2]
+basepython = python2.7
+commands =
+ python --version
+ nosetests -v tests/unit/ec2
+deps =
+ mock==0.8.0
+ nose==1.1.2
+
+[testenv:py32_ec2]
+basepython = python3.2
+commands =
+ python --version
+ nosetests -v tests/unit/ec2
+deps =
+ mock==0.8.0
+ nose==1.1.2
+
+[testenv:py33_ec2]
+basepython = python3.3
+commands =
+ python --version
+ nosetests -v tests/unit/ec2
+deps =
+ mock==0.8.0
+ nose==1.1.2
Something went wrong with that request. Please try again.