Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Python/Jython 2.5 compatibility fixes for glacier #1072

Closed
wants to merge 1 commit into from

5 participants

@staer

Import with statement from future and attempt to import simplejson when json is not available.

@staer staer Python/Jython 2.5 compatibility fixes for glacier
Import with statement from __future__ and attempt to import simplejson when json is not available.
1476cf4
@jamesls
Owner

I'm seeing the import simplejson as json as an occuring idiom in boto. Perhaps we should just put that code in a module somewhere in boto so all code needing the json module can just say something like from boto.compat import json. This will also give >py2.6 users the simplejson speedups if they have it installed. @garnaat thoughts?

@kopertop
Owner
@staer

Something along the lines of what Django does for JSON support perhaps:

https://github.com/django/django/blob/master/django/utils/simplejson.py

@jamesls
Owner

I started this change here

But one thing that django does in the link above is to only use simplejson if it's newer that the system json module or if it has the c extension compiled. Should we do a simliar check in boto, or just use simplejson if it's installed?

@staer

From a backwards compatibility standpoint it might make more sense to actually import the system json first, on a failure to try simplejson so that people expecting the json module get it even if they have simplejson installed. This is likely a very small backwards compatibility gain though (how many people actually want json over simplejson if they happen to have it installed?). Changing it like this would also skirt the issue of caring what version of simplejson or if it's using C extension since if you dont' have a system json you dont' much care what version of simplejson is there so long as it is there.

@jamesls
Owner

My vote is to just keep it simple and use simplejson if it's available. This is how several existing modules already do it, and it's how this PR does it :) In the commit I linked above, I only changed the modules that have the try: import simplejson as json... line so it should be backwards compatible. To minimize impact, for now I won't change the modules that just have import json, but in the future if someone does want to switch over they can just change import json to from boto.compat import json.

Does this sound good?

@staer

Sounds good to me, I will likely go in to modify existing "import json" calls as we run boto under Jython (python 2.5) in production and we run into these compatibility issues somewhat frequently, with statements are also a pain point on older python versions as seen from my initial commit.

@garnaat
Owner

I think having a boto.compat module is a good idea. Whether we do that now or later, we can debate about that. But simplejson is faster than the builtin json module so I think it makes sense to try to import that first. I notice that boto/iam/connection.py does the import in the opposite order which is, I think, a great reason to have all of this wrapped up in a compat module.

@garnaat
Owner

boto/beanstalk/layer1.py just does "import json". We seem to be all over the map. My vote is to fix this once and for all by making a boto/compat module.

@jamesls
Owner

Sounds good. I have the compat module in a branch in my repo (jamesls@5fb324a). I'll go ahead and merge it in.

@staer

Should I make a separate pull request for the "with" statement compat fixes?

@jamesls
Owner

That'd be great. The glacier module should now be updated with the json compat imports.

@disruptek

It might make sense to use anyjson, a tiny library that ensures a consistent interface over the best available json. As an example, I've noticed that parse errors throw ValueError exceptions in json and JSONDecodeErrors in simplejson. Using anyjson doesn't add a significant requirement and if there's expertise there (0.3.3 versions worth?), then we'll benefit from it.

http://pypi.python.org/pypi/anyjson/0.3.3

@staer

This can probably be closed now that eac8807 has been committed to fix the JSON import issues. I'll check later to see if there are any that were missed, but it seems like you've got them all.

I've opened a separate pull request boto/boto#1085 to deal with the 'with' statements not having been imported from future to retain older python/jython compatability.

@staer staer closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Oct 22, 2012
  1. @staer

    Python/Jython 2.5 compatibility fixes for glacier

    staer authored
    Import with statement from __future__ and attempt to import simplejson when json is not available.
This page is out of date. Refresh to see the latest.
View
5 boto/glacier/exceptions.py
@@ -20,7 +20,10 @@
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
# IN THE SOFTWARE.
#
-import json
+try:
+ import simplejson as json
+except ImportError:
+ import json
class UnexpectedHTTPResponseError(Exception):
View
1  boto/glacier/job.py
@@ -20,6 +20,7 @@
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
# IN THE SOFTWARE.
#
+from __future__ import with_statement
import math
import socket
View
5 boto/glacier/response.py
@@ -20,7 +20,10 @@
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
# IN THE SOFTWARE.
#
-import json
+try:
+ import simplejson as json
+except ImportError:
+ import json
class GlacierResponse(dict):
"""
View
2  boto/glacier/vault.py
@@ -20,7 +20,7 @@
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
# IN THE SOFTWARE.
#
-
+from __future__ import with_statement
from .job import Job
from .writer import Writer, compute_hashes_from_fileobj
from .concurrent import ConcurrentUploader
View
5 boto/glacier/writer.py
@@ -25,7 +25,10 @@
import urllib
import hashlib
import math
-import json
+try:
+ import simplejson as json
+except ImportError:
+ import json
_ONE_MEGABYTE = 1024 * 1024
Something went wrong with that request. Please try again.