Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Add ability to use Decimal for DynamoDB numeric types #1183

Merged
merged 10 commits into from

3 participants

James Saryerwinnie Andy Davidoff Michael Waterfall
James Saryerwinnie
Owner

This expands on the work of @disruptek based on the comments in #1060.

The gist of this change is to refactor the
serialization/deserialization of python types to the format
that the dynamodb expects into a single "Dynamizer" interface.

This interface is then plumbed into Layer2 and can be passed in
via the __init__. This also allows users to entirely override
this process if they need.

I implemented two types of Dynamizers:

  • Dynamizer - Uses Decimals to handle numeric types.
  • LossyFloatDynamizer - Uses int/float to handle numeric types.

I made the default dynamizer the LossyFloatDynamizer which will
maintain backwards compatibility (all of the dynamodb tests pass).
I imagine (hope) after some time we can default to Dynamizer
instead.

If a user wants to use decimals, they can either pass in the
appropriate dynamizer:

  dynamodb = boto.connect_dynamodb(dynamizer=Dynamizer)

or use the use_decimals convenience method::

  dynamodb = boto.connect_dynamodb()
  dynamodb.use_decimals()

If a user wants to customize this process, they can subclass
Dynamizer and either override encode or decode, or they
can override a specific conversion. For example:

  class UnicodeDynamizer(Dynamizer):
    def decode_s(self, attr):
        return unicode(attr)

  dynamodb = boto.connect_dynamodb(dynamizer=UnicodeDynamizer)

I've added unittests/integration tests for the new functionality.

If this approach looks reasonable I can update the user docs, API docstrings,
etc. for the new changes. Just looking for feedback first.

@garnaat, @disruptek thoughts?

disruptek and others added some commits
Andy Davidoff disruptek use Decimal for numeric validate/store/retrieve 2e6936c
Andy Davidoff disruptek raise an exception on NaN and Infinity decimal values 6158a69
James Saryerwinnie jamesls Minor pep8 formatting 20808a1
James Saryerwinnie jamesls Add toggle to use Decimals with DynamoDB Layer2
The gist of this change is to refactor the
serialization/deserialization of python types to the format
that the dynamodb expects into a single "Dynamizer" interface.

This interface is then plumbed into Layer2 and can be passed in
via the __init__.  This also allows users to entirely override
this process if they need.

I implemented two types of Dynamizers:

* Dynamizer - Uses Decimals to handle numeric types.
* LossyFloatDynamizer - Uses int/float to handle numeric types.

I made the default dynamizer the LossyFloatDynamizer which will
maintain backwards compatibility (all of the dynamodb tests pass).

If a user wants to use decimals, they can either pass in the
appropriate dynamizer:

  dynamodb = boto.connect_dynamodb(dynamizer=Dynamizer)

or use the `use_decimals` convenience method::

  dynamodb = boto.connect_dynamodb()
  dynamodb.use_decimals()

If a user wants to customize this process, they can subclass
Dynamizer and either override `encode` or `decode`, or they
can override a specific conversion.  For example:

  class UnicodeDynamizer(Dynamizer):
    def decode_s(self, attr):
        return unicode(attr)

  dynamodb = boto.connect_dynamodb(dynamizer=UnicodeDynamizer)

I've added unittests/integration tests for the new functionality.
6767cec
James Saryerwinnie jamesls Fix deprecated warning in python2.6 2195b8b
James Saryerwinnie
Owner

Well, it turns out you can't directly convert floats to decimals in python2.6 (this is the failing test). There's a recipe in the FAQ section for the decimal module that I'll try out.

James Saryerwinnie jamesls Fix python2.6 bug converting floats to Decimals
tox is now happy with all the unittests.
5211685
Andy Davidoff

Thanks for working on this; this issue has been gnawing at me for awhile now. Since your implementation matches my original proposal from 4 months ago, of course I'm in favor of it. :-)

I have only a couple comments on the code:

  • You suggested in #1060 that in Dynamize.encode_n() we narrow the exception handling to the TypeError we raise therein or to an appropriate Decimal exception. That change makes sense to me, did you simply overlook it?
  • It seems to me that Dynamize._get_dynamodb_type() shouldn't be _private, as it will probably need to be reimplemented in most subclasses and as such represents an interface for the user. Maybe I'm overlooking a key detail, but it's hard to think of a way to avoid reimplementing this method in any Real World subclass. If the point of the _private designation is to suggest that it will not be called outside the class, then I wonder if all the encode_X/decode_X methods should be similarly _private.
James Saryerwinnie jamesls Incorporate review feedback from disruptek
* Don't catch Exception, catch more specific exceptions
  when converting to Decimal.
* Mark the encode_X/decode_X methods as internal (client should
  use encode/decode which will delegate to the proper
  encode_X/decode_X method.
2ef10ba
James Saryerwinnie
Owner
  • I overlooked that, I've updated the code accordingly.
  • My intent with marking the methods _internal with the leading underscore is that it's not intended to be called outside the class. I wanted a simple public API with just two methods encode and decode. Subclasses are free to call these methods in any way they need. So, as per your suggestion, I've changed the encode_X/decode_X methods as internal.

Great feedback, thanks for reviewing the PR.

James Saryerwinnie jamesls Add an integration test for large integers
Noticed #1176 mentioned large integers so I figured
added an explicit integration test for this wouldn't be
a bad idea.
6f9ff24
James Saryerwinnie
Owner

Ok, so it sounds like I'll go ahead and flesh the rest of this change out (which I believe is mostly just docstring/comments and possibly an update to the dynamodb tutorial).

@garnaat Any thoughts?

James Saryerwinnie jamesls Add docs for dynamodb decimal changes
* Added docstrings
* Updated tutorial with section on using decimals
* Added boto.dynamodb.types to the list of modules
  for which to generate API docs.
7972dda
James Saryerwinnie jamesls Merge branch 'develop' into dynamodb-decimal
Conflicts:
	docs/source/dynamodb_tut.rst
d6f853a
James Saryerwinnie jamesls merged commit d6f853a into from
James Saryerwinnie jamesls deleted the branch
Michael Waterfall

Is there a reason that Decimal is being used for both whole and fractional numbers? As opposed to using int for integers and only replacing floats with Decimal?

Currently, when pulling items from DynamoDB using Decimals entirely, it's not possible for me to determine which fields/columns are integers and which ones are fractional.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Dec 11, 2012
  1. Andy Davidoff James Saryerwinnie

    use Decimal for numeric validate/store/retrieve

    disruptek authored jamesls committed
  2. Andy Davidoff James Saryerwinnie

    raise an exception on NaN and Infinity decimal values

    disruptek authored jamesls committed
  3. James Saryerwinnie

    Minor pep8 formatting

    jamesls authored
Commits on Dec 12, 2012
  1. James Saryerwinnie

    Add toggle to use Decimals with DynamoDB Layer2

    jamesls authored
    The gist of this change is to refactor the
    serialization/deserialization of python types to the format
    that the dynamodb expects into a single "Dynamizer" interface.
    
    This interface is then plumbed into Layer2 and can be passed in
    via the __init__.  This also allows users to entirely override
    this process if they need.
    
    I implemented two types of Dynamizers:
    
    * Dynamizer - Uses Decimals to handle numeric types.
    * LossyFloatDynamizer - Uses int/float to handle numeric types.
    
    I made the default dynamizer the LossyFloatDynamizer which will
    maintain backwards compatibility (all of the dynamodb tests pass).
    
    If a user wants to use decimals, they can either pass in the
    appropriate dynamizer:
    
      dynamodb = boto.connect_dynamodb(dynamizer=Dynamizer)
    
    or use the `use_decimals` convenience method::
    
      dynamodb = boto.connect_dynamodb()
      dynamodb.use_decimals()
    
    If a user wants to customize this process, they can subclass
    Dynamizer and either override `encode` or `decode`, or they
    can override a specific conversion.  For example:
    
      class UnicodeDynamizer(Dynamizer):
        def decode_s(self, attr):
            return unicode(attr)
    
      dynamodb = boto.connect_dynamodb(dynamizer=UnicodeDynamizer)
    
    I've added unittests/integration tests for the new functionality.
  2. James Saryerwinnie
  3. James Saryerwinnie

    Fix python2.6 bug converting floats to Decimals

    jamesls authored
    tox is now happy with all the unittests.
Commits on Dec 13, 2012
  1. James Saryerwinnie

    Incorporate review feedback from disruptek

    jamesls authored
    * Don't catch Exception, catch more specific exceptions
      when converting to Decimal.
    * Mark the encode_X/decode_X methods as internal (client should
      use encode/decode which will delegate to the proper
      encode_X/decode_X method.
Commits on Dec 14, 2012
  1. James Saryerwinnie

    Add an integration test for large integers

    jamesls authored
    Noticed #1176 mentioned large integers so I figured
    added an explicit integration test for this wouldn't be
    a bad idea.
Commits on Dec 21, 2012
  1. James Saryerwinnie

    Add docs for dynamodb decimal changes

    jamesls authored
    * Added docstrings
    * Updated tutorial with section on using decimals
    * Added boto.dynamodb.types to the list of modules
      for which to generate API docs.
Commits on Jan 8, 2013
  1. James Saryerwinnie

    Merge branch 'develop' into dynamodb-decimal

    jamesls authored
    Conflicts:
    	docs/source/dynamodb_tut.rst
This page is out of date. Refresh to see the latest.
7 boto/dynamodb/exceptions.py
View
@@ -29,6 +29,13 @@ class DynamoDBItemError(BotoClientError):
pass
+class DynamoDBNumberError(BotoClientError):
+ """
+ Raised in the event of incompatible numeric type casting.
+ """
+ pass
+
+
class DynamoDBConditionalCheckFailedError(DynamoDBResponseError):
"""
Raised when a ConditionalCheckFailedException response is received.
51 boto/dynamodb/layer2.py
View
@@ -25,8 +25,8 @@
from boto.dynamodb.schema import Schema
from boto.dynamodb.item import Item
from boto.dynamodb.batch import BatchList, BatchWriteList
-from boto.dynamodb.types import get_dynamodb_type, dynamize_value, \
- item_object_hook
+from boto.dynamodb.types import get_dynamodb_type, Dynamizer, \
+ LossyFloatDynamizer
def table_generator(tgen):
@@ -99,11 +99,24 @@ class Layer2(object):
def __init__(self, aws_access_key_id=None, aws_secret_access_key=None,
is_secure=True, port=None, proxy=None, proxy_port=None,
debug=0, security_token=None, region=None,
- validate_certs=True):
+ validate_certs=True, dynamizer=LossyFloatDynamizer):
self.layer1 = Layer1(aws_access_key_id, aws_secret_access_key,
is_secure, port, proxy, proxy_port,
debug, security_token, region,
validate_certs=validate_certs)
+ self.dynamizer = dynamizer()
+
+ def use_decimals(self):
+ """
+ Use the ``decimal.Decimal`` type for encoding/decoding numeric types.
+
+ By default, ints/floats are used to represent numeric types
+ ('N', 'NS') received from DynamoDB. Using the ``Decimal``
+ type is recommended to prevent loss of precision.
+
+ """
+ # Eventually this should be made the default dynamizer.
+ self.dynamizer = Dynamizer()
def dynamize_attribute_updates(self, pending_updates):
"""
@@ -118,13 +131,13 @@ def dynamize_attribute_updates(self, pending_updates):
d[attr_name] = {"Action": action}
else:
d[attr_name] = {"Action": action,
- "Value": dynamize_value(value)}
+ "Value": self.dynamizer.encode(value)}
return d
def dynamize_item(self, item):
d = {}
for attr_name in item:
- d[attr_name] = dynamize_value(item[attr_name])
+ d[attr_name] = self.dynamizer.encode(item[attr_name])
return d
def dynamize_range_key_condition(self, range_key_condition):
@@ -162,7 +175,7 @@ def dynamize_expected_value(self, expected_value):
elif attr_value is False:
attr_value = {'Exists': False}
else:
- val = dynamize_value(expected_value[attr_name])
+ val = self.dynamizer.encode(expected_value[attr_name])
attr_value = {'Value': val}
d[attr_name] = attr_value
return d
@@ -175,10 +188,10 @@ def dynamize_last_evaluated_key(self, last_evaluated_key):
d = None
if last_evaluated_key:
hash_key = last_evaluated_key['HashKeyElement']
- d = {'HashKeyElement': dynamize_value(hash_key)}
+ d = {'HashKeyElement': self.dynamizer.encode(hash_key)}
if 'RangeKeyElement' in last_evaluated_key:
range_key = last_evaluated_key['RangeKeyElement']
- d['RangeKeyElement'] = dynamize_value(range_key)
+ d['RangeKeyElement'] = self.dynamizer.encode(range_key)
return d
def build_key_from_values(self, schema, hash_key, range_key=None):
@@ -202,13 +215,13 @@ def build_key_from_values(self, schema, hash_key, range_key=None):
type defined in the schema.
"""
dynamodb_key = {}
- dynamodb_value = dynamize_value(hash_key)
+ dynamodb_value = self.dynamizer.encode(hash_key)
if dynamodb_value.keys()[0] != schema.hash_key_type:
msg = 'Hashkey must be of type: %s' % schema.hash_key_type
raise TypeError(msg)
dynamodb_key['HashKeyElement'] = dynamodb_value
if range_key is not None:
- dynamodb_value = dynamize_value(range_key)
+ dynamodb_value = self.dynamizer.encode(range_key)
if dynamodb_value.keys()[0] != schema.range_key_type:
msg = 'RangeKey must be of type: %s' % schema.range_key_type
raise TypeError(msg)
@@ -425,7 +438,7 @@ def get_item(self, table, hash_key, range_key=None,
key = self.build_key_from_values(table.schema, hash_key, range_key)
response = self.layer1.get_item(table.name, key,
attributes_to_get, consistent_read,
- object_hook=item_object_hook)
+ object_hook=self.dynamizer.decode)
item = item_class(table, hash_key, range_key, response['Item'])
if 'ConsumedCapacityUnits' in response:
item.consumed_units = response['ConsumedCapacityUnits']
@@ -445,7 +458,7 @@ def batch_get_item(self, batch_list):
"""
request_items = batch_list.to_dict()
return self.layer1.batch_get_item(request_items,
- object_hook=item_object_hook)
+ object_hook=self.dynamizer.decode)
def batch_write_item(self, batch_list):
"""
@@ -459,7 +472,7 @@ def batch_write_item(self, batch_list):
"""
request_items = batch_list.to_dict()
return self.layer1.batch_write_item(request_items,
- object_hook=item_object_hook)
+ object_hook=self.dynamizer.decode)
def put_item(self, item, expected_value=None, return_values=None):
"""
@@ -487,7 +500,7 @@ def put_item(self, item, expected_value=None, return_values=None):
response = self.layer1.put_item(item.table.name,
self.dynamize_item(item),
expected_value, return_values,
- object_hook=item_object_hook)
+ object_hook=self.dynamizer.decode)
if 'ConsumedCapacityUnits' in response:
item.consumed_units = response['ConsumedCapacityUnits']
return response
@@ -528,7 +541,7 @@ def update_item(self, item, expected_value=None, return_values=None):
response = self.layer1.update_item(item.table.name, key,
attr_updates,
expected_value, return_values,
- object_hook=item_object_hook)
+ object_hook=self.dynamizer.decode)
item._updates.clear()
if 'ConsumedCapacityUnits' in response:
item.consumed_units = response['ConsumedCapacityUnits']
@@ -561,7 +574,7 @@ def delete_item(self, item, expected_value=None, return_values=None):
return self.layer1.delete_item(item.table.name, key,
expected=expected_value,
return_values=return_values,
- object_hook=item_object_hook)
+ object_hook=self.dynamizer.decode)
def query(self, table, hash_key, range_key_condition=None,
attributes_to_get=None, request_limit=None,
@@ -640,14 +653,14 @@ def query(self, table, hash_key, range_key_condition=None,
else:
esk = None
kwargs = {'table_name': table.name,
- 'hash_key_value': dynamize_value(hash_key),
+ 'hash_key_value': self.dynamizer.encode(hash_key),
'range_key_conditions': rkc,
'attributes_to_get': attributes_to_get,
'limit': request_limit,
'consistent_read': consistent_read,
'scan_index_forward': scan_index_forward,
'exclusive_start_key': esk,
- 'object_hook': item_object_hook}
+ 'object_hook': self.dynamizer.decode}
return TableGenerator(table, self.layer1.query,
max_results, item_class, kwargs)
@@ -728,6 +741,6 @@ def scan(self, table, scan_filter=None,
'limit': request_limit,
'count': count,
'exclusive_start_key': esk,
- 'object_hook': item_object_hook}
+ 'object_hook': self.dynamizer.decode}
return TableGenerator(table, self.layer1.scan,
max_results, item_class, kwargs)
187 boto/dynamodb/types.py
View
@@ -25,10 +25,33 @@
Python types and vice-versa.
"""
import base64
+from decimal import (Decimal, DecimalException, Context,
+ Clamped, Overflow, Inexact, Underflow, Rounded)
+from exceptions import DynamoDBNumberError
+
+
+DYNAMODB_CONTEXT = Context(
+ Emin=-128, Emax=126, rounding=None, prec=38,
+ traps=[Clamped, Overflow, Inexact, Rounded, Underflow])
+
+
+# python2.6 cannot convert floats directly to
+# Decimals. This is taken from:
+# http://docs.python.org/release/2.6.7/library/decimal.html#decimal-faq
+def float_to_decimal(f):
+ n, d = f.as_integer_ratio()
+ numerator, denominator = Decimal(n), Decimal(d)
+ ctx = DYNAMODB_CONTEXT
+ result = ctx.divide(numerator, denominator)
+ while ctx.flags[Inexact]:
+ ctx.flags[Inexact] = False
+ ctx.prec *= 2
+ result = ctx.divide(numerator, denominator)
+ return result
def is_num(n):
- types = (int, long, float, bool)
+ types = (int, long, float, bool, Decimal)
return isinstance(n, types) or n in types
@@ -41,6 +64,15 @@ def is_binary(n):
return isinstance(n, Binary)
+def serialize_num(val):
+ """Cast a number to a string and perform
+ validation to ensure no loss of precision.
+ """
+ if isinstance(val, bool):
+ return str(int(val))
+ return str(val)
+
+
def convert_num(s):
if '.' in s:
n = float(s)
@@ -86,23 +118,13 @@ def dynamize_value(val):
needs to be sent to Amazon DynamoDB. If the type of the value
is not supported, raise a TypeError
"""
- def _str(val):
- """
- DynamoDB stores booleans as numbers. True is 1, False is 0.
- This function converts Python booleans into DynamoDB friendly
- representation.
- """
- if isinstance(val, bool):
- return str(int(val))
- return str(val)
-
dynamodb_type = get_dynamodb_type(val)
if dynamodb_type == 'N':
- val = {dynamodb_type: _str(val)}
+ val = {dynamodb_type: serialize_num(val)}
elif dynamodb_type == 'S':
val = {dynamodb_type: val}
elif dynamodb_type == 'NS':
- val = {dynamodb_type: [str(n) for n in val]}
+ val = {dynamodb_type: map(serialize_num, val)}
elif dynamodb_type == 'SS':
val = {dynamodb_type: [n for n in val]}
elif dynamodb_type == 'B':
@@ -159,3 +181,142 @@ def item_object_hook(dct):
if 'BS' in dct:
return set(map(convert_binary, dct['BS']))
return dct
+
+
+class Dynamizer(object):
+ """Control serialization/deserialization of types.
+
+ This class controls the encoding of python types to the
+ format that is expected by the DynamoDB API, as well as
+ taking DynamoDB types and constructing the appropriate
+ python types.
+
+ If you want to customize this process, you can subclass
+ this class and override the encoding/decoding of
+ specific types. For example::
+
+ 'foo' (Python type)
+ |
+ v
+ encode('foo')
+ |
+ v
+ _encode_s('foo')
+ |
+ v
+ {'S': 'foo'} (Encoding sent to/received from DynamoDB)
+ |
+ V
+ decode({'S': 'foo'})
+ |
+ v
+ _decode_s({'S': 'foo'})
+ |
+ v
+ 'foo' (Python type)
+
+ """
+ def _get_dynamodb_type(self, attr):
+ return get_dynamodb_type(attr)
+
+ def encode(self, attr):
+ """
+ Encodes a python type to the format expected
+ by DynamoDB.
+
+ """
+ dynamodb_type = self._get_dynamodb_type(attr)
+ try:
+ encoder = getattr(self, '_encode_%s' % dynamodb_type.lower())
+ except AttributeError:
+ raise ValueError("Unable to encode dynamodb type: %s" %
+ dynamodb_type)
+ return {dynamodb_type: encoder(attr)}
+
+ def _encode_n(self, attr):
+ try:
+ if isinstance(attr, float) and not hasattr(Decimal, 'from_float'):
+ # python2.6 does not support creating Decimals directly
+ # from floats so we have to do this ourself.
+ n = str(float_to_decimal(attr))
+ else:
+ n = str(DYNAMODB_CONTEXT.create_decimal(attr))
+ if filter(lambda x: x in n, ('Infinity', 'NaN')):
+ raise TypeError('Infinity and NaN not supported')
+ return n
+ except (TypeError, DecimalException), e:
+ msg = '{0} numeric for `{1}`\n{2}'.format(
+ e.__class__.__name__, attr, str(e) or '')
+ raise DynamoDBNumberError(msg)
+
+ def _encode_s(self, attr):
+ return str(attr)
+
+ def _encode_ns(self, attr):
+ return map(self._encode_n, attr)
+
+ def _encode_ss(self, attr):
+ return [self._encode_s(n) for n in attr]
+
+ def _encode_b(self, attr):
+ return attr.encode()
+
+ def _encode_bs(self, attr):
+ return [self._encode_b(n) for n in attr]
+
+ def decode(self, attr):
+ """
+ Takes the format returned by DynamoDB and constructs
+ the appropriate python type.
+
+ """
+ if len(attr) > 1 or not attr:
+ return attr
+ dynamodb_type = attr.keys()[0]
+ try:
+ decoder = getattr(self, '_decode_%s' % dynamodb_type.lower())
+ except AttributeError:
+ return attr
+ return decoder(attr[dynamodb_type])
+
+ def _decode_n(self, attr):
+ return DYNAMODB_CONTEXT.create_decimal(attr)
+
+ def _decode_s(self, attr):
+ return attr
+
+ def _decode_ns(self, attr):
+ return set(map(self._decode_n, attr))
+
+ def _decode_ss(self, attr):
+ return set(map(self._decode_s, attr))
+
+ def _decode_b(self, attr):
+ return convert_binary(attr)
+
+ def _decode_bs(self, attr):
+ return set(map(self._decode_b, attr))
+
+
+class LossyFloatDynamizer(Dynamizer):
+ """Use float/int instead of Decimal for numeric types.
+
+ This class is provided for backwards compatibility. Instead of
+ using Decimals for the 'N', 'NS' types it uses ints/floats.
+
+ This class is deprecated and its usage is not encouraged,
+ as doing so may result in loss of precision. Use the
+ `Dynamizer` class instead.
+
+ """
+ def _encode_n(self, attr):
+ return serialize_num(attr)
+
+ def _encode_ns(self, attr):
+ return [str(i) for i in attr]
+
+ def _decode_n(self, attr):
+ return convert_num(attr)
+
+ def _decode_ns(self, attr):
+ return set(map(self._decode_n, attr))
30 docs/source/dynamodb_tut.rst
View
@@ -272,6 +272,36 @@ To update an item's attributes, simply retrieve it, modify the value, then
>>> item['SentBy'] = 'User B'
>>> item.put()
+Working with Decimals
+---------------------
+
+To avoid the loss of precision, you can stipulate that the
+``decimal.Decimal`` type be used for numeric values::
+
+ >>> import decimal
+ >>> conn.use_decimals()
+ >>> table = conn.get_table('messages')
+ >>> item = table.new_item(
+ hash_key='LOLCat Forum',
+ range_key='Check this out!'
+ )
+ >>> item['decimal_type'] = decimal.Decimal('1.12345678912345')
+ >>> item.put()
+ >>> print table.get_item('LOLCat Forum', 'Check this out!')
+ {u'forum_name': 'LOLCat Forum', u'decimal_type': Decimal('1.12345678912345'),
+ u'subject': 'Check this out!'}
+
+You can enable the usage of ``decimal.Decimal`` by using either the ``use_decimals``
+method, or by passing in the
+:py:class:`Dynamizer <boto.dynamodb.types.Dynamizer>` class for
+the ``dynamizer`` param::
+
+ >>> from boto.dynamodb.types import Dynamizer
+ >>> conn = boto.connect_dynamodb(dynamizer=Dynamizer)
+
+This mechanism can also be used if you want to customize the encoding/decoding
+process of DynamoDB types.
+
Deleting Items
--------------
7 docs/source/ref/dynamodb.rst
View
@@ -8,7 +8,7 @@ boto.dynamodb
-------------
.. automodule:: boto.dynamodb
- :members:
+ :members:
:undoc-members:
boto.dynamodb.layer1
@@ -53,4 +53,9 @@ boto.dynamodb.batch
:members:
:undoc-members:
+boto.dynamodb.types
+-------------------
+.. automodule:: boto.dynamodb.types
+ :members:
+ :undoc-members:
46 tests/integration/dynamodb/test_layer2.py
View
@@ -23,10 +23,11 @@
"""
Tests for Layer2 of Amazon DynamoDB
"""
-
import unittest
import time
import uuid
+from decimal import Decimal
+
from boto.dynamodb.exceptions import DynamoDBKeyNotFoundError
from boto.dynamodb.exceptions import DynamoDBConditionalCheckFailedError
from boto.dynamodb.layer2 import Layer2
@@ -43,6 +44,16 @@ def setUp(self):
self.hash_key_proto_value = ''
self.range_key_name = 'subject'
self.range_key_proto_value = ''
+ self.table_name = 'sample_data_%s' % int(time.time())
+
+ def create_sample_table(self):
+ schema = self.dynamodb.create_schema(
+ self.hash_key_name, self.hash_key_proto_value,
+ self.range_key_name,
+ self.range_key_proto_value)
+ table = self.create_table(self.table_name, schema, 5, 5)
+ table.refresh(wait_for_active=True)
+ return table
def create_table(self, table_name, schema, read_units, write_units):
result = self.dynamodb.create_table(table_name, schema, read_units, write_units)
@@ -428,3 +439,36 @@ def test_binary_attrs(self):
self.assertEqual(retrieved['BinaryData'], bytes('\x01\x02\x03\x04'))
self.assertEqual(retrieved['BinarySequence'],
set([Binary('\x01\x02'), Binary('\x03\x04')]))
+
+ def test_put_decimal_attrs(self):
+ self.dynamodb.use_decimals()
+ table = self.create_sample_table()
+ item = table.new_item('foo', 'bar')
+ item['decimalvalue'] = Decimal('1.12345678912345')
+ item.put()
+ retrieved = table.get_item('foo', 'bar')
+ self.assertEqual(retrieved['decimalvalue'], Decimal('1.12345678912345'))
+
+ def test_lossy_float_conversion(self):
+ table = self.create_sample_table()
+ item = table.new_item('foo', 'bar')
+ item['floatvalue'] = 1.12345678912345
+ item.put()
+ retrieved = table.get_item('foo', 'bar')['floatvalue']
+ # Notice how this is not equal to the original value.
+ self.assertNotEqual(1.12345678912345, retrieved)
+ # Instead, it's truncated:
+ self.assertEqual(1.12345678912, retrieved)
+
+ def test_large_integers(self):
+ # It's not just floating point numbers, large integers
+ # can trigger rouding issues.
+ self.dynamodb.use_decimals()
+ table = self.create_sample_table()
+ item = table.new_item('foo', 'bar')
+ item['decimalvalue'] = Decimal('129271300103398600')
+ item.put()
+ retrieved = table.get_item('foo', 'bar')
+ self.assertEqual(retrieved['decimalvalue'], Decimal('129271300103398600'))
+ # Also comparable directly to an int.
+ self.assertEqual(retrieved['decimalvalue'], 129271300103398600)
82 tests/unit/dynamodb/test_types.py
View
@@ -0,0 +1,82 @@
+#!/usr/bin/env python
+# Copyright (c) 2012 Amazon.com, Inc. or its affiliates. All Rights Reserved
+#
+# Permission is hereby granted, free of charge, to any person obtaining a
+# copy of this software and associated documentation files (the
+# "Software"), to deal in the Software without restriction, including
+# without limitation the rights to use, copy, modify, merge, publish, dis-
+# tribute, sublicense, and/or sell copies of the Software, and to permit
+# persons to whom the Software is furnished to do so, subject to the fol-
+# lowing conditions:
+#
+# The above copyright notice and this permission notice shall be included
+# in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
+# OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABIL-
+# ITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT
+# SHALL THE AUTHOR BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+# WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+# IN THE SOFTWARE.
+#
+from decimal import Decimal
+from tests.unit import unittest
+
+from boto.dynamodb import types
+from boto.dynamodb.exceptions import DynamoDBNumberError
+
+
+class TestDynamizer(unittest.TestCase):
+ def setUp(self):
+ pass
+
+ def test_encoding_to_dynamodb(self):
+ dynamizer = types.Dynamizer()
+ self.assertEqual(dynamizer.encode('foo'), {'S': 'foo'})
+ self.assertEqual(dynamizer.encode(54), {'N': '54'})
+ self.assertEqual(dynamizer.encode(Decimal('1.1')), {'N': '1.1'})
+ self.assertEqual(dynamizer.encode(set([1, 2, 3])),
+ {'NS': ['1', '2', '3']})
+ self.assertEqual(dynamizer.encode(set(['foo', 'bar'])),
+ {'SS': ['foo', 'bar']})
+ self.assertEqual(dynamizer.encode(types.Binary('\x01')),
+ {'B': 'AQ=='})
+ self.assertEqual(dynamizer.encode(set([types.Binary('\x01')])),
+ {'BS': ['AQ==']})
+
+ def test_decoding_to_dynamodb(self):
+ dynamizer = types.Dynamizer()
+ self.assertEqual(dynamizer.decode({'S': 'foo'}), 'foo')
+ self.assertEqual(dynamizer.decode({'N': '54'}), 54)
+ self.assertEqual(dynamizer.decode({'N': '1.1'}), Decimal('1.1'))
+ self.assertEqual(dynamizer.decode({'NS': ['1', '2', '3']}),
+ set([1, 2, 3]))
+ self.assertEqual(dynamizer.decode({'SS': ['foo', 'bar']}),
+ set(['foo', 'bar']))
+ self.assertEqual(dynamizer.decode({'B': 'AQ=='}), types.Binary('\x01'))
+ self.assertEqual(dynamizer.decode({'BS': ['AQ==']}),
+ set([types.Binary('\x01')]))
+
+ def test_float_conversion_errors(self):
+ dynamizer = types.Dynamizer()
+ # When supporting decimals, certain floats will work:
+ self.assertEqual(dynamizer.encode(1.25), {'N': '1.25'})
+ # And some will generate errors, which is why it's best
+ # to just use Decimals directly:
+ with self.assertRaises(DynamoDBNumberError):
+ dynamizer.encode(1.1)
+
+ def test_lossy_float_conversions(self):
+ dynamizer = types.LossyFloatDynamizer()
+ # Just testing the differences here, specifically float conversions:
+ self.assertEqual(dynamizer.encode(1.1), {'N': '1.1'})
+ self.assertEqual(dynamizer.decode({'N': '1.1'}), 1.1)
+
+ self.assertEqual(dynamizer.encode(set([1.1])),
+ {'NS': ['1.1']})
+ self.assertEqual(dynamizer.decode({'NS': ['1.1', '2.2', '3.3']}),
+ set([1.1, 2.2, 3.3]))
+
+if __name__ == '__main__':
+ unittest.main()
Something went wrong with that request. Please try again.