New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How can I store 30.40 in DynamoDb? #665

Open
jonapich opened this Issue May 31, 2016 · 13 comments

Comments

Projects
None yet
@jonapich
Copy link

jonapich commented May 31, 2016

It's the end of the day, maybe i'm not seeing this clearly... But I am unable to store simple float values without adding some arcane magic to the mix.

Using .put_item on a Table resource that contains a float:

item = {'name': 'testing_row', 'foo': 30.40}
table.put_item(Item=item)
>> TypeError: Float types are not supported. Use Decimal types instead.

The same thing, using Decimal:

item = {'name': 'testing_row', 'foo': Decimal(30.40)}
table.put_item(Item=item)
>> Inexact: None

This last one should have gone through, no? The stack trace is this:

>       table.put_item(Item=item)

E:\Projects\Repos\tests\test_dynamodb.py:9: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
C:\Python27\lib\site-packages\boto3\resources\factory.py:518: in do_action
    response = action(self, *args, **kwargs)
C:\Python27\lib\site-packages\boto3\resources\action.py:83: in __call__
    response = getattr(parent.meta.client, operation_name)(**params)
C:\Python27\lib\site-packages\botocore\client.py:258: in _api_call
    return self._make_api_call(operation_name, kwargs)
C:\Python27\lib\site-packages\botocore\client.py:524: in _make_api_call
    api_params, operation_model, context=request_context)
C:\Python27\lib\site-packages\botocore\client.py:574: in _convert_to_request_dict
    params=api_params, model=operation_model, context=context)
C:\Python27\lib\site-packages\botocore\hooks.py:227: in emit
    return self._emit(event_name, kwargs)
C:\Python27\lib\site-packages\botocore\hooks.py:210: in _emit
    response = handler(**kwargs)
C:\Python27\lib\site-packages\boto3\dynamodb\transform.py:197: in inject_attribute_value_input
    'AttributeValue')
C:\Python27\lib\site-packages\boto3\dynamodb\transform.py:252: in transform
    model, params, transformation, target_shape)
C:\Python27\lib\site-packages\boto3\dynamodb\transform.py:259: in _transform_parameters
    model, params, transformation, target_shape)
C:\Python27\lib\site-packages\boto3\dynamodb\transform.py:274: in _transform_structure
    target_shape)
C:\Python27\lib\site-packages\boto3\dynamodb\transform.py:259: in _transform_parameters
    model, params, transformation, target_shape)
C:\Python27\lib\site-packages\boto3\dynamodb\transform.py:283: in _transform_map
    params[key] = transformation(value)
C:\Python27\lib\site-packages\boto3\dynamodb\types.py:103: in serialize
    return {dynamodb_type: serializer(value)}
C:\Python27\lib\site-packages\boto3\dynamodb\types.py:204: in _serialize_n
    number = str(DYNAMODB_CONTEXT.create_decimal(value))
C:\Python27\lib\decimal.py:3938: in create_decimal
    return d._fix(self)
C:\Python27\lib\decimal.py:1712: in _fix
    context._raise_error(Inexact)

I can make it go through if I use the string trick in the Decimal constructor:

item = {'name': 'testing_row', 'foo': Decimal('30.40')}
table.put_item(Item=item)

Having done that, using AWS's dashboard to look into the table, I can see a number type with the value of 30.4 but then I cannot assert its value in my tests:

item = table.get_item(Key={'name': 'testing_row'})['Item']['foo']
assert item == 30.4
>> False

This seems to work:

assert float(item) == 30.4

So how's this supposed to work exactly? Is it expected that I must 1) provide a string to decimal and 2) convert back to float myself for equality to work properly?

@jonathanwcrane

This comment has been minimized.

Copy link
Contributor

jonathanwcrane commented May 31, 2016

Yeah I've gotten this weird "inexact" error before. I think what I did was cast a number to decimal earlier on in the calculation, then I rounded it to two decimals points AFTER I had cast it, and it worked.

¯_(ツ)_/¯

@jonapich

This comment has been minimized.

Copy link
Author

jonapich commented May 31, 2016

@jonathanwcrane but in my case i'm not doing any calculations. Boto3 forces me to convert my floats to Decimal, which I do, and then it is unable to push it... If you take a look at the stacktrace, you'll notice boto tries to do some funky stuff with my Decimal... this sounds like a bug.

@kyleknap

This comment has been minimized.

Copy link
Member

kyleknap commented Jun 2, 2016

@jonapich
You just need to enclose the float value with quotes. So something like this:

item = {'name': 'testing_row', 'foo': Decimal('30.40')}
table.put_item(Item=item)

Passing it as a string is the typical usage for Decimal in python: https://docs.python.org/2/library/decimal.html.

Let me know if that helps. I think an example should be added for this.

@kyleknap kyleknap self-assigned this Jun 2, 2016

@jonathanwcrane

This comment has been minimized.

Copy link
Contributor

jonathanwcrane commented Jun 2, 2016

So let's say it's a value that's calculated on the fly. Do we cast it as a string, thusly:

our_value = some_other_value / yet_another_value
item = {'name': 'testing_row', 'foo': Decimal(str(our_value))}
table.put_item(Item=item)

?

@jonapich

This comment has been minimized.

Copy link
Author

jonapich commented Jun 2, 2016

@kyleknap did you look at the example I provided after the stack trace, in my original post?

I don't agree with this behavior being normal. Having to wrap these calls with home-made recursive conversion functions goes against the principle that you can usually just pass a boto resource around. It should be usable as-is.

I understand that python floats are annoying and unprecise, but boto really should handle the extra step of handling these conversions for the user. I agree that it shouldn't allow Inexact conversions, but for this it would be better to allow the user to attach his own conversion function when Inexact is raised.

@jonapich

This comment has been minimized.

Copy link
Author

jonapich commented Jun 2, 2016

@jonathanwcrane the problem with this approach is that it also has to be converted back if you pull it:

>>> val1 = Decimal(30.40)
>>> val2 = Decimal('30.40')
>>> val1 == 30.40
True
>>> val2 == 30.40
False
>>> float(val1) == 30.40
True
>>> float(val2) == 30.40
True
@csimmons0

This comment has been minimized.

Copy link

csimmons0 commented Jun 6, 2016

Hi. I was hitting the same error and came up with this solution. If you're okay with rounding, you may find it helpful.

def round_float_to_decimal(float_value):
    """
    Convert a floating point value to a decimal that DynamoDB can store,
    and allow rounding.
    """

    # Perform the conversion using a copy of the decimal context that boto3
    # uses. Doing so causes this routine to preserve as much precision as
    # boto3 will allow.
    with decimal.localcontext(boto3.dynamodb.types.DYNAMODB_CONTEXT) as \
         decimalcontext:

        # Allow rounding.
        decimalcontext.traps[decimal.Inexact] = 0
        decimalcontext.traps[decimal.Rounded] = 0
        decimal_value = decimalcontext.create_decimal_from_float(float_value)
        g_logger.debug("float: {}, decimal: {}".format(float_value,
                                                       decimal_value))

        return decimal_value
@robolivable

This comment has been minimized.

Copy link

robolivable commented Sep 30, 2016

@jonapich +1, I totally agree with this. I shouldn't have to be yelled at for using floats in my values. And if they must be used, the library should be responsible for handling the conversions, not me.

What seems off here as well, is that this isn't an issue when using the DynamoDB.Client class approach for saving items.

@bittlingmayer

This comment has been minimized.

Copy link

bittlingmayer commented Nov 17, 2016

We store a nested object, and don't want to write code that assumes a specific nesting or recursively searches for floats deep down in the tree. So for us the lazy workaround is to use json.dump and store the entire dict as string.

@Alonreznik

This comment has been minimized.

Copy link

Alonreznik commented Mar 13, 2017

I've solved that issue by creating a json.loads object hook, which can make that conversion from decimal to float.

from dateutils import parser
from boto3.dynamodb.types import TypeDeserializer, TypeSerializer

def encode_object_hook(dct):
    try:
        return parser.parse(TypeDeserializer().deserialize(dct))
    except (ValueError, AttributeError, TypeError):
        try:
            val = TypeDeserializer().deserialize(dct)
            if isinstance(val, Decimal):
                if val % 1 > 0:
                    return float(val)
                elif val < maxint:
                    return int(val)
                else:
                    return long(val)
            else:
                return val
        except:
            return dct

For "dumps" your dict into DynamoDB put_item valid value, you can use this serializer:

def decode_object_hook(dct):
    for key, val in dct.iteritems():
        if isinstance(val, float):
            dct[key] = Decimal(str(val))
        try:
            dct[key] = TypeSerializer().serialize(val)
        except:
            dct[key] = val
    return dct

def json_serial(val):
    if isinstance(val, datetime):
        serial = val.strftime('%Y-%m-%dT%H:%M:%S.%f')
        return serial
    elif isinstance(val, set):
        serial = list(val)
        return serial
    elif isinstance(val, uuid.UUID):
        serial = str(val.hex)
        return serial

def dumps(dct, *args, **kwargs):
    kwargs['object_hook'] = decode_object_hook
    return json.loads(json.dumps(dct, default=json_serial), *args, **kwargs)

item = dumps({'name': 'testing_row', 'foo': 30.40})
table.put_item(Item=item)
@agarwalvipin

This comment has been minimized.

Copy link

agarwalvipin commented Sep 23, 2017

is there any update on this issue/feature request?

@APIZone

This comment has been minimized.

Copy link

APIZone commented Oct 29, 2017

What worked for us is wrapping the floating point value into str and casting to Decimal, no loss of precision!

transaction_amount = 100.03

item = {
'subject_bank_xref': ext_reference,
'transaction_amount': Decimal(str(transaction_amount))
}

srp-synengco added a commit to Synergetic-Engineering/cloud-koala that referenced this issue Dec 12, 2017

@tarak1992

This comment has been minimized.

Copy link

tarak1992 commented Jan 10, 2019

This below implementation will solve the issue. Here each_item is the JSON Object
from decimal import Decimal
each_item_dump = json.dumps(each_item)
each_item = json.loads(each_item_dump, parse_float=Decimal)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment