Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ability to disable Decimal usage for DynamoDB number type. #369

Open
jonapich opened this issue Nov 16, 2015 · 58 comments
Open

Add ability to disable Decimal usage for DynamoDB number type. #369

jonapich opened this issue Nov 16, 2015 · 58 comments

Comments

@jonapich
Copy link

@jonapich jonapich commented Nov 16, 2015

Hi,

Is there any strong reason why using a DynamoDB's Table resource will convert the number type "N" to a Decimal() object?

Shouldn't it try looking up the right python type, such as int or float or long?

I am trying to unpack a record's data (a mapping) into a function call, specifically: next_execution = now() + datetime.timedelta(**dynamo_record['frequency']) but datetime.timedelta will not accept the Decimal object into its arguments, although it does accept long and float.

TypeError: unsupported type for timedelta days component: Decimal

@jamesls

This comment has been minimized.

Copy link
Member

@jamesls jamesls commented Nov 16, 2015

This was one of the lessons learned from boto. If I remember correctly, there were issues with round tripping float values and that the built in float() type could not handle the 38 digits of precision supported in dynamodb's numeric types. This would result in not being able to delete items in the table:

>>> d = Decimal('1234567890123.12345678901234567890')
>>> d
Decimal('1234567890123.12345678901234567890')
>>> float(d)
1234567890123.1235

More background info:

boto/boto#873

PR:
boto/boto#1183

Perhaps we could support using ints() if there's no floating point in the number. Would need to investigate what the impact of that would be. Would that help in your scenario?

@jonapich

This comment has been minimized.

Copy link
Author

@jonapich jonapich commented Nov 16, 2015

After a trip to python's doc on floating point limitations, I can see where this came from. It's a shame that the standard library doesn't support Decimal() in place of a float.

To answer your question specifically, it will not help my scenario. The timedelta supports floats, it's perfectly valid to do a timedelta(hours=1.5). I think it would only add to the confusion if boto3 would change between int and Decimal on a per-record basis as its default behavior.

Perhaps a small option somewhere to tell boto we don't really care about the added precision, so it can return floats and ints across the board?

@garnaat

This comment has been minimized.

Copy link
Member

@garnaat garnaat commented Nov 16, 2015

FWIW, I use this little function to recurse into Python objects returned by the boto3 DynamoDB resource layer and convert any Decimal values to int or float. It is by no means foolproof and doesn't in any way solve the problem of lack of precision in Python's float type but it solves my problem which is mainly to turn the data into something that can be returned to API Gateway via a Python Lambda function.

def replace_decimals(obj):
    if isinstance(obj, list):
        for i in xrange(len(obj)):
            obj[i] = replace_decimals(obj[i])
        return obj
    elif isinstance(obj, dict):
        for k in obj.iterkeys():
            obj[k] = replace_decimals(obj[k])
        return obj
    elif isinstance(obj, decimal.Decimal):
        if obj % 1 == 0:
            return int(obj)
        else:
            return float(obj)
    else:
        return obj
@astewart-twist

This comment has been minimized.

Copy link

@astewart-twist astewart-twist commented Nov 18, 2015

Thanks for the code @garnaat, that certainly works.

This issue is a big PITA.

@jamesls

This comment has been minimized.

Copy link
Member

@jamesls jamesls commented Nov 18, 2015

Would adding some sort of use_decimal=False option to the config object when creating clients/resources be helpful?

@jonapich

This comment has been minimized.

Copy link
Author

@jonapich jonapich commented Nov 18, 2015

Of course that would be useful :)

@jamesls

This comment has been minimized.

Copy link
Member

@jamesls jamesls commented Nov 18, 2015

Ok, let's mark this as a feature request. I'll update the title.

@jamesls jamesls changed the title DynamoDB number types Add ability to disable Decimal usage for DynamoDB number type. Nov 19, 2015
@jamesls jamesls added the dynamodb label Nov 19, 2015
@niallrobinson

This comment has been minimized.

Copy link

@niallrobinson niallrobinson commented Feb 10, 2016

👍 for this feature

@jonapich

This comment has been minimized.

Copy link
Author

@jonapich jonapich commented Mar 11, 2016

This particular problem likes to creep into my code in the most unusual places. Converting from Decimal to int/float is a thing, but it seems boto won't take my python floats anymore (did it take them before? i'm not sure) so I created the following function to prepare all of my data before sending it to dynamodb (kinda like @garnaat's method, but the other way around):

def _sanitize(data):
    """ Sanitizes an object so it can be updated to dynamodb (recursive) """
    if not data and isinstance(data, (basestring, Set)):
        new_data = None  # empty strings/sets are forbidden by dynamodb
    elif isinstance(data, (basestring, bool)):
        new_data = data  # important to handle these one before sequence and int!
    elif isinstance(data, Mapping):
        new_data = {key: _sanitize(data[key]) for key in data}
    elif isinstance(data, Sequence):
        new_data = [_sanitize(item) for item in data]
    elif isinstance(data, Set):
        new_data = {_sanitize(item) for item in data}
    elif isinstance(data, (float, int, long, complex)):
        new_data = Decimal(data)
    else:
        new_data = data
    return new_data
@mr337

This comment has been minimized.

Copy link

@mr337 mr337 commented Apr 8, 2016

+1 for this

@jonathanwcrane

This comment has been minimized.

Copy link
Contributor

@jonathanwcrane jonathanwcrane commented May 24, 2016

+1 for this feature request, I've now run into a similar problem to the one @jonapich experienced.

@jonapich

This comment has been minimized.

Copy link
Author

@jonapich jonapich commented May 31, 2016

@sridharrajagopal

This comment has been minimized.

Copy link

@sridharrajagopal sridharrajagopal commented Jun 1, 2016

+1 for this feature request

@ustroetz

This comment has been minimized.

Copy link

@ustroetz ustroetz commented Jun 15, 2016

+1

1 similar comment
@josepvalls

This comment has been minimized.

Copy link

@josepvalls josepvalls commented Aug 2, 2016

+1

@ghost

This comment has been minimized.

Copy link

@ghost ghost commented Oct 3, 2016

any update on this ?

@jong-eatsa

This comment has been minimized.

Copy link

@jong-eatsa jong-eatsa commented Nov 11, 2016

+1

7 similar comments
@BorsosWyatt

This comment has been minimized.

Copy link

@BorsosWyatt BorsosWyatt commented Nov 23, 2016

+1

@johnjjung

This comment has been minimized.

Copy link

@johnjjung johnjjung commented Dec 14, 2016

+1

@blieber

This comment has been minimized.

Copy link

@blieber blieber commented Jan 23, 2017

+1

@itssiva

This comment has been minimized.

Copy link

@itssiva itssiva commented Feb 8, 2017

+1

@pmranade

This comment has been minimized.

Copy link

@pmranade pmranade commented Feb 10, 2017

+1

@mwada

This comment has been minimized.

Copy link

@mwada mwada commented Feb 13, 2017

+1

@cdmbr

This comment has been minimized.

Copy link

@cdmbr cdmbr commented Feb 13, 2017

+1

@numberoverzero

This comment has been minimized.

Copy link

@numberoverzero numberoverzero commented Sep 18, 2017

The code below is for saving to DynamoDB; use @flomotlik's code in #369 (comment) to load floats from DynamoDB.

To allow rounding and inexact values and still prevent over/underflow and clamping, I'd recommend using a decimal.Context such as the one in boto3/dynamodb/types.py but drop the decimal.Inexact and decimal.Clamped traps. I'd also use numeric for the type check in the sanitizer instead of just checking for Decimal or float. The following serializer should be a bit more robust:

from collections.abc import Iterable, Mapping, ByteString, Set
import numbers
import decimal

context = decimal.Context(
    Emin=-128, Emax=126, rounding=None, prec=38,
    traps=[decimal.Clamped, decimal.Overflow, decimal.Underflow]
)


def dump_to_dynamodb(item):
    # don't catch str/bytes with Iterable check below;
    # don't catch bool with numbers.Number
    if isinstance(item, (str, ByteString, bool)):
        return item

    # ignore inexact, rounding errors
    if isinstance(item, numbers.Number):
        return context.create_decimal(item)
    
    # mappings are also Iterable
    elif isinstance(item, Mapping):
        return {
            key: dump_to_dynamodb(value)
            for key, value in item.values()
        }

    # boto3's dynamodb.TypeSerializer checks isinstance(o, Set)
    # so we can't handle this as a list
    elif isinstance(item, Set):
        return set(map(dump_to_dynamodb, item))
    
    # may not be a literal instance of list
    elif isinstance(item, Iterable):
        return list(map(dump_to_dynamodb, item))
    
    # datetime, custom object, None
    return item

<shameless plug> I wrote bloop to be a simpler interface to DynamoDB. It's overkill if float handling is the only thing you want to solve. It's good if you want more ergonomic systems for consuming streams, writing conditions and managing optimistic concurrency, sharing tables and simpler query projections. There is a pattern for a Float type which links to this issue. </shameless plug>

@kashoory

This comment has been minimized.

Copy link

@kashoory kashoory commented Sep 21, 2017

+1

@wangpin34

This comment has been minimized.

Copy link

@wangpin34 wangpin34 commented Sep 26, 2017

Thanks for all above post and code you guys provided and my problem is solved. But I still want to ask why don't we have a setting parameter to disable / enable decimal, use float / int instead?

@gibbymt

This comment has been minimized.

Copy link

@gibbymt gibbymt commented Nov 16, 2017

+1

14 similar comments
@valeriyaxref

This comment has been minimized.

Copy link

@valeriyaxref valeriyaxref commented Nov 29, 2017

+1

@gschlabitz

This comment has been minimized.

Copy link

@gschlabitz gschlabitz commented Mar 13, 2018

+1

@sb89

This comment has been minimized.

Copy link

@sb89 sb89 commented Mar 16, 2018

+1

@maayanlahav

This comment has been minimized.

Copy link

@maayanlahav maayanlahav commented Mar 21, 2018

+1

@smthfrmn

This comment has been minimized.

Copy link

@smthfrmn smthfrmn commented Mar 23, 2018

+1

@shilpad

This comment has been minimized.

Copy link

@shilpad shilpad commented Apr 5, 2018

+1

@c--m

This comment has been minimized.

Copy link

@c--m c--m commented Apr 9, 2018

+1

@tomhillable

This comment has been minimized.

Copy link

@tomhillable tomhillable commented Apr 30, 2018

+1

@gibbymt

This comment has been minimized.

Copy link

@gibbymt gibbymt commented Apr 30, 2018

+1

@lerebel103

This comment has been minimized.

Copy link

@lerebel103 lerebel103 commented May 3, 2018

+1

@weiwongfaye

This comment has been minimized.

Copy link

@weiwongfaye weiwongfaye commented Jun 1, 2018

+1

@nikhilym

This comment has been minimized.

Copy link

@nikhilym nikhilym commented Jun 15, 2018

+1

@rogergzou

This comment has been minimized.

Copy link

@rogergzou rogergzou commented Jul 11, 2018

+1

@nvsit

This comment has been minimized.

Copy link

@nvsit nvsit commented Jul 20, 2018

+1

@BigChief45

This comment has been minimized.

Copy link

@BigChief45 BigChief45 commented Jul 23, 2018

@jamesls Any updates on this issue?

@linds14sr20det

This comment has been minimized.

Copy link

@linds14sr20det linds14sr20det commented Sep 4, 2018

+1

@himanshu219

This comment has been minimized.

Copy link

@himanshu219 himanshu219 commented Nov 21, 2018

+1

@himanshu219

This comment has been minimized.

Copy link

@himanshu219 himanshu219 commented Nov 21, 2018

any updates ?

@gileri

This comment has been minimized.

Copy link

@gileri gileri commented Dec 19, 2018

I adapted @flomotlik 's code to support NumberSets (dicts) when reading DynamoDB items :

# Helper class to Decimals in an arbitrary object
def replace_decimals(obj):
    if isinstance(obj, list):
        for i in range(len(obj)):
            obj[i] = replace_decimals(obj[i])
        return obj
    elif isinstance(obj, dict):
        for k, v in obj.items():
            obj[k] = replace_decimals(v)
        return obj
    elif isinstance(obj, set):
        return set(replace_decimals(i) for i in obj)
    elif isinstance(obj, decimal.Decimal):
        if obj % 1 == 0:
            return int(obj)
        else:
            return float(obj)
    else:
        return obj
@oitan

This comment has been minimized.

Copy link

@oitan oitan commented Feb 19, 2019

+1 on this feature request too

@miroslawmajka

This comment has been minimized.

Copy link

@miroslawmajka miroslawmajka commented Feb 27, 2019

Why is this still open in 2019?
+1

@caiounderscore

This comment has been minimized.

Copy link

@caiounderscore caiounderscore commented May 15, 2019

+1

@nft2

This comment has been minimized.

Copy link

@nft2 nft2 commented May 31, 2019

If the main issue you guys are having is JSON serializing the Decimal values, simplejson can handle that. That's what I use in my Lambda functions.

@rbk

This comment has been minimized.

Copy link

@rbk rbk commented Sep 14, 2019

If the main issue you guys are having is JSON serializing the Decimal values, simplejson can handle that. That's what I use in my Lambda functions.

Thanks, nft2! The simplejson package works great.

I use:
import simplejson as json

Instead of:
import json

@vfilimonov

This comment has been minimized.

Copy link

@vfilimonov vfilimonov commented Jan 12, 2020

Hello @jamesls , @garnaat.

It's been 4 years+ since the opening of issue. What is the current take of the boto3 team on this? Are there plans to add support of native python int/float for DynamoDB?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.