Skip to content
This repository

dynamoDB, converting a range key to a float() or int() means the dynamoDB item can't be deleted. #873

Open
elee-nst opened this Issue · 3 comments

2 participants

Eric Lee Mitch Garnaat
Eric Lee

Thanks for this nice library!

While using this I found an issue when trying to delete dynamo key/value pairs from some tables that we had created.

The table has a composite key, with a string hash key and a numeric range key.

The numeric range keys that were used were very long, similar to 1234567890123.12345678901234567890

What I found was that although I could successfully query for keys that were within a range I could NOT delete individual keys. The issue was that when they are read into python a numeric field ('N') is converted to a float or an int in convert_nums() in types.py

def convert_num(s):
if '.' in s:
n = float(s)
else:
n = int(s)
return n

This dropped some of the precision in the numeric field, so later when I tried to delete that composite key it did not exactly match the key in dynamoDB and it could not delete the item.

From a co-workers suggestion I used the python decimal library to allow it to store higher precision numbers exactly, which seems to be working well.

The changes that I did were:
In types.py:
** add a reference to "from decimal import Decimal"
** add type Decimal to is_num()
def is_num(n):
types = (int, long, float, bool, Decimal)
return isinstance(n, types) or n in types

** comment out convert_nums()

In layer2.py:
**
In "def item_object_hook(dct):"
if 'N' in dct:
#return convert_num(dct['N'])
return Decimal(dct['N'])

if 'NS' in dct:
    #return set(map(convert_num, dct['NS']))
    return set(map(Decimal, dct['NS']))

This should store all of the dynamoDB numeric data as decimals in python, which allows access to the full precision of the dynamoDB values if desired.

I am not sure of the checkin policy/review policy for boto, is this something that is generally useful and should be added to master?

Regards
Eric Lee

Mitch Garnaat
Owner

It does sound like it's generally useful. I'm not sure what the impact would be for existing users and data. I'll investigate that. If you want to package the changes you describe as a pull request, that would be great. If not, I can work from your description.

Eric Lee

I am new to doing a pull request but have given it a try, following the github directions. This is pull request #890, I believe.

What I did:
Created a fork.
Copied changes from where I had originally done the edits into this new working copy.
Did testing and see that numeric keys from dynamoDb are converted to decimal objects in python.
Decimal('1343775601000.35412459980000001518')
Checked in to my fork.
Created the pull request.

I think I copied all of the bits of code correctly, and left out the remnants of my earlier experiment. Let me know if there are any issues!

Mitch Garnaat
Owner
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.