Large int values as hash_key can not be queried

[alimoeeny]

I have very large int values as hash keys and moto fails to find them using query, I don't know if this is a moto or boto error, my error looks like below, not that in like 46 PK_MID is 444881095079499277 and I just inserted it and if do a scan and print it is there, and if I do a manual comparison it matches:

2) ERROR: test_postSimpleMessage (test_CSMessage.TestPostMessage)

   Traceback (most recent call last):
    /usr/local/Cellar/python/2.7.4/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/moto-0.1.4-py2.7.egg/moto/core/models.py line 47 in wrapper
      result = func(*args, **kwargs)
    test_CSMessage.py line 46 in test_postSimpleMessage
  mitem = table.get_item(hash_key=i['PK_MID'])
    /usr/local/lib/python2.7/site-packages/boto/dynamodb/table.py line 287 in get_item
  item_class)
    /usr/local/lib/python2.7/site-packages/boto/dynamodb/layer2.py line 487 in get_item
  object_hook=self.dynamizer.decode)
    /usr/local/lib/python2.7/site-packages/boto/dynamodb/layer1.py line 307 in get_item
  object_hook=object_hook)
    /usr/local/lib/python2.7/site-packages/boto/dynamodb/layer1.py line 118 in make_request
  retry_handler=self._retry_handler)
    /usr/local/lib/python2.7/site-packages/boto/connection.py line 845 in _mexe
  status = retry_handler(response, i, next_sleep)
    /usr/local/lib/python2.7/site-packages/boto/dynamodb/layer1.py line 158 in _retry_handler
  data)
   DynamoDBResponseError: DynamoDBResponseError: 400 Bad Request
   {u'__type': u'com.amazonaws.dynamodb.v20111205#ResourceNotFoundException'}

[spulec]

I'm fairly sure this is an issue with how the underlying HTTP mocking library (HTTPretty) deals with message chunking.

Moto is currently on an older bundled version of HTTPretty, but I am actively working on getting it back on the upstream version. I put in a pull request for something else yesterday to HTTPretty that should also fix this: gabrielfalcao/HTTPretty#57

[alimoeeny]

thanks,

[spulec]

This should be fixed in the newest version. Can you confirm?

[alimoeeny]

You mean this commit 755fe65 or you mean the httpretty branch?
also I get an error "error: Could not find required distribution httpretty==0.6.0a" while I am sure I have the latests httpretty, I am looking into it to see where the problem is,

[spulec]

The merging of the httpretty-upstream branch at 000006e should have fixed this.

How are you installing moto? It is actually dependent upon the HEAD of httpretty right now, which is different than the latest release. This should automatically get picked up due to https://github.com/spulec/moto/blob/master/setup.py#L26.

If nothing else, I'll bug the httpretty author to push out a new version in the next couple days so that moto can get on to a real release of it.

[alimoeeny]

Aha, thanks man, yes the problem was the dependency was not properly handled, here is the log:

Installed /usr/local/lib/python2.7/site-packages/moto-0.2.0-py2.7.egg
Processing dependencies for moto==0.2.0
Searching for httpretty==0.6.0a
Best match: httpretty 0.6.0a
Downloading https://github.com/gabrielfalcao/HTTPretty/tarball/2347df40a3a3cd00e73f0353f5ea2670ad3405c1#egg=httpretty-0.6.0a
Processing 2347df40a3a3cd00e73f0353f5ea2670ad3405c1
Writing /tmp/easy_install-fMGv50/gabrielfalcao-HTTPretty-2347df4/setup.cfg
Running gabrielfalcao-HTTPretty-2347df4/setup.py -q bdist_egg --dist-dir /tmp/easy_install-fMGv50/gabrielfalcao-HTTPretty-2347df4/egg-dist-tmp-SuXwBe
zip_safe flag not set; analyzing archive contents...
httpretty.core: module MAY be using inspect.stack
httpretty 0.6.0 is already the active version in easy-install.pth

I just removed my httpretty and downloaded the dependency and manually installed it, when ran the setup still get the same complaint "error: Could not find required distribution httpretty==0.6.0a"

[alimoeeny]

OK, I think I installed everything as they should but now I get a different error:

--------------------- >> end captured stdout << ----------------------

======================================================================
ERROR: test_postSimpleMessage (test_CSMessage.TestPostMessage)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.4/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/moto-0.2.0-py2.7.egg/moto/core/models.py", line 47, in wrapper
    result = func(*args, **kwargs)
  File "/Users/ali/Dropbox/Projects/CS/sewingmachine/tests/test_CSMessage.py", line 40, in test_postSimpleMessage
    mitem = mtable.get_item(hash_key=r['PK_MID'])
  File "/usr/local/Cellar/python/2.7.4/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/boto-2.9.2_dev-py2.7.egg/boto/dynamodb/table.py", line 287, in get_item
    item_class)
  File "/usr/local/Cellar/python/2.7.4/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/boto-2.9.2_dev-py2.7.egg/boto/dynamodb/layer2.py", line 487, in get_item
    object_hook=self.dynamizer.decode)
  File "/usr/local/Cellar/python/2.7.4/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/boto-2.9.2_dev-py2.7.egg/boto/dynamodb/layer1.py", line 307, in get_item
    object_hook=object_hook)
  File "/usr/local/Cellar/python/2.7.4/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/boto-2.9.2_dev-py2.7.egg/boto/dynamodb/layer1.py", line 118, in make_request
    retry_handler=self._retry_handler)
  File "/usr/local/Cellar/python/2.7.4/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/boto-2.9.2_dev-py2.7.egg/boto/connection.py", line 843, in _mexe
    response = connection.getresponse()
  File "/usr/local/Cellar/python/2.7.4/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1043, in getresponse
    response = self.response_class(*args, **kwds)
  File "/usr/local/Cellar/python/2.7.4/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/boto-2.9.2_dev-py2.7.egg/boto/connection.py", line 389, in __init__
    httplib.HTTPResponse.__init__(self, *args, **kwargs)
  File "/usr/local/Cellar/python/2.7.4/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 346, in __init__
    self.fp = sock.makefile('rb', 0)
  File "/usr/local/Cellar/python/2.7.4/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/httpretty-0.6.0-py2.7.egg/httpretty/core.py", line 203, in makefile
    self._entry.fill_filekind(self.fd)
  File "/usr/local/Cellar/python/2.7.4/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/httpretty-0.6.0-py2.7.egg/httpretty/core.py", line 436, in fill_filekind
    status, headers, self.body = self.callable_body(self.request,self.info.full_url(),headers)
  File "/usr/local/Cellar/python/2.7.4/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/moto-0.2.0-py2.7.egg/moto/core/responses.py", line 32, in dispatch
    return self.call_action()
  File "/usr/local/Cellar/python/2.7.4/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/moto-0.2.0-py2.7.egg/moto/dynamodb/responses.py", line 59, in call_action
    response = getattr(self, endpoint)()
  File "/usr/local/Cellar/python/2.7.4/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/moto-0.2.0-py2.7.egg/moto/dynamodb/responses.py", line 193, in get_item
    item_dict = item.describe_attrs(attrs_to_get)
AttributeError: 'dict' object has no attribute 'describe_attrs'

[spulec]

I can't seem to reproduce this. Do you think you could write up a small test case that breaks?

[alimoeeny]

sorry for the late reply, I striped down one of my failing tests that I think runs into the problem we are discussing:

# -*- conding: utf-8 -*-

import boto
from moto import mock_dynamodb
from moto.dynamodb import dynamodb_backend
import unittest 
import time
import datetime




class TestPostMessage(unittest.TestCase):
  @mock_dynamodb
  def test_postSimpleMessage(self):
    sessionid = 1234567890
    uid = 9876543210
    segment_id = 3241526475
    messagebody = 'just testing you'
    replyto = 1324254568
    tag = 'testing' 

    dynamodb_backend.create_table('mockCSSessions', hash_key_attr="PK_SID", hash_key_type="N")
    dynamodb_backend.create_table('mockCSMessages', hash_key_attr="PK_MID", hash_key_type="N", range_key_attr="RK_SEGID", range_key_type="N")
    conn = boto.connect_dynamodb() 
    table = conn.get_table('mockCSSessions')
    item = table.new_item(hash_key=sessionid, attrs={'userid':uid, 'segmentid':segment_id})
    item.put()

    env = {}
    env['dbconnection'] = conn 
    env['dbsfx'] = 'mock' 
    env['remote'] = {'remote_ip':'1.1.1.1', 'remote_agent':'Mozilla FireDog'}
    r = {}
    if (uid==0):
      r['error'] += 'you must be logged in to post a message'
    else:
      if (segment_id==0):
        r['error'] += 'you must be checked into the segment to post a message'
      else:
        try:
          mTable = env['dbconnection'].get_table(env['dbsfx'] + 'CSMessages')
        except Exception as e:
          if (e.error_code == 'ResourceNotFoundException'):
            r['message'] += 'Table not found segment not setup correctly '
            raise e
        message_id = 1234567890987
        item_data = {}
        item_data['createdby'] = uid
        item_data['timestamp'] = time.mktime(datetime.datetime.now().timetuple())
        if(messagebody):
          item_data['body'] = messagebody
        if(replyto):
          item_data['replyto'] = replyto
        if(tag):
          item_data['tag'] = tag
        attachmentURLs = 'test.png'
        if(attachmentURLs):
          item_data['attachmentURLs'] = attachmentURLs
        rr = mTable.new_item(hash_key=message_id, range_key=segment_id, attrs=item_data)
        rr.put()
        r['messageid'] = message_id

        meTable = env['dbconnection'].get_table(env['dbsfx'] + 'CSMessagesSegment')
        meitem = meTable.new_item(hash_key=segment_id, range_key=message_id)
        meitem.put()


    self.assertTrue(int(r['messageid'])>0)
    mtable = conn.get_table('mockCSMessages')
    mitem = mtable.get_item(hash_key=r['PK_MID'])
    self.assertEqual(mitem['messageid'], r['messageid'])

[spulec]

I think the issue is that you're calling mtable.get_item(hash_key=r['PK_MID']), but that table has a range key also so you can't do a get without also passing in a range_key.

I've fixed moto to handle this case properly instead of blowing up.

[alimoeeny]

Sorry, that was embarrassing :|
Thanks,

[spulec]

No worries. Your mistake would have been much easier for you to spot if Moto had been giving the proper error messaging. I'm happy that this was reported and fixed so that the next user to come along will have an easier time.