New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Query all giving an error #9

Closed
eprparadocs opened this Issue Mar 11, 2014 · 7 comments

Comments

Projects
None yet
2 participants
@eprparadocs

eprparadocs commented Mar 11, 2014

Here is the code I execute:

def c_composite( analyzer_name, data_type, data_idx, creation_time ):
"""
Custom composite handler for range key of AppDataModel
"""
return( '%s:%s:%s:%s' % (analyzer_name,data_type,str(data_idx),str(creation_time)) )

class AppDataModel(Model):
metadata = {
'global_indexes': [
GlobalIndex.all( 'gcc-index','country_code' ),
GlobalIndex.all( 'gat-index','app_type' )
]
}

country_code = Field( data_type=STRING )
app_type = Field( data_type=STRING )
app_id = Field( data_type=STRING )
app_version = Field( data_type=STRING )
composite_appid = Composite( 'country_code', 'app_type', 'app_id', 'app_version', hash_key=True )
analyzer_name = Field( data_type=STRING, index='lan-index' )
data_type = Field( data_type=STRING, index='ldt-index' )
data_index = Field( data_type=NUMBER )
creation_datetime = Field( data_type=datetime, index='lc-index' )
r_key = Composite( 'analyzer_name', 'data_type', 'data_index', 'creation_datetime', merge=c_composite, range_key=True)

def init(....)

Later on I do this:

dyn = DynamoDBConnection( **args )
eng = Engine( dynamo=dyn, namespace=['appdata'] )

test = AppDataModel( country_code='united states',app_type='android',app_id='123', app_version='1.0')
test.eng = eng

i = test.eng.query(AppDataModel).all()
for r in i:
    print r

I get this traceback:

Traceback (most recent call last):
...
File .../flywheel/query.py line 79 in all

File ../flywheel/query.py line 48 in gen

File ../flywheel/fields/conditions.py line 54

ValueError: Bad query arguments. You must provide a hash key and may optionally constrain on exactly one range key

In fact when I check composite_appid it is set to u'USA:android:123:1.0' which is the composited hash key. Am I missing something?

@eprparadocs

This comment has been minimized.

eprparadocs commented Mar 11, 2014

I also tried this snippet:

dyn = DynamoDBConnection( **args )
eng = Engine( dynamo=dyn, namespace=['appdata'] )
i = eng.query(AppDataModel).filter(country_code='united states',app_type='android',app_id='123', app_version='1.0').all()
for r in i:
print r

I get the following exception:

File "marsdata.py", line 246, in
i = eng.query(AppDataModel).filter(country_code='united states',app_type='android',app_id='123', app_version='1.0').all()
File "/home/dev/Projects/MARS2/PYTHON/lib/python2.7/site-packages/flywheel/query.py", line 79, in all
attributes=attributes))
File "/home/dev/Projects/MARS2/PYTHON/lib/python2.7/site-packages/flywheel/query.py", line 48, in gen
kwargs = self.condition.query_kwargs(self.model)
File "/home/dev/Projects/MARS2/PYTHON/lib/python2.7/site-packages/flywheel/fields/conditions.py", line 51, in query_kwargs
self.fields.keys())
File "/home/dev/Projects/MARS2/PYTHON/lib/python2.7/site-packages/flywheel/model_meta.py", line 236, in get_ordering_from_fields
rng_needed = order.range_key.can_resolve(rng_fields)
AttributeError: 'NoneType' object has no attribute 'can_resolve'

I'm still not sure what I'm doing wrong since I'm following the examples in the docs with the exception on not doing the register() and create_schema() since I'm not creating the schema (it ought to exist in the system already)

@eprparadocs eprparadocs reopened this Mar 11, 2014

@eprparadocs

This comment has been minimized.

eprparadocs commented Mar 11, 2014

Figured it out...cockpit screw up on my part....:(

@stevearc

This comment has been minimized.

Owner

stevearc commented Mar 11, 2014

Really? It certainly looks like, if nothing else, a terrible error message. And I can't be certain, but I went through the stacktrace and that NoneType error is something that should never happen. What was the fix?

@eprparadocs

This comment has been minimized.

eprparadocs commented Mar 11, 2014

On 03/11/2014 03:47 PM, Steven Arcangeli wrote:

Really? It certainly looks like, if nothing else, a terrible error
message. And I can't be certain, but I went through the stacktrace and
that NoneType error is something that should never happen. What was
the fix?


Reply to this email directly or view it on GitHub
#9 (comment).

It turns out it wasn't fixed by me. I re-ran the test and it still
happens. Here is the code I execute:

if name == "main":

appinfo = AppData( host='localhost',port=8000,debug=1, country_code='united states',app_type='android',app_id='123', app_version='1.0')

 # Get all the stuff related to just a particular app.
 i = appinfo.eng.query(AppDataModel).filter( country_code='united 

states',app_type='android',app_id='123', app_version='1.0' ).all()
for j in i:
print j.country_code, j.app_type, j.app_id, j.app_version
print '\t', j.analyzer_name, j.data_type, j.data_index
print '\t', j.creation_datetime
print '\t\t', j.data

Here is the AppDataModel class I use:

from iso3166 import countries
from flywheel import Model, Composite, Field, Engine, STRING, NUMBER,
GlobalIndex
from boto.dynamodb2.layer1 import DynamoDBConnection

def c_composite( analyzer_name, data_type, data_idx, creation_time ):
"""
Custom composite handler for range key of AppDataModel
"""
return( '%s:%s:%s:%s' %
(analyzer_name,data_type,str(data_idx),str(creation_time)) )

class AppDataModel(Model):
"""

 The application data storage model. The indexed data includes:

     _country_code - ISO 3166-1 alpha-3, 3 character country code
     _app_type - application type, i.e. android, ios, etc. Always 

lower case
_app_id - unique application identifier, usually a hash of some
kind
_app_version - version of application
_analyzer_name - name of the analyzer that is responsible for
some specific generated data
_data_type - type of recorded data, i.e. log, call, output
_creation_datetime - the ISO 8601 formatted date and time of
when the data was created

 The following indices are available:

     Global index gcc-index allowing searching by _country_code
     Global index gat-index allowing searching by _app_type
     Global index lan-index allowing searching by _analyzer_name

     Primary key:
         Hash key composed of _country_code, _app_type, _app_id, 

_app_version
Range key composed of _analyzer_name

     Local index ldt-index allow searching within an application by 

_data_type
Local index lc-index allow searching within an application by
_creation_datetime

 """
 __metadata__ = {
    'global_indexes': [
         GlobalIndex.all( 'gcc-index','country_code' ),
         GlobalIndex.all( 'gat-index','app_type' )
     ]
 }

 country_code = Field( data_type=STRING )
 app_type = Field( data_type=STRING )
 app_id = Field( data_type=STRING )
 app_version = Field( data_type=STRING )
 composite_appid = Composite( 'country_code', 'app_type', 'app_id', 

'app_version', hash_key=True )
analyzer_name = Field( data_type=STRING, index='lan-index' )
data_type = Field( data_type=STRING, index='ldt-index' )
data_index = Field( data_type=NUMBER )
creation_datetime = Field( data_type=datetime, index='lc-index' )
r_key = Composite( 'analyzer_name', 'data_type', 'data_index',
'creation_datetime', merge=c_composite, range_key=True)

 def __init__( self, country_code, app_type, app_id, app_version, 

analyzer_name=None, data_type=None, data_idx=0, creation_datetime=None ):
# Set the obvious things...these things must be present!!!

     # The input country code is automatically converted to the 

ISO3166 3 character code! If the name isn't known
# a KeyError will be thrown. We accept ISO3166 2 character, 3
character and numeric codes. We also accept the
# conventional English names for the countries.
self.country_code = countries.get(country_code).alpha3

     # Make sure the app_type is something we can handle.
     c = app_type.lower()
     if c not in application_types:
         raise ValueError( 'app_type is not one of %s' % 

(','.join(application_types)) )
self.app_type = c

     # Store the easy stuff - app id and version
     self.app_id = app_id
     self.app_version = app_version

     # These are optional but must be set before any writing out to 

DynamoDB
self.analyzer_name = analyzer_name.lower() if analyzer_name
else None
self.data_type = data_type.lower() if data_type else None
self.data_index = data_idx if data_type and data_type.lower()
in ['data','log'] else 0

     # If the create_datetime argument is supplied it must be a 

datetime instance! It must
# also be in UTC time.
if creation_datetime:
# Make sure we have been given the right kind of
argument..a datetime instance.
if isinstance(creation_datetime,datetime):
self.creation_datetime = creation_datetime
else:
# Type error!!!
raise TypeError('creation_datetime argument must be an
instance of datetime')
else:
# Set in the default creation date/time of now. It must be
utc!!!!
self.creation_datetime = datetime.utcnow()

And here is the AppData class:

def AppDataEngine(kwargs):
# Connect to DynamoDB
dyn = DynamoDBConnection( **kwargs )

 # Connect to the server and create the table schema for the 

'appdata' table.
eng = Engine( dynamo=dyn, namespace=['appdata'] )
eng.register( AppDataModel )
eng.create_schema()

 # Return the engine.
 return( eng )

class AppData(object):
"""
This class is the interface between the data storage system of MARS
2.0 and the applications
that run inside it. This is the base class that ought to be used
for every application specific
instance.

 The correct way to think of the data related to a particular 

application (identified by the appID) is
to envision a journal file system. We record with each "record"
enough informnation to recreate the
stream of data about the application.

 Ultimately the data is expected to be hosted in a kev-value store 

though the initial implementation will
assume an underlying ACID database.
"""

 def __init__( self, **kwargs ):
     """
     This method, which initializes the class does three things:

     1) Pulls in the data related to 'appID' into the class for use 

(beware there could be
other apps using the same data simultaneously.);
2) Setups up the "faked out" logging facility allowing us to
capture all logging from the
application and write it to the appID data stream; and
3) Records the information about who we are and how we got here.

     At the end of the application all this information will be 

written out to the data storage
system. This method shouldn't need to be changed. You will need
to fill in __initData() though.

     country_code - what user is making the request.
     app_type - type of app (android,ios,blackberry,windows)
     app_id - what application is being "touched".
     analyzer_name - the name of the analyzer calling

     The following can be supplied and are used to connect to the 

DynamoDB server:

     aws_secret_access_key - AWS DynamoDB secret
     aws_access_key_id - AWS DynamoDB access
     debug - whether(1) or not(0) to do debug logging
     is_secure - whether(True) or not(False) to do secure communications

     """

     # Setup the connection to the DynamoDB server...
     args =  {'aws_secret_access_key':kwargs.pop( 'secret', 

AWS_SECRET_ACCESS_KEY ),
'aws_access_key_id':kwargs.pop( 'access',
AWS_ACCESS_KEY ), 'debug':kwargs.pop('debug',0),
'is_secure':kwargs.pop( 'is_secure', False )
}

     # If host/port are present then we are talking to a local 

dynamodb server.
if 'host' in kwargs:
args['host'] = kwargs.pop('host','')
if 'port' in kwargs:
args['port'] = int(kwargs.pop('port',''))

     # Create the engine.
     self.eng = AppDataEngine(args)
     #self.dyn = DynamoDBConnection( **args )

     # Connect to the server and create the table schema for the 

'appdata' table.
#self.eng = Engine( dynamo=self.dyn, namespace=['appdata'] )
#self.eng.register( AppDataModel )
#self.eng.create_schema()

     # Save the application information - this are all required! If 

not present it will
# thrown an exception back to the caller or in error an
exception will be raised.
self.cc = countries.get( kwargs.pop('country_code') ).alpha3
c = kwargs.pop('app_type').lower()
if c not in application_types:
raise ValueError( 'app_type is not one of %s' %
(','.join(application_types)) )
self.at = c
self.aid = kwargs.pop('app_id')
self.av = kwargs.pop('app_version')
an = kwargs.pop('analyzer_name',None)
self.an = an.lower() if an else None

     # Set up the data and log counters, just in case the 

application has to write lots and lots of data.
self.dcounter = self.lcounter = 0

     # Set the creation date and time - we want it common across all 

the calls regardless of how log
# the processing takes. This is how we "tie" things together.
self.dt = datetime.utcnow()

Here is the complete stack trace:

File "marsdata-test3.py", line 18, in
i = appinfo.eng.query(AppDataModel).filter( country_code='united
states',app_type='android',app_id='123', app_version='1.0' ).all()
File
"/home/dev/Projects/MARS2/PYTHON/lib/python2.7/site-packages/flywheel/query.py",
line 79, in all
attributes=attributes))
File
"/home/dev/Projects/MARS2/PYTHON/lib/python2.7/site-packages/flywheel/query.py",
line 48, in gen
kwargs = self.condition.query_kwargs(self.model)
File
"/home/dev/Projects/MARS2/PYTHON/lib/python2.7/site-packages/flywheel/fields/conditions.py",
line 51, in query_kwargs
self.fields.keys())
File
"/home/dev/Projects/MARS2/PYTHON/lib/python2.7/site-packages/flywheel/model_meta.py",
line 236, in get_ordering_from_fields
rng_needed = order.range_key.can_resolve(rng_fields)
AttributeError: 'NoneType' object has no attribute 'can_resolve'

@eprparadocs

This comment has been minimized.

eprparadocs commented Mar 11, 2014

If you want a tar file with the code that causes the problem let me know and I will email it to you.

@stevearc

This comment has been minimized.

Owner

stevearc commented Mar 11, 2014

Found, fixed, tagged, pushed, and uploaded. New version 0.1.3 should fix this issue.

The problem was in handling queries where there existed global indexes with no range key. Thanks for the issue!

@eprparadocs

This comment has been minimized.

eprparadocs commented Mar 11, 2014

Glad I could help out!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment