JSON decode issue when querying database

[slzatz]

First of all, thanks for creating flywheel. Very helpful.

I do have the following issue:

Version is 0.44 running on python 3.43 under Windows 7
The db structure is:

class scrobble(Model):
    artist = Field(hash_key=True)
    ts = Field(data_type=datetime, range_key=True)

Works fine when database is queried through boto or when exploring through the DynamoDB Web interface. I believe the issue is that for some reason the timestamps have a dozen or more digits to the right of the decimal point when I view them in the DynamoDB.

Whether that's it or not, all queries are throwing an Exception as follows:

>>> z = engine(scrobble).filter(artist="Lucinda Williams")
>>> z.first()
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
  File "C:\Python34\lib\site-packages\flywheel\query.py", line 155, in first
attributes=attributes, filter_or=filter_or):
  File "C:\Python34\lib\site-packages\flywheel\query.py", line 80, in gen
yield self.model.ddb_load_(self.engine, result)
  File "C:\Python34\lib\site-packages\flywheel\models.py", line 458, in ddb_load_
obj.set_ddb_val_(key, val)
  File "C:\Python34\lib\site-packages\flywheel\models.py", line 450, in set_ddb_val_
setattr(self, key, Field.ddb_load_overflow(val))
  File "C:\Python34\lib\site-packages\flywheel\fields\__init__.py", line 265, in ddb_load_overflow
return json.loads(val)
  File "C:\Python34\lib\json\__init__.py", line 318, in loads
return _default_decoder.decode(s)
  File "C:\Python34\lib\json\decoder.py", line 343, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Python34\lib\json\decoder.py", line 361, in raw_decode
    raise ValueError(errmsg("Expecting value", s, err.value)) from None
ValueError: Expecting value: line 1 column 1 (char 0)

[slzatz]

Note that I truncated all the timestamps to integers and am continuing to see the same type of error in C:\Python34\lib\json\decoder.py

[stevearc]

So what appears to be happening is the model is attempting to load values that are not specified in the schema because there are some additional fields present in the data. It will check to see if that field is a int/float/set, and if not, it will do a json.loads. When you look at your data, are there any fields present other than artist and ts?

[slzatz]

There are other fields like title and album. I have just added them to the model and the query does work. However, since it's a NoSQL database and records could have additional fields, I am not sure why the json.loads fails but, in any event, adding the fields does enable the query to run. Thank you.

[stevearc]

The problem is that when I first wrote the library I tried to be too clever. In addition to saving the declared fields, Flywheel will also save any other properties on the model that don't begin or end with _. This means that when it loads the dynamo data, it will attempt to deserialize any undeclared fields. Which works fine if you only ever added data from the python models, but can cause problems in your case.

If you look over at the 0.5 branch, I'm removing that behavior for the eventual 0.5.0 release. It's too confusing and not very useful.

[slzatz]

Thanks for the explanation. I'll look at the 0.5 branch. Again, I want to thank you for your work -- using a syntax similar to sqlalchemy's made using flywheel very easy. I'll close this issue.