Convert model instance to JSON

[karloscodes]

Is there a way to convert a model instance to JSON??, I'm getting <Obj> is not JSON serializable when trying to serialize with JSON module

[csimmons0]

Hi, Carlos. I just submitted a pull request (#156) that may provide what you're looking for. It lets you convert a model instance to a Python dictionary, and you can easily convert a dictionary to JSON.

[yedpodtrzitko]

@ccverak you can use Model()._serialize()

[jdno]

As @csimmons0 said in his pull request: _serialize() is too close to the data representation in DynamoDB and thus not suitable for (my) APIs. Besides, _serialize() looks like a "private" method and should not be used by third-party code.

I agree that we need a method that produces a dict or JSON representation of the model object that abstracts from the underlying data representation. For example, here is what I get when I serialize my object with _serialize():

{
  'attributes': {
    'owner': {'S': u'test|12345678'},
    'updated': {'S': u'2016-12-19T14:52:32.140080+00:00'},
    'name': {'S': u'Test Organization'},
    'members': {'SS': [u'test|12345678']},
    'created': {'S': u'2016-12-19T14:52:32.140106+00:00'}
  },
  'HASH': u'59fc2bcd-230d-427c-ae9f-9834bf29fc8a'
}

However, the following is what I need to pass it along to my user-facing API:

{
  'id': '59fc2bcd-230d-427c-ae9f-9834bf29fc8a',
  'name': 'Test Organization',
  'owner': 'test|12345678',
  'members': [
    'test|12345678'
  ],
  'created': '2016-12-19T14:52:32.140106+00:00',
  'updated': '2016-12-19T14:52:32.140106+00:00'
}

Would be great to get some feedback on this from @jlafon so that we can merge (or work on) #156.

[mathom]

@jdno and @ccverak:

I'm using this in my project - I hope it helps if you're still stuck!

class ModelEncoder(json.JSONEncoder):
    def default(self, obj):
        if hasattr(obj, 'attribute_values'):
            return obj.attribute_values
        elif isinstance(obj, datetime.datetime):
            return obj.isoformat()
        return json.JSONEncoder.default(self, obj)


def json_dumps(obj):
    return json.dumps(obj, cls=ModelEncoder)

[kadrach]

You can also utilize the built-in serializers, see e.g. here for datetime. Something along the lines of

class Model(pynamodb.models.Model):
    ...

    def __iter__(self):
        for name, attr in self._get_attributes().items():
            yield name, attr.serialize(getattr(self, name))

dict(model_instance) should return the serialized object. Works for my simple usecase, not extensively tested 😄

[toshke]

by simple extension or using base class, it is easy to get dict for the dynamo item, which can be furhter serialised.. i tried @mathom answer, but it did not worked for MapAttribute type. However just extending models from base class below was enough to extract dict and further serialise to JSON/xml

from pynamodb.models import Model
from pynamodb.attributes import UnicodeAttribute, ListAttribute, MapAttribute
import os
import json


class BaseModel(Model):
    def to_dict(self):
        rval = {}
        for key in self.attribute_values:
            rval[key] = self.__getattribute__(key)
        return rval


class AddressModel(MapAttribute):
    street = UnicodeAttribute()
    city = UnicodeAttribute()


class Person(BaseModel):
    class Meta:
        table_name = os.environ.get('PERSONS_TABLE', 'persons')
    
    id = UnicodeAttribute(hash_key=True)
    addresses = ListAttribute(of=AddressModel)


if not Person.exists():
    Person.create_table(read_capacity_units=1, write_capacity_units=1, wait=True)

p = Person(hash_key='1', range_key=None, addresses=[{'street': 'JohnSt', 'city': 'New York, NYC'}])
print(json.dumps(p.to_dict(), indent=2))

[gabrielBusta]

Here is yet another possible solution.

Each instance of a model is serialized into a JSON list where the hashing key is the 1st element. Then, the 2nd element is a JSON object with the values of each one of the model's attributes.

i.e.

[
    "hashing key",
    {
        "attribute 1": "value 1"t
        "attribute 2": "value 2",
        ...
        "attribute n": "value n"
    }
]

This approach has been working well for me so far.

import json
from pynamodb.models import Model


def head(iterable):
    return iterable[0]


class EZJSONModel(Model):
    """
    A model that can be serialized to JSON easily.
    """
    def serialize(self):
        """
        :return: a json list in the following format [hash_key, attribute_dict]
        """
        hash_key, data = self._get_json()
        data = data['attributes']
        data = {
            attr: head(list(data[attr].values())) for attr in data
        }
        return json.dumps([hash_key, data])

[chrnorm]

I expanded on @kadrach's solution - this worked for my table which contained some MapAttributes as well as NumberAttributes and UnicodeAttributes.

class MyModel(pynamodb.models.Model)
...
    def __iter__(self):
        for name, attr in self.get_attributes().items():
            if isinstance(attr, MapAttribute):
                yield name, getattr(self, name).as_dict()
            else:
                yield name, attr.serialize(getattr(self, name))

dict(model_instance) then returned a dict of the model. Untested for models containing ListAttributes!

[codeocelot]

There's a small typo in @chrisnorman27's snippet.

-         for name, attr in self.get_attributes().items():
+         for name, attr in self._get_attributes().items():

From his solution, this how I added support for ListAttributes:

class MyModel(pynamo.models.Model):
    def __iter__(self):
        for name, attr in self._get_attributes().items():
            if isinstance(attr, MapAttribute):
                yield name, getattr(self, name).as_dict()
            if isinstance(attr, ListAttribute):
                yield name, [el.as_dict() for el in getattr(self, name)]
            else:
                yield name, attr.serialize(getattr(self, name))

I feel like this is something this library should natively support, or least document. It's easy to get sucked up into thinking deserialize/serialize methods are for JSON serialization.

[mikegrima]

@codeocelot I implemented this in my project, but made a slight tweak where I needed to add in elif statements after your initial if. I also needed to add a special check for NumberAttributes to make them ints (which is not totally correct, because they can also be floats). Also added a null check.

Code here: https://github.com/Netflix-Skunkworks/historical/blob/master/historical/models.py#L33-L49

[leobarcellos]

There's a small typo in @chrisnorman27's snippet.

-         for name, attr in self.get_attributes().items():
+         for name, attr in self._get_attributes().items():

From his solution, this how I added support for ListAttributes:

class MyModel(pynamo.models.Model):
    def __iter__(self):
        for name, attr in self._get_attributes().items():
            if isinstance(attr, MapAttribute):
                yield name, getattr(self, name).as_dict()
            if isinstance(attr, ListAttribute):
                yield name, [el.as_dict() for el in getattr(self, name)]
            else:
                yield name, attr.serialize(getattr(self, name))

I feel like this is something this library should natively support, or least document. It's easy to get sucked up into thinking deserialize/serialize methods are for JSON serialization.

I've added a UTCDateTimeAttribute test, since it was failing when None (trying to get tzinfo)

def __iter__(self):
    for name, attr in self.get_attributes().items():
        if isinstance(attr, MapAttribute):
            if getattr(self, name):
                # app.logger.debug(name)
                yield name, getattr(self, name).as_dict()
        elif isinstance(attr, UTCDateTimeAttribute):
            if getattr(self, name):
                yield name, attr.serialize(getattr(self, name))
        else:
            yield name, attr.serialize(getattr(self, name))

[mikegrima]

Does anyone know how to deal with nested values? In my case, I'm having an problem with a nested Decimal inside of a MapAttribute :/

I suppose I could replace the yield line here with some code that would recursively check for and remove the stupid Decimals (really hate that boto went that way...):

                if isinstance(attr, MapAttribute):
                    yield name, getattr(self, name).as_dict()

EDIT: I fixed my issue by adding this code in: boto/boto3#369 (comment). I basically pass the getattr(self, name).as_dict() into that function.

[bendog]

I've added a numeric test to return numbers as numerics as this understandably was causing issues with sorting.

    def __iter__(self):
        for name, attr in self.get_attributes().items():
            if isinstance(attr, attributes.MapAttribute):
                if getattr(self, name):
                    yield name, getattr(self, name).as_dict()
            elif isinstance(attr, attributes.UTCDateTimeAttribute):
                if getattr(self, name):
                    yield name, attr.serialize(getattr(self, name))
            elif isinstance(attr, attributes.NumberAttribute):
                # if numeric return value as is.
                yield name, getattr(self, name)
            else:
                yield name, attr.serialize(getattr(self, name))

[gowtham800]

I am getting empty attribute object response. Can some one help me please....

        class PNFGDINDEX(GlobalSecondaryIndex):

            class Meta:
                index_name = 'PN-FGD-Index'
                read_capacity_units = 5
                write_capacity_units = 5
                projection = AllProjection()

            ProjectName = UnicodeAttribute(hash_key=True)
            FileGeneratedDate = UnicodeAttribute(range_key=True)

        class TestModel(Model):
          
            class Meta:
                aws_access_key_id = "xxxxxxxxx"
                aws_secret_access_key = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
                table_name = 'TEST_EDCBUILDDATA1'
                region = 'ap-south-1'
                
            Id = UnicodeAttribute(hash_key=True)
            createdTimeStamp = UnicodeAttribute(range_key=True)
            ProjectName = UnicodeAttribute
            FileGeneratedDate = UnicodeAttribute
            FileNameKey = UnicodeAttribute
            VehicleName = UnicodeAttribute
            view_index = PNFGDINDEX()
            

        for item in TestModel.view_index.query(pn):
            print(item._serialize())

[shungok]

Here is my solution for multi tier structure.

from datetime import datetime
import json

from pynamodb.models import Model
from pynamodb.attributes import MapAttribute

class BaseModel(Model):
    def to_json(self, indent=2):
        return json.dumps(self.to_dict(), indent=indent)
        
    def to_dict(self):
        ret_dict = {}
        for name, attr in self.attribute_values.items():
            ret_dict[name] = self._attr2obj(attr)
        
        return ret_dict

    def _attr2obj(self, attr):
        # compare with list class. It is not ListAttribute.
        if isinstance(attr, list):
            _list = []
            for l in attr:
                _list.append(self._attr2obj(l))
            return _list
        elif isinstance(attr, MapAttribute):
            _dict = {}
            for k, v in attr.attribute_values.items():
                _dict[k] = self._attr2obj(v)
            return _dict
        elif isinstance(attr, datetime):
            return attr.isoformat()
        else:
            return attr

[Swington]

Because my models had many nested attributes (e.g. my model had a list of map, which contained a map) none of the solutions described above seemed to work.

I was able to compose a solution using dynamodb-json module, though:

json_util.loads(model._serialize()['attributes'])

The only inconvenience here is that Keys are kept separately from attributes, you have to compose a common dict from all of those. As an instance method, of a class with single Hash Key, it would like something like that:

class BaseModel(Model):
    def to_dict(self):
        serialized_self = self._serialize()
        serialized_self_attributes = serialized_self['attributes']
        if self._hash_keyname is not None:
            serialized_self_attributes.update({self._hash_keyname: serialized_self['HASH']})
        if self._range_keyname is not None:
            serialized_self_attributes.update({self._range_keyname: serialized_self['RANGE']})
        return json_util.loads(serialized_self_attributes)

[hassan-bazzi]

If you have a custom MapAttribute you can use .as_dict() i.e.

YourModel(user_id).application.as_dict()

Where application is a custom attribute that extends MapAttribute.

The only issue with that was that any ListAttributes embedded in another MapAttribute inside of our application MapAttribute don't jsonify properly. I made a custom BaseMapAttribute class and used the solution here:

#491

To override as_dict and make it transform the objects in the list as dicts

import six
from pynamodb.attributes import MapAttribute


class BaseMapAttribute(MapAttribute):
    def as_dict(self):
        result = {}
        for key, value in six.iteritems(self.attribute_values):
            if isinstance(value, list):
                result[key] = []
                for v in value:
                    if isinstance(v, MapAttribute):
                        result[key].append(v.as_dict())
                    else:
                        result[key].append(v)
            elif isinstance(value, MapAttribute):
                result[key] = value.as_dict()
            else:
                result[key] = value
        return result

[jpinner-lyft]

Closed by #857