Nested resource serialization

[swistakm]

I run on this from time to time. In most cases nested resource can be handled by using custom fields but they cannot be trivially validated/serialized/deserialized. Also those new fields classes cannot be reused as unique resource types so in some cases it would be useful to have support for nesting resource definitions.

I see two approaches:

1. wrapper field class (like SerializerField) that can work like a single field and delegates validation and (de)serialisation to inner serialised stored as a attribute. Pros: more explicit, allows both types of objects to have different apis. Cons: feels like more complex and also will require more coding to define API that uses nested resources.
2. make BaseSerializer a descendant of BaseField class. This should also work but will require serialisers and fields to have the same api. I'm afraid that this will make auto-describing of API more painful. Pros: shorter API descriptions. Cons: fields and serialisers have slightly different purposes so such coupling may introduce difficulties in future.

[swistakm]

It looks that option 1. is as simple as:

index 25728e3..6368521 100644
--- a/src/graceful/serializers.py
+++ b/src/graceful/serializers.py
@@ -64,7 +64,7 @@ class MetaSerializer(type):
         )


-class BaseSerializer(metaclass=MetaSerializer):
+class BaseSerializer(BaseField, metaclass=MetaSerializer):
     """Base serializer class for describing internal object serialization.

     Example:
@@ -81,6 +81,8 @@ class BaseSerializer(metaclass=MetaSerializer):
             height = FloatField("cat height in cm")

     """
+    def __init__(self, details=None, **kwargs):
+        super().__init__(details=details, **kwargs)

     @property
     def fields(self):

Here is a small snippet of code that shows how nested resource serializers could be defined and proofs that it works:

from pprint import pprint

from graceful.serializers import BaseSerializer
from graceful.fields import StringField


class Inner(BaseSerializer):
    fooX = StringField("Foo field", source='foo')
    barX = StringField("Bar field", source='bar')


class Outer(BaseSerializer):
    innerX = Inner(details="inner object", source='inner')
    nameX = StringField("Name field", source='name')


def main():
    outer = Outer()

    internal = {
        "inner": {
            "foo": "something",
            "bar": "something else",
        },
        "name": "my outer"
    }

    print("\n ====== internal ======")
    pprint(internal)

    print("\n === representation ===")
    pprint(outer.to_representation(internal))

    print("\n ====== internal ======")
    pprint(outer.from_representation(outer.to_representation(internal)))

if __name__ == "__main__":
    main()

And example output:

$ python nested.py 

 ====== internal ======
{'inner': {'bar': 'something else', 'foo': 'something'}, 'name': 'my outer'}

 === representation ===
{'innerX': {'barX': 'something else', 'fooX': 'something'}, 'nameX': 'my outer'}

 ====== internal ======
{'inner': {'bar': 'something else', 'foo': 'something'}, 'name': 'my outer'}

But the problem is in how we will handle "describe()" methods in serializers. If they are fields now we should make them described the same way as fields so with "details", "type", "label", "spec", "read_only" attributes. Right now we put all definition to "fields" dictionary of resource metadata dict but we cannot handle this that way anymore.

I think we could utilise serializer class docstring as a field details and completely change the overall layout of description output. Instead of having single fields dict in the resource metadata we could have something more nested:

{
    ...,
    'representation': {  # => result of serializer.describe()
         'details': ... # => serializers docstring
         'label': ...    # => optional label for consistency
         'spec': ...    # => spec, here developer can add link for docs in case of nested serializers
         'type': 'representation', # => optionally 'object'/'dict' to inform it is another dict
         'fields': {  # => content of former 'fields' dictionary
             { # example of single field like str, int, float etc.
                  'details': ...
                  'label': ...
                  'spec': ...
                  'type': ....
                  # => in case of simple fields there is no 'fields' key
              },
              { # example of field coming from different serializer
                  'details': ...
                  'label': ...
                  'spec': ...
                  'type': 'representation'
                  'fields': {...}  # => fields that are serializers will have the same description
              },
         },
    },
}

Alternative way of handling nested descriptions is utilisation of spec field attribute that will now
come really handy. We could assume that resource description structure stays as it is (flat fields dict) and whenever we include serializer as a field the user should supply spec with the link of nested resource documentation. Because resources are self-descriptive (full metadata on OPTIONS request) this can be simply a link to URI of separate nested resource endpoint.

Pros:

• if user would like to create mutually nested resources (it is possible) we will not end up with infinite description loop
• it makes possible auto documenters simpler beach they will not need to deal with nested fields descriptions
• it does not require backwards incompatible changes

Cons:

• it will be harder to describe nested resources that do not have their own query endpoints

For the problem with describing nested resources that do not have their now API there is a simple solution. One could create virtual endpoint that serves only documentation purpose. It would accept only OPTIONS requests to serve specification but would not deliver any of other manipulation methods (GET/POST/PUT/DELETE).

Also the main reason for having nested resource capabilities is a use case when serializer that is nested is actually reused in some other existing resource endpoint.

[swistakm]

Seems like with the new validation in 1.0.0 version it will be even simpler and more clear. Other problem that have surfaced is how to present validation issues in nested representations and representations that have fields with many=True enables. We need to came up with some new format for validation issues that.

[holic-cl]

Hi, i am trying to do a nested serialization. My project is using peewee as orm, i am trying to fetch a list of awards with a nested user on the results set.

class AwardListSerializer(BaseSerializer):
    redeemed_at = IntField("Date when the award was redeemed.")
    user = UserSerializer()

This code is not working, my response is returning this:

{
  "meta": {
    "params": {
      "indent": 0,
      "page_size": 10,
      "page": 0
    },
    "length": 1,
    "total_items": 1,
    "page_size": 10,
    "page": 0,
    "prev": null,
    "next": "page=1&page_size=10"
  },
  "content": [
    {
      "redeemed_at": null
    }
  ]
}

I don't know if i am doing something wrong or what. I saw the test with the example to achieve it, but i can't use details neither source params on the constructor of my nested serializer.

Any clue or help i'll be very grateful.

Regards

[swistakm]

@holic-cl, there is little hiatus in this projects. As far as I remember, the proposed change was never included and released. It's mostly because I had no idea how to approach the problem of nested resource descriptions returned from .describe() handler.

Still, this feature is mostly complete so if we agree on some format, I could apply the patch, modify the handlers and release such feature as part of 1.x version.

[holic-cl]

Oh, ok, thanks for answer.