New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

No attribute '_add_to_schema' when dumping Schema and Nested Field is None #948

Closed
vke-code opened this Issue Sep 12, 2018 · 4 comments

Comments

Projects
None yet
3 participants
@vke-code

vke-code commented Sep 12, 2018

Hi Marshmallow Code!

I have recently discovered and began implementing your code for my own projects and I must say you have changed how I work with databases forever!

I have been running up against a wall with this issue however, and decided to reach out to the pros! It could be that I am missing something simple.

For context, I am storing hosts in a database. 'Hosts' have a one-to-many relationship with 'Ports'.

I have developed a HostSchema and PortSchema using ModelSchema from marshmallow-sqlalchemy. This works BEAUTIFULLY, except when a host does not have any ports associated with it.

Attached below is example code which demonstrates the expected behavior as well as the error. Please excuse me if this isn't an actual bug and just user error, although some insight would be very much appreciated.

Truncated Pip Freeze

  • Flask==1.0.2
  • flask-marshmallow==0.9.0
  • Flask-SQLAlchemy==2.3.2
  • marshmallow==2.15.4
  • marshmallow-sqlalchemy==0.14.1
  • SQLAlchemy==1.2.11

Minimal Example

from flask import Flask
from flask_sqlalchemy import SQLAlchemy
from flask_marshmallow import Marshmallow

from marshmallow import fields, post_dump, pre_dump
import json

app = Flask(__name__)
db = SQLAlchemy(app)
ma = Marshmallow(app)

class Host(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    ip = db.Column(db.String(16), index=True, unique=True, nullable=False)

    ports = db.relationship('Port', backref='host', lazy='dynamic')


class Port(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    host_id = db.Column(db.Integer, db.ForeignKey('host.id'), index=True)
    port = db.Column(db.Integer)
    transport = db.Column(db.String(8))
    service = db.Column(db.String(16))


class PortSchema(ma.ModelSchema):
    class Meta:
        model = Port
        exclude = ('id' , 'host')
        sqla_session = db.Session

class HostSchema(ma.ModelSchema):
    ports = fields.Nested(PortSchema, many=True)
    class Meta:
        model = Host
        sqla_session = db.Session

Behavior when 'ports' is defined in 'host' object.

This works as expected.

host = Host(ip='127.0.0.1')
port = Port(port=80, transport='tcp', service='http')
host.ports.append(port)

result = HostSchema().dump(host)

print json.dumps(result.data, indent=2)


# {
#   "ip": "127.0.0.1",
#   "ports": [
#     {
#       "transport": "tcp",
#       "service": "http",
#       "port": 80
#     }
#   ],
#   "id": null
# }

Unexpected behavior when 'ports' is undefined in 'host' object.

From what I've gathered from the documentation the default behavior is to skip any fields which are missing during serialization. I have also tried using missing and default, but if I understand correctly these only apply for deserialization.

host = Host(ip='127.0.0.1')
result = HostSchema().dump(host)

Expected Behavior

{
  "ip": "127.0.0.1",
  "ports": {},
  "id": null
}

Actual result:

Traceback (most recent call last):
  File "scratch/missing-field-nested-bug.py", line 72, in <module>
    result = HostSchema().dump(host)
  File "/Users/kai/.virtualenvs/SPEZZI/lib/python2.7/site-packages/marshmallow/schema.py", line 509, in dump
    **kwargs
  File "/Users/kai/.virtualenvs/SPEZZI/lib/python2.7/site-packages/marshmallow/marshalling.py", line 138, in serialize
    index=(index if index_errors else None)
  File "/Users/kai/.virtualenvs/SPEZZI/lib/python2.7/site-packages/marshmallow/marshalling.py", line 62, in call_and_store
    value = getter_func(data)
  File "/Users/kai/.virtualenvs/SPEZZI/lib/python2.7/site-packages/marshmallow/marshalling.py", line 132, in <lambda>
    getter = lambda d: field_obj.serialize(attr_name, d, accessor=accessor)
  File "/Users/kai/.virtualenvs/SPEZZI/lib/python2.7/site-packages/marshmallow/fields.py", line 252, in serialize
    return self._serialize(value, attr, obj)
  File "/Users/kai/.virtualenvs/SPEZZI/lib/python2.7/site-packages/marshmallow/fields.py", line 447, in _serialize
    schema._update_fields(obj=nested_obj, many=self.many)
  File "/Users/kai/.virtualenvs/SPEZZI/lib/python2.7/site-packages/marshmallow/schema.py", line 767, in _update_fields
    self.__set_field_attrs(ret)
  File "/Users/kai/.virtualenvs/SPEZZI/lib/python2.7/site-packages/marshmallow/schema.py", line 788, in __set_field_attrs
    field_obj._add_to_schema(field_name, self)
AttributeError: 'NoneType' object has no attribute '_add_to_schema'

Thank you all for any help or insights you can offer! Please let me know if further clarification is required.

@deckar01

This comment has been minimized.

Member

deckar01 commented Sep 12, 2018

@vke-code Thanks for providing a detailed report. I was able to reproduce the issue. It looks like lazy='dynamic' is causing the problem. I confirmed that disabling that setting works around the issue.

The underlying bug is caused by the special AppenderBaseQuery instance used for ports. It is iterable, but it doesn't implement len, which causes bool(host.ports) to be true when it is empty.

The fix for this is to not return self.declared_fields unfiltered from __filter_fields when "there is nothing to serialize". If we want to optimize it, we should just return an empty dict.

@deckar01 deckar01 added the bug label Sep 12, 2018

@vke-code

This comment has been minimized.

vke-code commented Sep 12, 2018

@deckar01 Thank you so much for your quick response!

I must admit this is my first issue I've posted to Github, so I am glad that you found the information helpful and were able to identify the issue. I honestly just assumed it was user error!

Removing lazy='dynamic' in the original program I was developing fixes the issue. I will go ahead and poke around on my end and see if this is something I can patch, though I am relatively new to marshmallow and Python in general so this may be a little out of my reach.

@deckar01

This comment has been minimized.

Member

deckar01 commented Sep 13, 2018

This bug was patched in v3 by #618. We just need to cherry pick that into 2.x-line.

@lafrech

This comment has been minimized.

Member

lafrech commented Oct 11, 2018

I think this was closed in #950 but didn't get automatically closed when merging.

@lafrech lafrech closed this Oct 11, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment