Marshmallow 3.0.1 - 'NoneType' object has no attribute 'opts'

[davidzwa]

Exception Value: 'NoneType' object has no attribute 'opts'

Description: when updating marshmallow from 3.0.0 to 3.0.1, all my API endpoints using any marshmallow serializer get this error. @sloria I am not sure if this is mmallow or django-mmallow
Expected behaviour: being able to use django 4.0+ and marshmallow 3.4 in combination with this interfacing library without error.

Pipfile extract:

Working version (before changing to marshmallow 3.0.1).

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]
django-rainbowtests = "*"
psycopg2 = "*"

[packages]
...
django = "==2.1.15"
djangorestframework = "==3.10.2"
djangorestframework_simplejwt = "==4.4.0"
django-cors-headers = "==3.2.1"
django-rest-marshmallow = "~=4.0.1"
marshmallow = "==3.0.0"
django-url-filter = "==0.3.15"
...

[requires]
python_version = "3.7"

Complete error trace:

ERROR 2020-02-17 17:55:38,623 log 9152 6372 Internal Server Error: /api/v1/user/14/
Traceback (most recent call last):
  File "C:\Users\david\.virtualenvs\ds4reboot-pVDuRnsS\lib\site-packages\django\core\handlers\exception.py", line 34, in inner
    response = get_response(request)
  File "C:\Users\david\.virtualenvs\ds4reboot-pVDuRnsS\lib\site-packages\django\core\handlers\base.py", line 126, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "C:\Users\david\.virtualenvs\ds4reboot-pVDuRnsS\lib\site-packages\django\core\handlers\base.py", line 124, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "C:\Users\david\.virtualenvs\ds4reboot-pVDuRnsS\lib\site-packages\django\views\decorators\csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "C:\Users\david\.virtualenvs\ds4reboot-pVDuRnsS\lib\site-packages\rest_framework\viewsets.py", line 114, in view
    return self.dispatch(request, *args, **kwargs)
  File "C:\Users\david\.virtualenvs\ds4reboot-pVDuRnsS\lib\site-packages\rest_framework\views.py", line 497, in dispatch
    response = self.handle_exception(exc)
  File "C:\Users\david\.virtualenvs\ds4reboot-pVDuRnsS\lib\site-packages\rest_framework\views.py", line 457, in handle_exception
    self.raise_uncaught_exception(exc)
  File "C:\Users\david\.virtualenvs\ds4reboot-pVDuRnsS\lib\site-packages\rest_framework\views.py", line 468, in raise_uncaught_exception
    raise exc
  File "C:\Users\david\.virtualenvs\ds4reboot-pVDuRnsS\lib\site-packages\rest_framework\views.py", line 494, in dispatch
    response = handler(request, *args, **kwargs)
  File "C:\Users\david\.virtualenvs\ds4reboot-pVDuRnsS\lib\site-packages\rest_framework\mixins.py", line 56, in retrieve
    return Response(serializer.data)
  File "C:\Users\david\.virtualenvs\ds4reboot-pVDuRnsS\lib\site-packages\rest_marshmallow\__init__.py", line 70, in data
    self._serializer_data = self.to_representation(self.instance)
  File "C:\Users\david\.virtualenvs\ds4reboot-pVDuRnsS\lib\site-packages\rest_marshmallow\__init__.py", line 41, in to_representation
    return self.dump(instance)
  File "C:\Users\david\.virtualenvs\ds4reboot-pVDuRnsS\lib\site-packages\marshmallow\schema.py", line 541, in dump
    result = self._serialize(processed_obj, many=many)
  File "C:\Users\david\.virtualenvs\ds4reboot-pVDuRnsS\lib\site-packages\marshmallow\schema.py", line 505, in _serialize
    value = field_obj.serialize(attr_name, obj, accessor=self.get_attribute)
  File "C:\Users\david\.virtualenvs\ds4reboot-pVDuRnsS\lib\site-packages\marshmallow\fields.py", line 309, in serialize
    return self._serialize(value, attr, obj, **kwargs)
  File "C:\Users\david\.virtualenvs\ds4reboot-pVDuRnsS\lib\site-packages\marshmallow\fields.py", line 513, in _serialize
    schema = self.schema
  File "C:\Users\david\.virtualenvs\ds4reboot-pVDuRnsS\lib\site-packages\marshmallow\fields.py", line 498, in schema
    dump_only=self._nested_normalized_option("dump_only"),
  File "C:\Users\david\.virtualenvs\ds4reboot-pVDuRnsS\lib\site-packages\rest_marshmallow\__init__.py", line 36, in __init__
    MarshmallowSchema.__init__(self, **schema_kwargs)
  File "C:\Users\david\.virtualenvs\ds4reboot-pVDuRnsS\lib\site-packages\marshmallow\schema.py", line 383, in __init__
    self.fields = self._init_fields()
  File "C:\Users\david\.virtualenvs\ds4reboot-pVDuRnsS\lib\site-packages\marshmallow\schema.py", line 913, in _init_fields
    self._bind_field(field_name, field_obj)
  File "C:\Users\david\.virtualenvs\ds4reboot-pVDuRnsS\lib\site-packages\marshmallow\schema.py", line 969, in _bind_field
    field_obj._bind_to_schema(field_name, self)
  File "C:\Users\david\.virtualenvs\ds4reboot-pVDuRnsS\lib\site-packages\marshmallow\fields.py", line 1117, in _bind_to_schema
    or getattr(self.root.opts, self.SCHEMA_OPTS_VAR_NAME)
AttributeError: 'NoneType' object has no attribute 'opts'

[sloria]

Thanks for reporting. I unfortunately won't have time to do a code dive in the near future. Feel free to send a PR if you figure it out.

[davidzwa]

This seems to be an easy fix in schema.py or fields.py on some marshmallow change to self.root w.r.t self.root.opts.

The self.root, what does it represent? Was there any change on the marshmallow side of things?

[sloria]

self.root is the schema that a field is bound to. Can you post what your schema looks like?

[davidzwa]

I actually can, this project is opensource:
https://github.com/studentenhuisDS4/ds4reboot/blob/develop/user/api/serializers/user.py

Find the urls/routing in the default app of the project
https://github.com/studentenhuisDS4/ds4reboot/blob/develop/ds4reboot/urls.py

[sloria]

Cool; do you know which schema is raising the error?

[davidzwa]

All of 'm! Every endpoint using any Schema derivative is throwing the error. Also: the stacktrace is not pointing to any of my files, so that's why I thought there must be an incompatibility issue.

In this case UserSchema shows it, but any other does as well

[sloria]

Have you tried upgrading your marshmallow version? We currently test this package against the latest marshmallow 3.x (3.4.0 as of this writing) and DRF 3.{8,9}.

[davidzwa]

Yes I did :) Both updated DRF and MM showed the same problems, so I had to take steps back in my Pipfile. I tried Django, DRF, MM and DRF-MM, until I noticed it started working. I re-upgraded everything to find DRF-MM 3.0.0 -> 3.0.1 (and beyond) to be the problem.

Will try again tonight

[michaelwiles]

So the issue for this one and for #110 is that marshmallow base expects parent to not exist.

When searching for root it does this:

    @property
    def root(self):
        """Reference to the `Schema` that this field belongs to even if it is buried in a
        container field (e.g. `List`).
        Return `None` for unbound fields.
        """
        ret = self
        while hasattr(ret, "parent"):
            ret = ret.parent
        return ret if isinstance(ret, SchemaABC) else None

Notice how the code checks the for the existence of the parent field.

So it expects it not to be there. If parent is is parent on ret but is null, this method returns null and thus you have the error outlined above in this method in marshmallow.fields.DateTime

    def _bind_to_schema(self, field_name, schema):
        super()._bind_to_schema(field_name, schema)
        self.format = (
            self.format
            or getattr(self.root.opts, self.SCHEMA_OPTS_VAR_NAME)
            or self.DEFAULT_FORMAT
        )

The error happens becuse root is None.

Now the question remains... why is parent None and not not present?

This is because there is a parent field initialised to None in django-rest-framework/rest_framework/fields.py#358

For one thing, the test is simple, just add a date field to the ExampleSerializer schema in the tests...

Now consider:

marshmallow/fields.py:Field.root

    @property
    def root(self):
        print('in marshmallow fields.root')
        """Reference to the `Schema` that this field belongs to even if it is buried in a
        container field (e.g. `List`).
        Return `None` for unbound fields.
        """
        ret = self
        while hasattr(ret, "parent"):
            ret = ret.parent
        return ret if isinstance(ret, SchemaABC) else None

and

rest_framework/fields.py:root

    @property
    def root(self):
        print ('in rest_framework.fields.root')
        """
        Returns the top-level serializer for this field.
        """
        root = self
        while root.parent is not None:
            root = root.parent
        return root

Notice how it looks for None parent and not parent that is not present - consistent with the code in
django-rest-framework/rest_framework/fields.py#358 - referenced above...

So what to do?

I don't think we can switch to using the other root method as the constructor of Schema explicitly calls

MarshmallowSchema.__init__(self, **schema_kwargs)

So considering:

1. the marshmallow root method is being called
2. this is django-rest-marshmallow
3. we can't chagne the rest_framework behaviour
4. and we can't switch to using a different root method (I don't think)

The only alternative, in my opinion, is to change the semantics of the parent field to match what marshmallow expects - that is, delete the attribute.

So in the pull request in Schema the code

        super(Schema, self).__init__(*args, **kwargs)
        MarshmallowSchema.__init__(self, **schema_kwargs)

has a delattr inserted between:

        super(Schema, self).__init__(*args, **kwargs)
        delattr(self, 'parent')
        MarshmallowSchema.__init__(self, **schema_kwargs)

[davidzwa]

Great work! @sloria, kindly check this hero's work out, if you find the time.

[sloria]

Fix released in 4.0.2

[michaelwiles]

@sloria thank you so much merging and releasing so quickly. Wow, I was surprised.

Never had a pull accepted that quickly before.

I'm desperate for the fix but it has not reflected on pypi yet...

Comparing the 4.0.1 build logs to 4.0.2 build logs it seems that 4.0.2 did not run the Pypi release stage... see 4.0.1 and 4.0.2

It makes no sense as I can't see any difference between the builds.

[sloria]

Ah, travis doesn't like the config

unfortunately I won't have time to look into this today; @michaelwiles would you like to do some digging on how to get matrix set correctly for the testing stage?

[sloria]

Looking into the travis issues in #136

[sloria]

P.S. we should really switch to Azure Pipelines to get this project in line with the rest of marshmallow-code... just haven't had the time.. =/

[sloria]

Fixed! https://pypi.org/project/django-rest-marshmallow/4.0.2/

[davidzwa]

[Moved answer about Azure to new issue #139]