Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

AttributeError after pickling queryset with related instances #25

Open
jhonatan-lopes opened this issue Sep 21, 2023 · 2 comments
Open

AttributeError after pickling queryset with related instances #25

jhonatan-lopes opened this issue Sep 21, 2023 · 2 comments
Labels
bug Something isn't working

Comments

@jhonatan-lopes
Copy link

jhonatan-lopes commented Sep 21, 2023 •
edited

When trying to request a URL after installing the package, I get an AttributeError when trying to access a page:

foundationmozillaorg-backend-1   | 17:12:17 web.1            | Internal Server Error: /en/privacynotincluded/
foundationmozillaorg-backend-1   | 17:12:17 web.1            | Traceback (most recent call last):
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 829, in _resolve_lookup
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     current = current[bit]
foundationmozillaorg-backend-1   | 17:12:17 web.1            | TypeError: 'ProductPage' object is not subscriptable
foundationmozillaorg-backend-1   | 17:12:17 web.1            | 
foundationmozillaorg-backend-1   | 17:12:17 web.1            | During handling of the above exception, another exception occurred:
foundationmozillaorg-backend-1   | 17:12:17 web.1            | 
foundationmozillaorg-backend-1   | 17:12:17 web.1            | Traceback (most recent call last):
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/db/models/fields/related_descriptors.py", line 173, in __get__
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     rel_obj = self.field.get_cached_value(instance)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/db/models/fields/mixins.py", line 15, in get_cached_value
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return instance._state.fields_cache[cache_name]
foundationmozillaorg-backend-1   | 17:12:17 web.1            | KeyError: 'image'
foundationmozillaorg-backend-1   | 17:12:17 web.1            | 
foundationmozillaorg-backend-1   | 17:12:17 web.1            | During handling of the above exception, another exception occurred:
foundationmozillaorg-backend-1   | 17:12:17 web.1            | 
foundationmozillaorg-backend-1   | 17:12:17 web.1            | Traceback (most recent call last):
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/core/handlers/exception.py", line 47, in inner
foundationmozillaorg-backend-1   | 
foundationmozillaorg-backend-1   | 
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     response = get_response(request)
foundationmozillaorg-backend-1   | 
foundationmozillaorg-backend-1   | 
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/core/handlers/base.py", line 204, in _get_response
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     response = response.render()
foundationmozillaorg-backend-1   | 
foundationmozillaorg-backend-1   | 
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/response.py", line 105, in render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     self.content = self.rendered_content
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/response.py", line 83, in rendered_content
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return template.render(context, self._request)
foundationmozillaorg-backend-1   | 
foundationmozillaorg-backend-1   | 
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/backends/django.py", line 61, in render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return self.template.render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 170, in render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return self._render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 162, in _render
foundationmozillaorg-backend-1   | 
foundationmozillaorg-backend-1   | 
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return self.nodelist.render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 938, in render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     bit = node.render_annotated(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 905, in render_annotated
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return self.render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/pattern_library/loader_tags.py", line 38, in render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return super().render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/loader_tags.py", line 150, in render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return compiled_parent._render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 162, in _render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return self.nodelist.render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 938, in render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     bit = node.render_annotated(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 905, in render_annotated
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return self.render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/pattern_library/loader_tags.py", line 38, in render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return super().render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/loader_tags.py", line 150, in render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return compiled_parent._render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 162, in _render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return self.nodelist.render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 938, in render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     bit = node.render_annotated(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 905, in render_annotated
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return self.render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/pattern_library/loader_tags.py", line 38, in render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return super().render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/loader_tags.py", line 150, in render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return compiled_parent._render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 162, in _render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return self.nodelist.render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 938, in render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     bit = node.render_annotated(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 905, in render_annotated
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return self.render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/loader_tags.py", line 62, in render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     result = block.nodelist.render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 938, in render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     bit = node.render_annotated(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 905, in render_annotated
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return self.render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/loader_tags.py", line 62, in render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     result = block.nodelist.render(context)
foundationmozillaorg-backend-1   | 
foundationmozillaorg-backend-1   | 
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 938, in render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     bit = node.render_annotated(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 905, in render_annotated
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return self.render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/defaulttags.py", line 315, in render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return nodelist.render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 938, in render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     bit = node.render_annotated(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 905, in render_annotated
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return self.render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/templatetags/cache.py", line 47, in render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     value = self.nodelist.render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 938, in render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     bit = node.render_annotated(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 905, in render_annotated
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return self.render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/defaulttags.py", line 214, in render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     nodelist.append(node.render_annotated(context))
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 905, in render_annotated
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return self.render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/pattern_library/loader_tags.py", line 90, in render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return super().render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/loader_tags.py", line 195, in render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return template.render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 172, in render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return self._render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 162, in _render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return self.nodelist.render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 938, in render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     bit = node.render_annotated(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 905, in render_annotated
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return self.render(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/wagtail/images/templatetags/wagtailimages_tags.py", line 101, in render
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     image = self.image_expr.resolve(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 671, in resolve
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     obj = self.var.resolve(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 796, in resolve
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     value = self._resolve_lookup(context)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/template/base.py", line 837, in _resolve_lookup
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     current = getattr(current, bit)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/dj_tracker/tracker.py", line 247, in wrapper
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     value = get_attr(instance, attr)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/dj_tracker/field_descriptors.py", line 15, in __get__
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return self.descriptor.__get__(instance, cls)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/db/models/fields/related_descriptors.py", line 187, in __get__
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     rel_obj = self.get_object(instance)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/db/models/fields/related_descriptors.py", line 154, in get_object
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     return qs.get(self.field.get_reverse_related_filter(instance))
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/db/models/query.py", line 431, in get
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     num = len(clone)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/db/models/query.py", line 262, in __len__
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     self._fetch_all()
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/django/db/models/query.py", line 1324, in _fetch_all
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     self._result_cache = list(self._iterable_class(self))
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/dj_tracker/tracker.py", line 117, in __iter__
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     qs_tracker = QuerySetTracker(
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/dj_tracker/datastructures.py", line 370, in __init__
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     instance_tracker.queryset.add_related_queryset(self)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/dj_tracker/datastructures.py", line 377, in add_related_queryset
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     self.related_querysets.append(qs_tracker)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/dj_tracker/datastructures.py", line 330, in __getattr__
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     self.constructed.add(name)
foundationmozillaorg-backend-1   | 17:12:17 web.1            |   File "/app/dockerpythonvenv/lib/python3.9/site-packages/dj_tracker/datastructures.py", line 333, in __getattr__
foundationmozillaorg-backend-1   | 17:12:17 web.1            |     raise AttributeError(name)

To reproduce:

  1. Clone the Mozilla Foundation repo: https://github.com/MozillaFoundation/foundation.mozilla.org
  2. Install dj_tracker (adding either to dev-requirements.in or requirements.in)
  3. Update pip locks by running inv pip-compile-lock
  4. Run inv setup locally to setup the dev environment and create some fake data
  5. Navigate to the homepage ("localhost:8000") and see that it works fine
  6. Navigate to the *privacy not included page ("localhost:8000/en/privacynotincluded") and see the bug
@Tijani-Dia
Copy link
Owner

Tijani-Dia commented Sep 21, 2023 •
edited

Hey @jhonatan-lopes, thanks for raising this. I spent some time debugging it and I found that it's related to how query trackers are pickled.

get_product_subset returns the products after they've been pickled via cache.get_or_set.

I recently implemented custom hooks to solve a similar issue: Fix pickling errors with related instances.

Currently, the __getstate__ method returns only values that were added to the tracker dictionary, not the one defined in the __slots__ attribute. That explains why we have the AttributeError: constructed since constructed is defined in the __slots__.

I thought about adding all the values in __slots__ when pickling but that would introduce more serious bugs if everything else remains unchanged. Handling stateful objects is quite tricky with pickle so I think we we'll need to think more about it.

When testing the view that raises errors, you can modify the last line from get_product_subset to the following as a temporary workaround:

return cache.get_or_set(key, products, 24 * 60 * 60)  # Set cache for 24h

# Change to

cache.get_or_set(key, products, 24 * 60 * 60)  # Set cache for 24h
return products

@Tijani-Dia Tijani-Dia added the bug Something isn't working label Sep 21, 2023
@Tijani-Dia Tijani-Dia changed the title AttributeError after installing package AttributeError after pickling queryset Sep 21, 2023
@Tijani-Dia Tijani-Dia changed the title AttributeError after pickling queryset AttributeError after pickling queryset with related instances Sep 21, 2023
@jhonatan-lopes
Copy link
Author

Hi @Tijani-Dia, thanks for the quick investigation! Your proposed solution worked like a charm!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jhonatan-lopes @Tijani-Dia