Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ensure related posts are of the same locale #8125

Merged
merged 10 commits into from Jan 24, 2022
Merged

Conversation

tbrlpld
Copy link
Collaborator

@tbrlpld tbrlpld commented Jan 19, 2022

Before, it could happen that related pages would be returned that had a
different locale than the reference page.

If a reference page would have a fairly unique combination of tags, it could
happen that all related pages would be different locales of the
reference page.

This is now prevented by filtering related pages to be of the same
locale as the reference page.

Closes #7991
Related PRs/issues #

Link to sample test page:

Checklist

Remove unnecessary checks

Tests

  • Is the code I'm adding covered by tests? No

Changes in Models:

  • Did I update or add new fake data? No
  • Did I squash my migration? No migrations added
  • Are my changes backward-compatible. If not, did I schedule a deploy with the rest of the team? Yes (no migrations)

Documentation:

  • Is my code documented? The comment is updated.
  • Did I update the READMEs or wagtail documentation? No

Before, it could happen that related pages would be returned that had a
different locale than the reference page.

If a reference page would have a fairly unique combination of tags, it could
happen that all related pages would be different locales of the
reference page.

This is now prevented by filtering related pages to be of the same
locale as the reference page.
The `.locale` property is a foreign key relation ship from the
translateable page to the `Locale` model. It should be avoided that this
look up is repeated on every loop over the different page types.
Before, the locale property was access with no regard to the page type.
Not all page types have to be translatable and have the property.

The lookup is now protected and only used when present.
@mofodevops mofodevops temporarily deployed to foundation-s-tibor-7991-6vi1ck January 19, 2022 20:56 Inactive
@mofodevops mofodevops temporarily deployed to foundation-s-tibor-7991-100pra January 19, 2022 20:56 Inactive
@mofodevops mofodevops temporarily deployed to foundation-s-tibor-7991-100pra January 19, 2022 20:58 Inactive
@mofodevops mofodevops temporarily deployed to foundation-s-tibor-7991-100pra January 19, 2022 21:02 Inactive
Much simpler... 🤦
@mofodevops mofodevops temporarily deployed to foundation-s-tibor-7991-100pra January 19, 2022 21:05 Inactive
@mofodevops mofodevops temporarily deployed to foundation-s-tibor-7991-100pra January 19, 2022 21:07 Inactive
Copy link
Contributor

@Pomax Pomax left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's see if we can do this without getattr

network-api/networkapi/wagtailpages/utils.py Outdated Show resolved Hide resolved
@mofodevops mofodevops temporarily deployed to foundation-s-tibor-7991-100pra January 19, 2022 22:20 Inactive
@tbrlpld tbrlpld requested a review from Pomax January 19, 2022 22:22
@Pomax
Copy link
Contributor

Pomax commented Jan 19, 2022

The code looks good, but visual regression doesn't ssem to like this change, so we'll want to do some debugging to find out why.

In order to run the visual regression tests locally, run docker-compose up in one terminal, and then in a second terminal make sure you've run npm install locally (you may need to install Node if you don't have that installed yet), and install playwright using npm install playwright, then run npm run playwright:install (a one time installation instruction specific to the visual regression testing framework), after which you can run npm run playwright to initial a visual test run.

@Pomax
Copy link
Contributor

Pomax commented Jan 19, 2022

Looking at the regression logs, I see:

Traceback (most recent call last):
  File "/home/runner/work/foundation.mozilla.org/foundation.mozilla.org/network-api/manage.py", line 34, in <module>
    execute_from_command_line(sys.argv)
  File "/opt/hostedtoolcache/Python/3.9.9/x64/lib/python3.9/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/opt/hostedtoolcache/Python/3.9.9/x64/lib/python3.9/site-packages/django/core/management/__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/opt/hostedtoolcache/Python/3.9.9/x64/lib/python3.9/site-packages/django/core/management/base.py", line 330, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/opt/hostedtoolcache/Python/3.9.9/x64/lib/python3.9/site-packages/django/core/management/base.py", line 371, in execute
    output = self.handle(*args, **options)
  File "/home/runner/work/foundation.mozilla.org/foundation.mozilla.org/network-api/networkapi/management/commands/load_fake_data.py", line 83, in handle
    [app_factory.generate(seed) for app_factory in [
  File "/home/runner/work/foundation.mozilla.org/foundation.mozilla.org/network-api/networkapi/management/commands/load_fake_data.py", line 83, in <listcomp>
    [app_factory.generate(seed) for app_factory in [
  File "/home/runner/work/foundation.mozilla.org/foundation.mozilla.org/network-api/networkapi/wagtailpages/factory/__init__.py", line 32, in generate
    blog.generate(seed)
  File "/home/runner/work/foundation.mozilla.org/foundation.mozilla.org/network-api/networkapi/wagtailpages/factory/blog.py", line 113, in generate
    post = BlogPageFactory.create(parent=blog_namespace, title=title)
  File "/opt/hostedtoolcache/Python/3.9.9/x64/lib/python3.9/site-packages/factory/base.py", line 528, in create
    return cls._generate(enums.CREATE_STRATEGY, kwargs)
  File "/opt/hostedtoolcache/Python/3.9.9/x64/lib/python3.9/site-packages/factory/django.py", line 117, in _generate
    return super()._generate(strategy, params)
  File "/opt/hostedtoolcache/Python/3.9.9/x64/lib/python3.9/site-packages/factory/base.py", line 465, in _generate
    return step.build()
  File "/opt/hostedtoolcache/Python/3.9.9/x64/lib/python3.9/site-packages/factory/builder.py", line 262, in build
    instance = self.factory_meta.instantiate(
  File "/opt/hostedtoolcache/Python/3.9.9/x64/lib/python3.9/site-packages/factory/base.py", line 317, in instantiate
    return self.factory._create(model, *args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.9.9/x64/lib/python3.9/site-packages/wagtail_factories/factories.py", line 69, in _create
    instance = cls._create_instance(model_class, parent, kwargs)
  File "/opt/hostedtoolcache/Python/3.9.9/x64/lib/python3.9/site-packages/wagtail_factories/factories.py", line 77, in _create_instance
    parent.add_child(instance=instance)
  File "/opt/hostedtoolcache/Python/3.9.9/x64/lib/python3.9/site-packages/treebeard/mp_tree.py", line 1083, in add_child
    return MP_AddChildHandler(self, **kwargs).process()
  File "/opt/hostedtoolcache/Python/3.9.9/x64/lib/python3.9/site-packages/treebeard/mp_tree.py", line 387, in process
    newobj.save()
  File "/home/runner/work/foundation.mozilla.org/foundation.mozilla.org/network-api/networkapi/wagtailpages/pagemodels/blog/blog.py", line 282, in save
    self.ensure_related_posts()
  File "/home/runner/work/foundation.mozilla.org/foundation.mozilla.org/network-api/networkapi/wagtailpages/pagemodels/blog/blog.py", line 269, in ensure_related_posts
    for post in self.get_missing_related_posts():
  File "/home/runner/work/foundation.mozilla.org/foundation.mozilla.org/network-api/networkapi/wagtailpages/pagemodels/blog/blog.py", line 247, in get_missing_related_posts
    related_posts = get_content_related_by_tag(self)
  File "/home/runner/work/foundation.mozilla.org/foundation.mozilla.org/network-api/networkapi/wagtailpages/utils.py", line 175, in get_content_related_by_tag
    own_locale = page.locale
  File "/opt/hostedtoolcache/Python/3.9.9/x64/lib/python3.9/site-packages/django/db/models/fields/related_descriptors.py", line 197, in __get__
    raise self.RelatedObjectDoesNotExist(
wagtail.core.models.Page.locale.RelatedObjectDoesNotExist: Page has no locale.

so I guess we'll have to go with getattr for now, but it's probably a good idea for us to file a bug over on the wagtail repo to report that as a Page model field, locale should probably default to None rather than throwing an access error.

@Pomax
Copy link
Contributor

Pomax commented Jan 20, 2022

Also, since the visual tests caught this, but the regular tests didn't, let's also add a small test in wagtailpages/tests.py, where we create a blog page and try to save it in a way that this codepath kicks in. (we can probably make that a direct test rather than faking url requests_

Copy link
Contributor

@Pomax Pomax left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

apparently page.locale can throw, which is... not so great. Let's revert to getattr, write a small test for this codepath, and let's also file an issue with wagtail to report that this property can throw and that it probably should instead universally report None if there is no localization, or the default locale if localization is active.

@tbrlpld
Copy link
Collaborator Author

tbrlpld commented Jan 20, 2022

apparently page.locale can throw, which is... not so great. Let's revert to getattr, write a small test for this codepath, and let's also file an issue with wagtail to report that this property can throw and that it probably should instead universally report None if there is no localization, or the default locale if localization is active.

I am actually confused as to why getattr worked.

@tbrlpld
Copy link
Collaborator Author

tbrlpld commented Jan 20, 2022

Re: exception vs. None: I think this comed from Django and not Wagtail. The issue seems to be that the field is not nullable but there is no value set apparently.

https://github.com/wagtail/wagtail/blob/main/wagtail/core/models/i18n.py#L89

@tbrlpld
Copy link
Collaborator Author

tbrlpld commented Jan 20, 2022

It appears the issue is not the visual test itself, but the loading of the fake data: https://github.com/mozilla/foundation.mozilla.org/runs/4874848441?check_suite_focus=true#step:10:620

@tbrlpld
Copy link
Collaborator Author

tbrlpld commented Jan 21, 2022

Ok, I am able to recreate locally with inv new-db. It appears the blog post generation is failing.

@tbrlpld
Copy link
Collaborator Author

tbrlpld commented Jan 21, 2022

@Pomax I think I finally got it.

The locale is only set when the page is first saved (cleaned really). But, the blog post save is overridden to add the related posts. Adding the related posts requires the locale. This is why we get the DoesNotExist exception on a field that is not allowed to be empty.

Only including the locale in the relation when it exists does not quite work, because all three spots will be filled with related posts potentially of other locales. This is the same problem that is being reported.

The ensure_related_posts seems like a validation step. We could probably move that to one of the cleaning methods. clean is already being used.

It also appears that the factory it not expecting for the save method to add the related posts, because the method is explicitly called before the save method is called.

https://github.com/mozilla/foundation.mozilla.org/blob/main/network-api/networkapi/wagtailpages/factory/blog.py#L132-L134

for post in BlogPage.objects.all():
        post.ensure_related_posts()
        post.save()

I have just tested it locally and that seems to work.

Before, when creating the blog post with the factory, there would be the
problem that ensuring the related blog posts required the locale to be
set. Wagtail only sets the local during the full_clean step for a page.
The factory does not use the full_clean method, but does call the save
method. This means when we try to ensure the related posts, the local
access would fail, because this non-nullable field would be null,
raising an exception.

To fix this issue, I moved the ensure_related_posts call from the save
method to the clean method. This still works as expected when creating
pages through the Wagtail admin, because it runs the clean method before
saving the page.

This now also allows the factory to successfully create the page. On
first save, the related posts will not be set. We need to make sure that
the factory ensures that the related posts have been set.

This is not a problem, because that is how the blog factory generate
function already worked.

We could potentially optimize this with the post-generation hooks that
factory boy provides: https://factoryboy.readthedocs.io/en/stable/reference.html#post-generation-hooks
@mofodevops mofodevops temporarily deployed to foundation-s-tibor-7991-100pra January 21, 2022 21:18 Inactive
@tbrlpld tbrlpld requested a review from Pomax January 21, 2022 21:49
Copy link
Contributor

@Pomax Pomax left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

such a good find.

@Pomax Pomax temporarily deployed to foundation-s-tibor-7991-100pra January 24, 2022 18:31 Inactive
@Pomax Pomax merged commit 888b859 into main Jan 24, 2022
@Pomax Pomax deleted the tibor/7991-related-posts-locale branch January 24, 2022 19:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Autofilled related posts are picking translated pages
3 participants