Refresh Epic: Wagtail in Bedrock

[stevejalim]

Wagtail in Bedrock

This epic outlines the work needed to integrate Wagtail into Bedrock in a way that's compliant with our RRA and generally effective.

## Necessary tasks

These will be split out into individual Issues as we prep them

• Move from django_jinja to core Django Jinja2 support - this will make it easer to integrate Wagtail Decision: stick with django-jinja, because its feature set is better
• Refactor i18n mechanism to gain compatibility with wagtail-localize Refactor legacy, custom i18n mechanism to use core django i18n #14256
• 🚧 Integrate Wagtail into Bedrock
  • Groundwork Wagtail in Bedrock: main groundwork #14250
  • Ensure SessionMiddleware doesn't get run in the main Web deployment - it must remain 100% stateless
  • Add a simple migration to change text content of the root page, to avoid user confusion Issue here
  • Disable the Wagtail Userbar extension for jinja #14657
  • CSP tuning for Wagtail admin
    • Support different CSP policies for different URL paths #14576
  • Support pre-generating image renditions
    • Core behaviour Support pre-generating renditions #14578
    • Add django-rq support so we can run background tasks #14601
    • move rendition pre-generation to a background task via queue worker
    • Disable rendition generation if doesn't exist already - catch the request early in middleware on the web deployment invalid concern
    • add workers in k8s to proccess the queue
      • Dev: https://github.com/mozilla-it/webservices-infra/pull/2138
      • Stage: TODO after Dev proven working
      • Prod: TODO after Dev proven working
    • Support scheduled publishing
• Tooling to support local and community dev easier
  • Support filling postgres from sqlite and dumping postgres to sqlite #14541
    • Minorly refactor ProductRelease.notes field #14574 (blocks the above)
  • Add tooling to give local developers an easy way to get appropriate CMS/Bedrock data to work with #14659
  • Add a test that checks for missing migrations #14665
• Plan changes needed to bedrock's startup/management scripts
  • Update the database-upload script to generate a new DB export (from postgres) #14660
• Integrate Bedrock's Wagtail with translation vendor
  • Custom work agreed and started
  • Integrate and evaluate wagtail-localize-smartling #14774
  • Add a scheduled task to sync Smartling translations #14776
  • Set up new Smartling project credentials for CMS-backed production #14802
  • Integration hooked in to Bedrock and working end-to-end
• Risk assessment
  • Request inital RRA for updated Bedrock + Wagtail service
  • Act on RRA and adjust plan accordingly
  • Request follow-up RRA ahead of go-live and respond accordingly
• SSO
  • Request SSO credentials for Wagtail in Bedrock -> https://mozilla-hub.atlassian.net/browse/IAM-1350
  • Add SSO support locally
    • Ensure sqlite DB export definitely excludes OIDC tables
  • Apply credentials to k8s, ready for use
  • Add convenience links to Wagtail Admin to link to Django admin and RQ queues #14650
• Infra update enable secure and stable editing of content for Bedrock
  • Confirm and document infra changes
  • JIRA tickets raised (parent epic here)
  • Only expose REDIS_URL to Editing Deployment, not Web Deployment
  • Infra updates started on
    • Dev
    • Stage
    • Prod
• Infra fully updated on
  • Dev
  • Stage
  • Prod
• Update from 5.2.x LTS to 6.1.x ahead of next LTS
• Content management and access planning (Will move to own Epic)

Rollout plan

• Use an SSH tunnel to connect to the relevant postgres DB as the appropriate user and run the pre-fill-postgres script

  • Done for Dev
  • Done for Stage
  • Done for Prod

• Stand up Editing deployment, connected to postgres

• Set MEDIA_URL in infra and also align with the CDN Add helper to get only the hostname out of MEDIA_URL #14723

• CMS mode tuning:

  • Backport K8S config changes so that we capture custom-tuned values for workers and blocking threads
  • Expand CSP for the editing deployment, but only on the editing deployment
  • Add custom CSP for Wagtail that allows just-uploaded images to be shown #14733

• Add new command to keep the editing deployment's DB up to date #14715

  • Update healthz-cron page re postgres backend #14737
  • https://github.com/mozilla-it/webservices-infra/pull/2232

• Once we're happy that the editing deployment is consuming data properly

  • Switch web pods to use the readonly postgres DB replicas and GCS storage bucket
    • Initially, add DATABASE_URL and MEDIA_URL via secrets
    • GRANT pg_read_all_data TO <BEDROCK READONLY USER NAME>;
    • once stable, move MEDIA_URL to env vars and out of secrets
  • Disable script to download new sqlite DB from the bucket if postgres is in play (this is now redundant for Dev/Stage/Prod, but still relevant to Bedrock-test and Demos)

• Add tooling to allow devs to download a sqlite version of the PG database

Final push tasks (ordered in priority terms)

Beta Give feedback

1. Add option to disable case-changing of language codes wagtail-localize-smartling#18
   +0
   
2. Update the postgres-to-sqlite-export script to be a lot more cautious #14937
   +0
   
3. No indication in UI of pages that have had new, unpublished translations land? wagtail-localize-smartling#19
   +0
4. Fine tune wagtail-localize fields to exclude slug from translated fields
5. Add signal handler to alert when translations land, so they can be published #14934
   L10N Wagtail +2
6. Stage Web deployment to use postgres
7. Prod CMS mode
8. Update the database-upload script to generate a new DB export (from postgres) #14660
   Backend +1
9. Prod Web deployment to use postgres
10. Add tooling to give local developers an easy way to get appropriate CMS/Bedrock data to work with #14659
    Backend Wagtail +2
    

Post-MVP CMS tasks

Beta Give feedback

1. Support CAT tool in Smartling, including support for draft/unpublished pages if possible #14938
   L10N Needs Info WIP 🚧 Wagtail +4
   
2. Support automatic linting of translated content #14948
   L10N Needs Info Wagtail +3