Problem
When we deploy to dev, we copy prod data to dev so we can see how the new code and migrations run against a clone of production data. We have no automated way to verify that pages render the same (or intentionally differently) between prod and dev. Rendering regressions — broken pages, missing content, structural changes — can slip through to production undetected if we're not thorough enough with our manual testing.
Proposal
Build a script that compares rendering between production and dev after each dev deployment:
- Discover production tenants - we'll need a way to determine which tenant's sites to compare - we might as well only do this for production tenants.
- Collect URLs from per-tenant sitemaps (
pages-sitemap.xml, posts-sitemap.xml)
- Fetch each URL from both prod and dev
- Compare HTTP status codes and normalized HTML body content (this part might be tricky)
- Report results as a markdown summary + JSON artifact
Potential design decisions
- HTML normalization: Strip
<head>, <script>, <style>, <noscript>, <time> contents, class attributes (CSS module hashes differ per build), <img> src/srcset (blob prefixes differ), HTML comments, and collapse whitespace
- URL translation: Prod uses custom domains (
nwac.us), dev uses subdomains (nwac.dev.preview.avy-fx.org). Extract path from prod URL, construct dev URL with subdomain
- Non-blocking: Always exits 0. Report-only — developers review and decide if diffs are expected
- Dual-use: Runs as a post-deploy GitHub Actions job in
development.yaml and can be run locally via a package.json script
Problem
When we deploy to dev, we copy prod data to dev so we can see how the new code and migrations run against a clone of production data. We have no automated way to verify that pages render the same (or intentionally differently) between prod and dev. Rendering regressions — broken pages, missing content, structural changes — can slip through to production undetected if we're not thorough enough with our manual testing.
Proposal
Build a script that compares rendering between production and dev after each dev deployment:
pages-sitemap.xml,posts-sitemap.xml)Potential design decisions
<head>,<script>,<style>,<noscript>,<time>contents,classattributes (CSS module hashes differ per build),<img>src/srcset (blob prefixes differ), HTML comments, and collapse whitespacenwac.us), dev uses subdomains (nwac.dev.preview.avy-fx.org). Extract path from prod URL, construct dev URL with subdomaindevelopment.yamland can be run locally via a package.json script