Create static CMS status page when website is down

[patphongs]

Summary

What we're after:
To better handle messaging to users while our website is down, we need to create a static CMS page that can be served up to inform users.

Related issues

See this issue for related design mock ups

• Create static fec.gov homepage when website is down #3694

Completion criteria

• Within the CMS, create an status page when the website is down
• The page should be accessible even when the Wagtail DB is down The site will fail if the Wagtail DB is down. So there is no way to show any error messaging if this happens.
• Add emergency page to the incident response documentation so that the team knows when to use it

Tech steps or considerations

• Create a separate emergency status template
• Add a feature flag for this template so that it can be turned on as needed
• Ensure that the page does not fail when the Wagtail DB is down The site will fail if the Wagtail DB is down. So there is no way to show any error messaging if this happens.

[patphongs]

@JonellaCulmer @lbeaufort I think we need to revisit when we want this page to display. I think the main purpose of the page is to display when our entire website is down. Down can have multiple scenarios:

Scenario 1

The database is experiencing a high amount of connections and will start maxing out connection pools. This will cause 500 error pages to appear across the website.

In this scenario, we can have a feature flag on the 500 error page to display the below screenshot instead of the normal 500 error message.

This is what the 500 error page looks like:

Scenario 2

The postgres database has completely failed and stop working. This means that absolutely no CMS page will come up, because pages are hooked up to the Wagtail DB to render.

In this scenario, we should do the same as Scenario 1. Turn on the feature flag for the 500 error page to show the emergency page.

Scenario 3

The CMS app itself is completely down. This means that the website will not function at all and will default to a 404 in the proxy like this:

404 Not Found: Requested route ('fec-dev-cms.app.cloud.gov') does not exist.

In this scenario, we must have an emergency page to failover to in a separate bucket and redirect with the proxy.

[JonellaCulmer]

@patphongs @lbeaufort If for the first scenario, other pages are available, but inconsistently, we should think of something else. Perhaps we stick with our existing 500 error pages which links to our status page while will provide additional details about what's happening.

[patphongs]

@lbeaufort I found out that if the CMS database is completely down, a rebuild will fail. So we will need to failover to the separate website bucket if this were ever to occur.

[patphongs]

@lbeaufort This is the proposed documentation 🔒 I would like to add to our incident response docs. Please take a look when you have a moment.

[patphongs]

Added documentation 🔒 on how to implement in case of website outage.