🐞 [Bug Report] - Out of Box 404 page

[grayside]

This is a bug report companion to #57. Where that issue is focused on an intermittent delay in functionality that we could provide better user feedback around, this issue points out a sustained blocker for demo usage.

Problem

When a new deployment is made to a project the frontend app deployed to Firebase Hosting is stuck in a 404 page.

In theory, this works fine if the user waits long enough before first trying to load the frontend. In practice, that wait appears to be up to a couple minutes and we don't expect folks to wait that long after being told they are ready to go.

Expand to see screenshot of 404 page

Current Workarounds

• Append an arbitrary querystring parameter, e.g. ?cachebuster=42.
• Use https://[project-id].firebaseapp.com instead of https://[project-id].web.app

Analysis

This blueprint uses a postdeployment mechanism to run firebase deploy via a GCE startup script.

sequenceDiagram
🏗️ Terraform-->>⚙️ GCE: Start running
⚙️ GCE-->>⚙️ GCE: Run Startup script
⚙️ GCE-->>👟 Cloud Run Jobs: Start Firebase Deployer
👟 Cloud Run Jobs-->>🔥 Firebase Hosting: Start deploy
🏗️ Terraform->> 👤 User: Apply complete!
👤 User->>🔥 Firebase Hosting: Request page<br/>Site not ready
Note right of 👤 User: 404 page stuck in cache
👟 Cloud Run Jobs-->>🔥 Firebase Hosting: Finish deploy
⚙️ GCE-->>⚙️ GCE: shutdown

Loading

Potential Fixes

We are still in the expansion/brainstorming phase on potential solutions.

1. GCE Startup Script Retries: Check for 404's and retry deploy job. Firebase Hosting redeploy clears cache
2. Big Change: Switch to an alternate post-deployment mechanism, such as using Terraform to define and trigger a Cloud Workflow

[grayside]

https://[project-id].web.app has 404
https://[project-id].firebaseapp.com loads fine.

[glasnt]

I suspect that because the https://[project-id].web.app version is encoded in Terraform, walkthroughs and other places, something in the process might be preloading this URL.

The https://[project-id].firebaseapp.app 'workaround' in this case may experience the same issues if it's added to the configurations that power this solution.

That aside, the asynchronous nature of the startup script on the gce is the main culprit. There's a ~2 min window between when Terraform says the process is complete and when it's actually complete.

A way to synchronously fire the Cloud Run jobs, and wait for them to complete, before Terraform returns, would be ideal. It need not be related to gce, that is just an implementation detail.

[grayside]

In #69 we're exploring how to get it fixed after the two minute window. It's less clear how to solve the 2 minute window.

Some further brainstorms I've had on this:

1. Split Firebase deploy from Cloud Run readiness: Have a separate startup for firebase deploy, and run that while Cloud Run server job & Cloud SQL are still being provisioned.
2. Duct-tape: Add a sleep to terraform, delaying terraform completion by most of two minutes so terraform doesn't report the app is ready before it's ready. This is a hacky prevention mechanism.
3. Offer a Script: At the end of terraform apply, give developers a command to run that will tell them when deploy is finished. This helps folks using terraform directly to know they need to wait longer.
4. Placeholder "Deploy in progress" site: Could we potentially deploy a static placeholder page to Firebase, announcing "Avocano: Coming Soon", and have that run concurrently to more of the infrastructure? This communicates to users that we know things aren't setup yet, and they don't need to worry about a surprise 404 page. Depends on whether the firebase deploy sequencing is likely to complete before terraform does.

(1) seems promising. While normally I'd prefer to have the API in place before the frontend, during the first-time setup does that matter? Especially if we follow-up the API deploy with the cache-purge implemented in #69.

(1), (2), and (4) would all require us to remove the test added in #69 confirming the frontend 404s when terraform apply first completes.

[grayside]

Dropping priority because #69 took care of the most egregious parts.

[glasnt]

Based on discussions:

The plan is to create a fast-follow 'placeholder' deployment to firebase.

Theory:

If we split firebase and cloud run's job execution (client and setup jobs, respectively), we can't make the client job any sooner in the execution order, as the firebase.json specifies the cloud run service, which itself depends on cloud sql, meaning it's one of the later things that could be run anyway.

By using a pre-built image (like we're currently doing) with no service/infra dependency (apart from the job and job executor workaround), we can quickly update from a "Site Not Found" to an avocano-specific "Deployment in progress". Similar to the Placeholder page for Cloud Run source-based deployments, the 'sad unicorn' page is overridden on successful code deployment. If it isn't, it includes information for debugging. Since we know there's more of a chance this page will be seen before things are complete, our messaging should be "in progress" rather than "broken".

Mockup of page:

This would be an extra job/job-executor, which (after confirming in the dependency graph from a terraform plan) should be launched and complete before TF finishes, giving a useful output should a user load the page between the TF done and Done done times.

New Firebase Hosting deployments purge the cache, so this placeholder should be overridden promptly when the second deploy is created.

We can additionally have logic that only deploys the placeholder if there's not already a deployment, such as demonstrated in hashicorp/terraform-provider-google#10351 (comment), as a future addition to this feature.