From 6211084e7bf9a6821b7fea1edc4fedb692f4486c Mon Sep 17 00:00:00 2001 From: flakey5 <73616808+flakey5@users.noreply.github.com> Date: Mon, 20 Oct 2025 19:36:40 -0700 Subject: [PATCH 1/4] doc: incident report template, 10/20 incident report Add a incident report template and a incident report for the incident that happened on 10/20. Signed-off-by: flakey5 <73616808+flakey5@users.noreply.github.com> --- INCIDENT_REPORT_TEMPLATE.md | 28 ++++++++++++++++++++++++ incidents/2025-10-20.md | 43 +++++++++++++++++++++++++++++++++++++ 2 files changed, 71 insertions(+) create mode 100644 INCIDENT_REPORT_TEMPLATE.md create mode 100644 incidents/2025-10-20.md diff --git a/INCIDENT_REPORT_TEMPLATE.md b/INCIDENT_REPORT_TEMPLATE.md new file mode 100644 index 0000000..d4884a4 --- /dev/null +++ b/INCIDENT_REPORT_TEMPLATE.md @@ -0,0 +1,28 @@ +# YYYY-MM-DD Incident Report + + + +- Incident Commander: +- Severity Level: + + + +## Timeline + + + +## Impact + + + +## Root Cause + + + +## Fix + + + +## Follow-up Work + + diff --git a/incidents/2025-10-20.md b/incidents/2025-10-20.md new file mode 100644 index 0000000..1afdb5f --- /dev/null +++ b/incidents/2025-10-20.md @@ -0,0 +1,43 @@ +# 2025-10-20 Incident Report + +- Incident Commander: +- Severity Level: P2 + +For a brief period of time, the downloads page was blank due to a false assumption on their always being an active LTS version of Node. + +## Timeline + +- **2025-10-20 11:58 PM UTC**: Start of impact. v22.21.0 was promoted, updating [index.json](https://nodejs.org/index.json). + +- **2025-10-21 12:33 AM UTC**: Initial report of incident [nodejs/nodejs.org#8248](https://github.com/nodejs/nodejs.org/issues/8248) created. + +- **2025-10-21 12:55 AM UTC**: Initial report acknowledged in [OpenJS Slack](https://openjs-foundation.slack.com/archives/CVAMEJ4UV/p1761008123880979), investigation began. + +- **2025-10-21 01:09 AM UTC**: Root cause identified. + +- **2025-10-21 01:21 AM UTC**: [nodejs/nodejs.org#8251](https://github.com/nodejs/nodejs.org/pull/8251) opened to resolve issue. + +- **2025-10-21 01:30 AM UTC**: [nodejs/nodejs.org#8251](https://github.com/nodejs/nodejs.org/pull/8251) merged. Impact stopped shortly after. + +## Impact + +Users navigating to `https://nodejs.org/en/download` would see a blank page instead of the proper download instructions. + +`https://nodejs.org/en/download/current` was unaffected by this issue. + +## Root Cause + +It was assumed that there would always be an active LTS version of Node. +This was not the case with the release of v22.21.0, since v22 moved into maintenance LTS while v24 still had a week until it became active LTS. + +Since there was no active LTS release, the downloads page rendered nothing. + +## Fix + +The fix ([nodejs/nodejs.org#8251](https://github.com/nodejs/nodejs.org/pull/8251)) was it make the downloads page look for the active maintenance LTS version if an active LTS version didn't exist. + +## Follow-up Work + +- Better Playwright tests are needed + - We need to test the actual content of the pages instead of just testing for a successful status code. + - [nodejs/nodejs.org#8249](https://github.com/nodejs/nodejs.org/issues/8249) From 1b61589594b683fc26a02ca4a9dab822dd7c0f72 Mon Sep 17 00:00:00 2001 From: flakey5 <73616808+flakey5@users.noreply.github.com> Date: Mon, 20 Oct 2025 19:51:14 -0700 Subject: [PATCH 2/4] Update incidents/2025-10-20.md Co-authored-by: Brian Muenzenmeyer --- incidents/2025-10-20.md | 1 + 1 file changed, 1 insertion(+) diff --git a/incidents/2025-10-20.md b/incidents/2025-10-20.md index 1afdb5f..0a62046 100644 --- a/incidents/2025-10-20.md +++ b/incidents/2025-10-20.md @@ -38,6 +38,7 @@ The fix ([nodejs/nodejs.org#8251](https://github.com/nodejs/nodejs.org/pull/8251 ## Follow-up Work +- Better in-the-moment contemplation of use of status page - Better Playwright tests are needed - We need to test the actual content of the pages instead of just testing for a successful status code. - [nodejs/nodejs.org#8249](https://github.com/nodejs/nodejs.org/issues/8249) From 5bfde9b1a47869ee8b5010a75aa31f57a07c5a12 Mon Sep 17 00:00:00 2001 From: flakey5 <73616808+flakey5@users.noreply.github.com> Date: Tue, 21 Oct 2025 13:46:41 -0700 Subject: [PATCH 3/4] Apply suggestions from code review Co-authored-by: Aviv Keller --- incidents/2025-10-20.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/incidents/2025-10-20.md b/incidents/2025-10-20.md index 0a62046..8bd96d8 100644 --- a/incidents/2025-10-20.md +++ b/incidents/2025-10-20.md @@ -27,14 +27,14 @@ Users navigating to `https://nodejs.org/en/download` would see a blank page inst ## Root Cause -It was assumed that there would always be an active LTS version of Node. -This was not the case with the release of v22.21.0, since v22 moved into maintenance LTS while v24 still had a week until it became active LTS. +It was assumed that there would always be an active LTS version of Node.js. +This was not the case with the release of v22.21.0, since v22 moved into Maintenance LTS while v24 still had a week until it became Active LTS. Since there was no active LTS release, the downloads page rendered nothing. ## Fix -The fix ([nodejs/nodejs.org#8251](https://github.com/nodejs/nodejs.org/pull/8251)) was it make the downloads page look for the active maintenance LTS version if an active LTS version didn't exist. +The fix ([nodejs/nodejs.org#8251](https://github.com/nodejs/nodejs.org/pull/8251)) was to make the downloads page look for the Maintenance LTS version if the Active LTS version does not exist. ## Follow-up Work From b45facda3e558b220620965afa3049b364e316dc Mon Sep 17 00:00:00 2001 From: flakey5 <73616808+flakey5@users.noreply.github.com> Date: Tue, 21 Oct 2025 13:47:39 -0700 Subject: [PATCH 4/4] Update incidents/2025-10-20.md --- incidents/2025-10-20.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/incidents/2025-10-20.md b/incidents/2025-10-20.md index 8bd96d8..50e89f5 100644 --- a/incidents/2025-10-20.md +++ b/incidents/2025-10-20.md @@ -1,6 +1,6 @@ # 2025-10-20 Incident Report -- Incident Commander: +- Incident Commander: @bmuenzenmeyer - Severity Level: P2 For a brief period of time, the downloads page was blank due to a false assumption on their always being an active LTS version of Node.