Indexing dashboard: add full-width Completed Indexing Jobs panel#4861
Merged
Conversation
Adds a panel below "Queued Indexing Jobs" listing indexing jobs (from-scratch-index, incremental-index, full-reindex) finished in the past 24 hours, newest first — with queue time, run time, and a per-row link to the worker logs. Shifts the two "Longest ... jobs (24h)" panels down to make room. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
Observability diff (vs staging)Show diffdiff --git a/tmp/remote-canon.PblBgR/dashboards/boxel-status/indexing.json b/tmp/committed-canon.MOJFJt/dashboards/boxel-status/indexing.json
index 23fa4b3..5ed2858 100644
--- a/tmp/remote-canon.PblBgR/dashboards/boxel-status/indexing.json
+++ b/tmp/committed-canon.MOJFJt/dashboards/boxel-status/indexing.json
@@ -69,6 +69,10 @@
"uid": "cef5v5sl9k7i8f"
},
"description": "System-wide operator action: queue a full reindex across every realm. The button disables itself while a `full-reindex` orchestration job is already pending or running. Per-realm reindex moved to the Realms dashboard. Click POSTs with `Authorization: Bearer ${grafana_secret}` (substituted from SSM at apply time, CS-10929).",
+ "fieldConfig": {
+ "defaults": {},
+ "overrides": []
+ },
"gridPos": {
"h": 8,
"w": 24,
@@ -1095,6 +1099,155 @@
"title": "Queued Indexing Jobs",
"type": "table"
},
+ {
+ "datasource": {
+ "type": "grafana-postgresql-datasource",
+ "uid": "cef5v5sl9k7i8f"
+ },
+ "description": "Indexing jobs (from-scratch-index, incremental-index, full-reindex) that finished in the past 24 hours, newest first. `wait_seconds` is queue time (created_at → first reservation). `run_seconds` is the duration of the attempt that completed it (latest reservation → finished_at). Force-cancelled jobs that never got a reservation appear with NULL started_at / wait_seconds / run_seconds / worker_id.",
+ "fieldConfig": {
+ "defaults": {
+ "color": {
+ "mode": "thresholds"
+ },
+ "custom": {
+ "align": "left",
+ "cellOptions": {
+ "type": "auto"
+ },
+ "filterable": true,
+ "inspect": false,
+ "minWidth": 100
+ },
+ "mappings": [],
+ "thresholds": {
+ "mode": "absolute",
+ "steps": [
+ {
+ "color": "green"
+ },
+ {
+ "color": "red",
+ "value": 80
+ }
+ ]
+ }
+ },
+ "overrides": [
+ {
+ "matcher": {
+ "id": "byName",
+ "options": "wait_seconds"
+ },
+ "properties": [
+ {
+ "id": "unit",
+ "value": "s"
+ }
+ ]
+ },
+ {
+ "matcher": {
+ "id": "byName",
+ "options": "run_seconds"
+ },
+ "properties": [
+ {
+ "id": "unit",
+ "value": "s"
+ }
+ ]
+ },
+ {
+ "matcher": {
+ "id": "byName",
+ "options": "reservation_id"
+ },
+ "properties": [
+ {
+ "id": "custom.hidden",
+ "value": true
+ }
+ ]
+ },
+ {
+ "matcher": {
+ "id": "byName",
+ "options": "worker_id"
+ },
+ "properties": [
+ {
+ "id": "links",
+ "value": [
+ {
+ "targetBlank": true,
+ "title": "View logs",
+ "url": "/d/fetquzizsej28b?${__url_time_range}&var-job_id=${__data.fields.id}.${__data.fields.reservation_id}&orgId=1&viewPanel=3"
+ }
+ ]
+ },
+ {
+ "id": "mappings",
+ "value": [
+ {
+ "options": {
+ "pattern": "^(.{6}).*$",
+ "result": {
+ "index": 0,
+ "text": "View logs ($1)"
+ }
+ },
+ "type": "regex"
+ }
+ ]
+ }
+ ]
+ }
+ ]
+ },
+ "gridPos": {
+ "h": 10,
+ "w": 24,
+ "x": 0,
+ "y": 54
+ },
+ "id": 22,
+ "options": {
+ "cellHeight": "sm",
+ "footer": {
+ "countRows": false,
+ "enablePagination": false,
+ "fields": "",
+ "reducer": [
+ "sum"
+ ],
+ "show": false
+ },
+ "showHeader": true,
+ "sortBy": [
+ {
+ "desc": true,
+ "displayName": "finished_at"
+ }
+ ]
+ },
+ "pluginVersion": "10.4.1",
+ "targets": [
+ {
+ "datasource": {
+ "type": "grafana-postgresql-datasource",
+ "uid": "cef5v5sl9k7i8f"
+ },
+ "editorMode": "code",
+ "format": "table",
+ "rawQuery": true,
+ "rawSql": "SELECT\n j.id,\n lr.reservation_id,\n j.job_type,\n CASE\n WHEN j.job_type = 'full-reindex' THEN '(all realms)'\n ELSE COALESCE(\n NULLIF(RTRIM(REGEXP_REPLACE(COALESCE(j.args->>'realmURL',''), '^https?://[^/]+/', ''), '/'), ''),\n REGEXP_REPLACE(COALESCE(j.args->>'realmURL',''), '^https?://([^./:]+).*$', '\\1')\n )\n END AS realm,\n j.status,\n j.created_at,\n lr.started_at,\n j.finished_at,\n EXTRACT(EPOCH FROM (fr.first_started_at - j.created_at)) AS wait_seconds,\n EXTRACT(EPOCH FROM (j.finished_at - lr.started_at)) AS run_seconds,\n lr.worker_id\nFROM jobs j\nLEFT JOIN LATERAL (\n SELECT jr.id AS reservation_id, jr.created_at AS started_at, jr.worker_id\n FROM job_reservations jr\n WHERE jr.job_id = j.id\n ORDER BY jr.created_at DESC\n LIMIT 1\n) lr ON TRUE\nLEFT JOIN LATERAL (\n SELECT MIN(jr.created_at) AS first_started_at\n FROM job_reservations jr\n WHERE jr.job_id = j.id\n) fr ON TRUE\nWHERE j.job_type IN ('from-scratch-index','incremental-index','full-reindex')\n AND j.finished_at IS NOT NULL\n AND j.finished_at > NOW() - INTERVAL '24 hours'\nORDER BY j.finished_at DESC\nLIMIT 500;",
+ "refId": "A"
+ }
+ ],
+ "title": "Completed Indexing Jobs",
+ "type": "table"
+ },
{
"datasource": {
"type": "grafana-postgresql-datasource",
@@ -1193,7 +1346,7 @@
"h": 10,
"w": 12,
"x": 0,
- "y": 54
+ "y": 64
},
"id": 20,
"options": {
@@ -1330,7 +1483,7 @@
"h": 10,
"w": 12,
"x": 12,
- "y": 54
+ "y": 64
},
"id": 21,
"options": {
(Run: https://github.com/cardstack/boxel/actions/runs/26048352196) |
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new “Completed Indexing Jobs” table panel to the Indexing Grafana dashboard to make recently finished indexing work easier to inspect, and shifts the existing “Longest … jobs (24h)” panels down to make room.
Changes:
- Add a full-width Completed Indexing Jobs table (last 24h, newest first) with wait/run duration columns and a worker log link.
- Adjust layout (
ypositions) of the two “Longest … jobs (24h)” panels to appear below the new table.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- LEFT JOIN LATERAL for the latest reservation so force-cancelled pending jobs (`finished_at` set, no `job_reservations` row) still appear in the table. - `wait_seconds` now computes from the FIRST reservation, not the latest, so retried jobs don't conflate prior attempt time with queue time. `run_seconds` still uses the latest reservation (the attempt that completed the job), matching the convention in the Longest-jobs panels. - Description updated to reflect both changes and the NULL behaviour for cancelled-without-reservation rows. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
lukemelia
approved these changes
May 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
from-scratch-index,incremental-index,full-reindex) that finished in the past 24 hours, sorted newest first. Columns: id, job_type, realm, status, created_at, started_at, finished_at, wait_seconds, run_seconds, worker_id (linked to logs).Test plan
pnpm --filter @cardstack/observability lintis green (already verified locally).finished_atdesc, and the worker_id "View logs" link opens the correct logs panel.Longest from-scratch-index jobs (24h)andLongest incremental-index jobs (24h)still render side-by-side directly underneath.🤖 Generated with Claude Code