Skip to content

Indexing dashboard: add full-width Completed Indexing Jobs panel#4861

Merged
habdelra merged 2 commits into
mainfrom
worktree-grafana-completed-indexing-panel
May 18, 2026
Merged

Indexing dashboard: add full-width Completed Indexing Jobs panel#4861
habdelra merged 2 commits into
mainfrom
worktree-grafana-completed-indexing-panel

Conversation

@habdelra
Copy link
Copy Markdown
Contributor

Summary

  • Adds a full-width Completed Indexing Jobs panel to the Indexing dashboard, directly below Queued Indexing Jobs.
  • Lists indexing jobs (from-scratch-index, incremental-index, full-reindex) that finished in the past 24 hours, sorted newest first. Columns: id, job_type, realm, status, created_at, started_at, finished_at, wait_seconds, run_seconds, worker_id (linked to logs).
  • Shifts the two "Longest ... jobs (24h)" panels down (y 54 → 64) to make room — no other layout changes.

Test plan

  • pnpm --filter @cardstack/observability lint is green (already verified locally).
  • Apply to staging and confirm the new panel renders, sorts by finished_at desc, and the worker_id "View logs" link opens the correct logs panel.
  • Confirm Longest from-scratch-index jobs (24h) and Longest incremental-index jobs (24h) still render side-by-side directly underneath.

🤖 Generated with Claude Code

Adds a panel below "Queued Indexing Jobs" listing indexing jobs
(from-scratch-index, incremental-index, full-reindex) finished in the
past 24 hours, newest first — with queue time, run time, and a per-row
link to the worker logs. Shifts the two "Longest ... jobs (24h)" panels
down to make room.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@habdelra habdelra requested a review from Copilot May 18, 2026 16:59
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 18, 2026

Observability diff (vs staging)

Show diff
diff --git a/tmp/remote-canon.PblBgR/dashboards/boxel-status/indexing.json b/tmp/committed-canon.MOJFJt/dashboards/boxel-status/indexing.json
index 23fa4b3..5ed2858 100644
--- a/tmp/remote-canon.PblBgR/dashboards/boxel-status/indexing.json
+++ b/tmp/committed-canon.MOJFJt/dashboards/boxel-status/indexing.json
@@ -69,6 +69,10 @@
           "uid": "cef5v5sl9k7i8f"
         },
         "description": "System-wide operator action: queue a full reindex across every realm. The button disables itself while a `full-reindex` orchestration job is already pending or running. Per-realm reindex moved to the Realms dashboard. Click POSTs with `Authorization: Bearer ${grafana_secret}` (substituted from SSM at apply time, CS-10929).",
+        "fieldConfig": {
+          "defaults": {},
+          "overrides": []
+        },
         "gridPos": {
           "h": 8,
           "w": 24,
@@ -1095,6 +1099,155 @@
         "title": "Queued Indexing Jobs",
         "type": "table"
       },
+      {
+        "datasource": {
+          "type": "grafana-postgresql-datasource",
+          "uid": "cef5v5sl9k7i8f"
+        },
+        "description": "Indexing jobs (from-scratch-index, incremental-index, full-reindex) that finished in the past 24 hours, newest first. `wait_seconds` is queue time (created_at → first reservation). `run_seconds` is the duration of the attempt that completed it (latest reservation → finished_at). Force-cancelled jobs that never got a reservation appear with NULL started_at / wait_seconds / run_seconds / worker_id.",
+        "fieldConfig": {
+          "defaults": {
+            "color": {
+              "mode": "thresholds"
+            },
+            "custom": {
+              "align": "left",
+              "cellOptions": {
+                "type": "auto"
+              },
+              "filterable": true,
+              "inspect": false,
+              "minWidth": 100
+            },
+            "mappings": [],
+            "thresholds": {
+              "mode": "absolute",
+              "steps": [
+                {
+                  "color": "green"
+                },
+                {
+                  "color": "red",
+                  "value": 80
+                }
+              ]
+            }
+          },
+          "overrides": [
+            {
+              "matcher": {
+                "id": "byName",
+                "options": "wait_seconds"
+              },
+              "properties": [
+                {
+                  "id": "unit",
+                  "value": "s"
+                }
+              ]
+            },
+            {
+              "matcher": {
+                "id": "byName",
+                "options": "run_seconds"
+              },
+              "properties": [
+                {
+                  "id": "unit",
+                  "value": "s"
+                }
+              ]
+            },
+            {
+              "matcher": {
+                "id": "byName",
+                "options": "reservation_id"
+              },
+              "properties": [
+                {
+                  "id": "custom.hidden",
+                  "value": true
+                }
+              ]
+            },
+            {
+              "matcher": {
+                "id": "byName",
+                "options": "worker_id"
+              },
+              "properties": [
+                {
+                  "id": "links",
+                  "value": [
+                    {
+                      "targetBlank": true,
+                      "title": "View logs",
+                      "url": "/d/fetquzizsej28b?${__url_time_range}&var-job_id=${__data.fields.id}.${__data.fields.reservation_id}&orgId=1&viewPanel=3"
+                    }
+                  ]
+                },
+                {
+                  "id": "mappings",
+                  "value": [
+                    {
+                      "options": {
+                        "pattern": "^(.{6}).*$",
+                        "result": {
+                          "index": 0,
+                          "text": "View logs ($1)"
+                        }
+                      },
+                      "type": "regex"
+                    }
+                  ]
+                }
+              ]
+            }
+          ]
+        },
+        "gridPos": {
+          "h": 10,
+          "w": 24,
+          "x": 0,
+          "y": 54
+        },
+        "id": 22,
+        "options": {
+          "cellHeight": "sm",
+          "footer": {
+            "countRows": false,
+            "enablePagination": false,
+            "fields": "",
+            "reducer": [
+              "sum"
+            ],
+            "show": false
+          },
+          "showHeader": true,
+          "sortBy": [
+            {
+              "desc": true,
+              "displayName": "finished_at"
+            }
+          ]
+        },
+        "pluginVersion": "10.4.1",
+        "targets": [
+          {
+            "datasource": {
+              "type": "grafana-postgresql-datasource",
+              "uid": "cef5v5sl9k7i8f"
+            },
+            "editorMode": "code",
+            "format": "table",
+            "rawQuery": true,
+            "rawSql": "SELECT\n  j.id,\n  lr.reservation_id,\n  j.job_type,\n  CASE\n    WHEN j.job_type = 'full-reindex' THEN '(all realms)'\n    ELSE COALESCE(\n      NULLIF(RTRIM(REGEXP_REPLACE(COALESCE(j.args->>'realmURL',''), '^https?://[^/]+/', ''), '/'), ''),\n      REGEXP_REPLACE(COALESCE(j.args->>'realmURL',''), '^https?://([^./:]+).*$', '\\1')\n    )\n  END AS realm,\n  j.status,\n  j.created_at,\n  lr.started_at,\n  j.finished_at,\n  EXTRACT(EPOCH FROM (fr.first_started_at - j.created_at)) AS wait_seconds,\n  EXTRACT(EPOCH FROM (j.finished_at - lr.started_at)) AS run_seconds,\n  lr.worker_id\nFROM jobs j\nLEFT JOIN LATERAL (\n  SELECT jr.id AS reservation_id, jr.created_at AS started_at, jr.worker_id\n  FROM job_reservations jr\n  WHERE jr.job_id = j.id\n  ORDER BY jr.created_at DESC\n  LIMIT 1\n) lr ON TRUE\nLEFT JOIN LATERAL (\n  SELECT MIN(jr.created_at) AS first_started_at\n  FROM job_reservations jr\n  WHERE jr.job_id = j.id\n) fr ON TRUE\nWHERE j.job_type IN ('from-scratch-index','incremental-index','full-reindex')\n  AND j.finished_at IS NOT NULL\n  AND j.finished_at > NOW() - INTERVAL '24 hours'\nORDER BY j.finished_at DESC\nLIMIT 500;",
+            "refId": "A"
+          }
+        ],
+        "title": "Completed Indexing Jobs",
+        "type": "table"
+      },
       {
         "datasource": {
           "type": "grafana-postgresql-datasource",
@@ -1193,7 +1346,7 @@
           "h": 10,
           "w": 12,
           "x": 0,
-          "y": 54
+          "y": 64
         },
         "id": 20,
         "options": {
@@ -1330,7 +1483,7 @@
           "h": 10,
           "w": 12,
           "x": 12,
-          "y": 54
+          "y": 64
         },
         "id": 21,
         "options": {

(Run: https://github.com/cardstack/boxel/actions/runs/26048352196)

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new “Completed Indexing Jobs” table panel to the Indexing Grafana dashboard to make recently finished indexing work easier to inspect, and shifts the existing “Longest … jobs (24h)” panels down to make room.

Changes:

  • Add a full-width Completed Indexing Jobs table (last 24h, newest first) with wait/run duration columns and a worker log link.
  • Adjust layout (y positions) of the two “Longest … jobs (24h)” panels to appear below the new table.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread packages/observability/grafanactl/resources/dashboards/boxel-status/indexing.json Outdated
Comment thread packages/observability/grafanactl/resources/dashboards/boxel-status/indexing.json Outdated
- LEFT JOIN LATERAL for the latest reservation so force-cancelled
  pending jobs (`finished_at` set, no `job_reservations` row) still
  appear in the table.
- `wait_seconds` now computes from the FIRST reservation, not the
  latest, so retried jobs don't conflate prior attempt time with
  queue time. `run_seconds` still uses the latest reservation
  (the attempt that completed the job), matching the convention in
  the Longest-jobs panels.
- Description updated to reflect both changes and the NULL
  behaviour for cancelled-without-reservation rows.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@habdelra habdelra requested a review from a team May 18, 2026 17:21
@habdelra habdelra merged commit b1b7720 into main May 18, 2026
29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants