Skip to content

feat(data-deletion): link the last dagster run from deletion requests - #70638

Merged
orian merged 4 commits into
masterfrom
pawel/feat/data-deletion-dagster-run-link
Jul 14, 2026
Merged

feat(data-deletion): link the last dagster run from deletion requests#70638
orian merged 4 commits into
masterfrom
pawel/feat/data-deletion-dagster-run-link

Conversation

@orian

@orian orian commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Problem

When a DataDeletionRequest is running (or has failed), there is no way to get from the request to the Dagster run that executed it.
The model already tracks attempt_count, first_executed_at and last_executed_at, so you know an attempt happened, but not which run did it.
Debugging then means hunting through the Dagster UI for the right run.

Changes

Adds last_dagster_run_id to DataDeletionRequest, set on every APPROVED → IN_PROGRESS transition (inside the existing select_for_update block in _record_execution_attempt, so it lands atomically with the status change and the attempt counter).
All three load ops pass it: event removal, property removal, person removal.
Each retry overwrites it, matching the "last" semantics of last_executed_at.

The admin renders it in the audit trail as a link to the run.
The URL is built at render time rather than stored, because a stored URL bakes in the environment:

  • DAGSTER_DOMAIN when set (the existing pattern from the Slack failure alerts)
  • otherwise derived from CLOUD_DEPLOYMENT, since the admin runs on web pods which don't carry DAGSTER_DOMAIN: USprod-us, EUprod-eu, DEVdev, producing https://posthog.dagster.cloud/<deployment>/runs/<run_id>
  • locally and self-hosted neither is set, so the bare run id is shown with no link

Migration 1254 is a nullable ADD COLUMN on posthog_datadeletionrequest, which is not a hot table and has no non-Django writers.

How did you test this code?

Automated, run by me (Claude):

  • posthog/dags/tests/test_data_deletion_requests.py — 57 passed. I extended the three existing load_* transition tests with one assertion each (request.last_dagster_run_id == context.run_id). That is one per load op, and each guards its own call site: the regression is a load op flipping a request to IN_PROGRESS without recording which run did it. No new test functions, since the existing ones already set up exactly this state.
  • posthog/admin/test_data_deletion_request_admin.py and posthog/models/test/test_data_deletion_request.py — 120 passed.
  • manage.py sqlmigrate shows a plain ALTER TABLE ... ADD COLUMN ... varchar(64) NULL, and makemigrations --check reports no drift.
  • I exercised the URL helper directly against all four settings combinations and confirmed the output matches the intended Dagster Cloud URL shape.

I did not click through the admin page in a browser, so the rendered link is unverified visually. Locally it would render as a bare run id anyway, since neither DAGSTER_DOMAIN nor CLOUD_DEPLOYMENT is set in dev.

Automatic notifications

  • Publish to changelog?
  • Alert Sales and Marketing teams?

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Written by Claude (Claude Code, Opus 4.8) working from a plan I reviewed and approved.
Skills invoked: /django-migrations and /writing-tests.

The main decision was whether to store the full run URL or just the run id.
We went with the id: a stored URL hardcodes the deployment into the row, and the run id is the durable fact.
That pushed the question of where the admin gets the deployment from, since DAGSTER_DOMAIN is only present on Dagster pods, not on the web pods serving the admin.
CLOUD_DEPLOYMENT is always set there, so the helper prefers DAGSTER_DOMAIN when available and otherwise maps CLOUD_DEPLOYMENT onto the Dagster Cloud deployment slug, falling back to plain text when neither is set.

@orian orian self-assigned this Jul 14, 2026
@github-actions

github-actions Bot commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Migration SQL Changes

Hey 👋, we've detected some migrations on this PR. Here's the SQL output for each migration, make sure they make sense:

posthog/migrations/1254_datadeletionrequest_last_dagster_run_id.py

BEGIN;
--
-- Add field last_dagster_run_id to datadeletionrequest
--
ALTER TABLE "posthog_datadeletionrequest" ADD COLUMN "last_dagster_run_id" varchar(255) NULL;
COMMIT;

Last updated: 2026-07-14 09:57 UTC (e105cc8)

@github-actions

github-actions Bot commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

🔍 Migration Risk Analysis

We've analyzed your migrations for potential risks.

Summary: 1 Safe | 0 Needs Review | 0 Blocked

✅ Safe

Brief or no lock, backwards compatible

posthog.1254_datadeletionrequest_last_dagster_run_id
  └─ #1 ✅ AddField
     Adding nullable field requires brief lock
     model: datadeletionrequest, field: last_dagster_run_id

📚 How to Deploy These Changes Safely

AddField:

This operation acquires a brief lock but doesn't rewrite the table.

Deployment uses lock timeouts with automatic retries, so lock contention will cause retries rather than connection pile-up.

Last updated: 2026-07-14 09:58 UTC (e105cc8)

@trunk-io

trunk-io Bot commented Jul 14, 2026

Copy link
Copy Markdown

Static BadgeStatic BadgeStatic BadgeStatic Badge

View Full Report ↗︎Docs

@orian
orian marked this pull request as ready for review July 14, 2026 09:12
Store the Dagster run id on every APPROVED -> IN_PROGRESS transition and render
it in the Django admin as a link to the run, so debugging an in-flight or failed
deletion request no longer means hunting for the run in the Dagster UI.

The url is built from the deployment rather than stored: DAGSTER_DOMAIN when set,
otherwise derived from CLOUD_DEPLOYMENT (web pods don't carry DAGSTER_DOMAIN).
@orian
orian force-pushed the pawel/feat/data-deletion-dagster-run-link branch from c63d86d to 2e23194 Compare July 14, 2026 09:14
@greptile-apps

greptile-apps Bot commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Reviews (1): Last reviewed commit: "feat(data-deletion): link the last dagst..." | Re-trigger Greptile

Comment thread posthog/admin/admins/data_deletion_request_admin.py Outdated
Comment thread posthog/models/data_deletion_request.py Outdated
@orian
orian marked this pull request as draft July 14, 2026 09:15
@github-actions

github-actions Bot commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

🤖 CI report

Playwright — all passed

All tests passed.

View test results →

Backend coverage — all changed backend lines covered

🧪 Backend test coverage

Patch coverage — changed backend lines (products + core): ████████████████████ 100.0% (43 / 43)

All changed backend lines are covered ✅

Per-product line coverage (touched products)
Product Coverage Lines
platform_features ██░░░░░░░░░░░░░░░░░░ 12.1% 7 / 58
batch_exports ████████░░░░░░░░░░░░ 39.7% 8,413 / 21,215
demo ███████████░░░░░░░░░ 55.2% 1,436 / 2,601
warehouse_sources_queue ████████████░░░░░░░░ 59.2% 148 / 250
tasks █████████████░░░░░░░ 66.8% 24,736 / 37,026
data_tools ██████████████░░░░░░ 70.0% 63 / 90
ai_gateway ███████████████░░░░░ 75.0% 9 / 12
data_modeling ████████████████░░░░ 78.4% 4,696 / 5,987
signals ████████████████░░░░ 78.5% 18,105 / 23,059
cdp ████████████████░░░░ 80.6% 3,105 / 3,851
wizard ████████████████░░░░ 82.5% 772 / 936
cohorts █████████████████░░░ 82.9% 3,048 / 3,675
notebooks █████████████████░░░ 83.8% 6,086 / 7,259
agent_platform █████████████████░░░ 84.1% 3,095 / 3,678
actions █████████████████░░░ 86.6% 717 / 828
product_tours █████████████████░░░ 87.5% 1,266 / 1,447
engineering_analytics ██████████████████░░ 87.6% 4,417 / 5,040
exports ██████████████████░░ 88.3% 6,857 / 7,763
visual_review ██████████████████░░ 88.5% 5,565 / 6,287
business_knowledge ██████████████████░░ 88.5% 4,400 / 4,969
conversations ██████████████████░░ 88.9% 15,924 / 17,921
mcp_analytics ██████████████████░░ 89.1% 2,485 / 2,790
dashboards ██████████████████░░ 89.1% 5,650 / 6,342
error_tracking ██████████████████░░ 89.2% 9,473 / 10,615
streamlit_apps ██████████████████░░ 90.4% 2,499 / 2,764
slack_app ██████████████████░░ 90.6% 9,460 / 10,444
links ██████████████████░░ 90.6% 183 / 202
marketing_analytics ██████████████████░░ 90.7% 11,476 / 12,646
product_analytics ██████████████████░░ 91.2% 5,652 / 6,195
managed_migrations ██████████████████░░ 91.9% 908 / 988
workflows ██████████████████░░ 92.0% 4,795 / 5,210
mcp_store ██████████████████░░ 92.1% 3,665 / 3,981
data_warehouse ██████████████████░░ 92.1% 17,281 / 18,764
alerts ██████████████████░░ 92.1% 3,389 / 3,678
web_analytics ███████████████████░ 92.7% 13,702 / 14,787
notifications ███████████████████░ 92.7% 1,026 / 1,107
ai_observability ███████████████████░ 92.7% 14,670 / 15,822
surveys ███████████████████░ 92.9% 5,660 / 6,094
posthog_ai ███████████████████░ 93.2% 1,311 / 1,407
tracing ███████████████████░ 93.2% 2,423 / 2,599
approvals ███████████████████░ 93.3% 3,395 / 3,640
reminders ███████████████████░ 93.4% 468 / 501
early_access_features ███████████████████░ 93.8% 848 / 904
legal_documents ███████████████████░ 94.1% 1,568 / 1,667
endpoints ███████████████████░ 94.1% 8,606 / 9,143
messaging ███████████████████░ 94.3% 2,366 / 2,508
skills ███████████████████░ 94.4% 2,827 / 2,995
revenue_analytics ███████████████████░ 94.4% 3,586 / 3,797
review_hog ███████████████████░ 94.5% 6,429 / 6,802
growth ███████████████████░ 94.9% 2,393 / 2,522
logs ███████████████████░ 95.3% 9,429 / 9,895
experiments ███████████████████░ 95.6% 24,017 / 25,124
replay_vision ███████████████████░ 95.6% 12,690 / 13,272
feature_flags ███████████████████░ 96.0% 14,600 / 15,203
warehouse_sources ███████████████████░ 96.1% 213,035 / 221,723
annotations ███████████████████░ 96.2% 732 / 761
user_interviews ███████████████████░ 96.4% 2,242 / 2,325
access_control ███████████████████░ 96.8% 849 / 877
customer_analytics ███████████████████░ 97.3% 7,277 / 7,481
data_catalog ███████████████████░ 97.3% 1,288 / 1,324
analytics_platform ████████████████████ 98.2% 2,098 / 2,137
metrics ████████████████████ 98.3% 2,363 / 2,403
live_debugger ████████████████████ 99.2% 613 / 618
field_notes ████████████████████ 99.4% 158 / 159

Report-only. Patch coverage = changed backend lines covered vs origin/master. Sorted lowest first.
Known gaps: lines covered only by Temporal tests show as uncovered; core line numbers may drift if master changed the same file.

orian and others added 3 commits July 14, 2026 11:29
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
The run id is written by the same save() that marks a request IN_PROGRESS, so an
over-long id would fail the deletion job over a field that only exists to help
debugging. varchar(255) removes that edge entirely.

Also covers dagster_run_url and the admin display method, including the
lowercase CLOUD_DEPLOYMENT case.
@orian
orian marked this pull request as ready for review July 14, 2026 09:52
@greptile-apps

greptile-apps Bot commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Reviews (2): Last reviewed commit: "fix(data-deletion): widen run id column ..." | Re-trigger Greptile

@orian
orian enabled auto-merge (squash) July 14, 2026 09:55
@orian
orian merged commit 1507f77 into master Jul 14, 2026
303 of 317 checks passed
@orian
orian deleted the pawel/feat/data-deletion-dagster-run-link branch July 14, 2026 10:12
@deployment-status-posthog

deployment-status-posthog Bot commented Jul 14, 2026

Copy link
Copy Markdown

Deploy status

Environment Status Deployed At Workflow
dev ✅ Deployed 2026-07-14 10:44 UTC Run
prod-us ✅ Deployed 2026-07-14 10:59 UTC Run
prod-eu ✅ Deployed 2026-07-14 11:02 UTC Run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants