fix(reports): enforce dashboard readiness and execution budget - #42624
fix(reports): enforce dashboard readiness and execution budget#42624fitzee wants to merge 18 commits into
Conversation
✅ Deploy Preview for superset-docs-preview ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #42624 +/- ##
==========================================
- Coverage 65.43% 65.39% -0.04%
==========================================
Files 2810 2811 +1
Lines 159422 159769 +347
Branches 36382 36437 +55
==========================================
+ Hits 104312 104487 +175
- Misses 53068 53217 +149
- Partials 2042 2065 +23
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Griffen review (Claude + Codex dual pass)Verdict: Needs changes — full agreement between both reviewers on verdict; noting this is already in draft with an explicit "do not merge" from the author, so treat the below as pre-merge punch list rather than a blocker on the current state. Door classification: MixedCode-level revert is clean (no schema/migration). But two behaviors have one-way-door side effects during the live window: customer notifications missed on Celery soft-timeout can't be un-missed after the fact, and the stale-row recovery race (below) can write incorrect terminal state into the execution-log audit trail that a later code revert doesn't repair. HIGH — Celery soft-timeout now suppresses all customer notificationFor every REPORT-type schedule (dashboard screenshots and CSV/Excel/data exports) that hits the Celery soft time limit, the recipient now gets zero notification — MEDIUM — stale-WORKING-row recovery can race the original worker, not just itself
MEDIUM — budget/reserve config validated lazily, not at boot
MEDIUM — coverage gap on exactly the branches that decide over/under-aggressive terminationCodecov: MEDIUM (non-blocking) — undisclosed tiling-decision changeIn LOW
What's soundThe core architecture — one shared monotonic deadline instead of independent screenshot/Celery/working_timeout constants — is the right fix, backed by real staging evidence (dashboard 10, 52 charts, 300s+ captures) rather than a guessed constant. Reviewed via Claude (Griffen) + Codex dual pass, synthesized. Full internal writeup on file. |
Staging review evidence: dashboard 10Dashboard 10 has 52 charts and produced a 7,504px tiled report. Two staging
Budget reviewRun 1 required more than 300 seconds for capture alone. A 300-second Latency classificationThe ~30m/~60m late starts are separate queue, worker-capacity, beat, or Semantic-success reviewDelivery is pipeline success; it does not prove that every chart rendered. The first run's user-reported chunk error still needs a holder-state audit.
Until those counts are recovered from holder-state diagnostics, the evidence |
|
Addressed the Griffen review in
Focused suite: 255 passed. Changed-file pre-commit (including mypy, Ruff format/check, and pylint) passes. Mandatory all-files pre-commit was rerun; its failures remain unrelated base/environment issues already listed in the PR description. The PR remains draft while the new CI run is monitored. |
|
CI follow-up for |
…ecovery-budget-rebased # Conflicts: # superset/utils/screenshot_utils.py # superset/utils/webdriver.py # tests/unit_tests/utils/test_screenshot_utils.py # tests/unit_tests/utils/webdriver_test.py
|
Merged current master into this branch (covering Matt while he's out) — it was 5 relevant merges behind (#42273, #42153, #42120, #42118, plus a CroniterBadDateError fix). Conflicts were concentrated in
Verification: |
… runtime ceiling Two behavior-preserving adjustments to the execution-budget rollout so the upstream default changes as little existing behavior as possible: - Default ALERT_REPORTS_EXECUTION_BUDGET_SECONDS is now one hour, matching the historical effective ceiling (the ReportSchedule.working_timeout model default). Default installations keep today's maximum report runtime and gain only the clean-failure/readiness semantics; deployments with tighter SLAs lower the value. - The effective budget for a REPORT schedule is min(ALERT_REPORTS_EXECUTION_BUDGET_SECONDS, working_timeout), centralized in resolve_report_execution_budget_seconds() and used consistently by the Celery limit derivation, the execution deadline construction, and stale- WORKING detection (which previously implemented its own inline min). The per-schedule working_timeout field keeps its historical user-facing meaning as a cap instead of being silently ignored for reports. A working_timeout below the summed phase reserves is floored at the minimum viable budget (reserves + 30s working allowance) with a warning, so such schedules fail cleanly at their first phase check instead of erroring while constructing the execution context. Also adds the UPDATING.md entry for the semantics change and documents the infrastructure sizing rules (pod termination grace vs budget + hard grace; web-server per-request timeout bounds single chart requests, not the report). Co-Authored-By: Claude <noreply@anthropic.com>
|
Pushed
Tests: 5 new cases for the resolver (cap, no-raise above budget, None passthrough, floor-with-warning, Celery alignment at the cap); the recovery-bound test updated to exercise a deployment-tightened 900s budget explicitly rather than asserting the global default. |
…hot start Folds open PR apache#42661 into this branch (its standalone form patched code this branch restructures): take_tiled_screenshot() accepts the caller's screenshot_started_at so navigation/headstart/element-wait time counts against the non-report task budget, matching the clock _wait_for_charts_ready already uses. Report captures are unaffected -- their deadline starts at task start, which supersedes the anchor. Falls back to "now" when omitted. Co-Authored-By: Claude <noreply@anthropic.com>
The integration test still asserted the 900s draft default. The default budget now resolves to min(3600, working_timeout default 3600) = 3600 with a 30s hard grace; the 900/930 expectation is kept by explicitly setting working_timeout=900, which also exercises the capping path end to end through the scheduler. Co-Authored-By: Claude <noreply@anthropic.com>
The SoftTimeLimitExceeded handler in reports.execute is deliberately type-unconditional; this pins the alert path (metric, warning log, explicit FAILURE, re-raise) that the PR body describes as observability-only for alerts. Co-Authored-By: Claude <noreply@anthropic.com>
eschutho
left a comment
There was a problem hiding this comment.
Posting on Elizabeth's behalf — this is her PR reviewer agent. Forward any pushback to her and she'll loop me back in.
Left a few notes below — the first four are functional items worth a look before merge, the rest are optional. All line numbers verified against HEAD 9b84ce711b.
superset/commands/report/execute.py:1602-1631
ReportWorkingState.next()'s timeout-recovery branch calls update_report_schedule_and_log(ERROR, ...) unconditionally, with no "is this still the latest active execution" check and no DB-level locking (no SELECT FOR UPDATE, no unique constraint on ReportExecutionLog.uuid, no version column). Under genuinely concurrent transactions — e.g. two racing recovery invocations, or a recovery invocation racing a just-started new execution — this looks like it could still stomp a newer WORKING state back to ERROR. The narrower retry-on-DB-error path (persist_owned_report_execution_terminal_error) does have a real ownership check before it writes, but this timeout-recovery branch doesn't appear to share that guard.
WDYT — should this branch re-check ownership (uuid still the latest active execution) immediately before the write, the same way persist_owned_report_execution_terminal_error does, or is the in-process check considered sufficient given how narrow the race window is in practice?
superset/commands/report/execute.py:1636-1644 and superset/reports/models.py:388
The "refused duplicate" (non-timeout) branch always inserts a new ReportExecutionLog row via reuse_working_log=False, even when the replaying invocation's execution_id matches the currently-active WORKING row's uuid. Since ReportExecutionLog.uuid has no unique constraint, this can leave two rows sharing the same uuid (one WORKING, one ERROR), which seems to undercut the "one row per execution uuid" invariant implied elsewhere in this change. test_report_schedule_same_execution_replay_stays_working exercises this exact scenario but only asserts row counts/states, not uuid uniqueness.
Could we add a uuid-uniqueness assertion to that test (or a DB constraint) to lock this invariant in, or is the uuid collision here considered harmless for downstream audit queries?
superset/commands/report/execute.py:1813-1827
persist_owned_report_execution_terminal_error's retry-on-DB-error safety net is gated on owns_report_working_state, computed once at invocation start as last_state != WORKING. A stale-recovery invocation (one that finds the schedule already WORKING on entry) never has owns_report_working_state=True, so if its own terminal write hits a transient DB error, it looks like there's no retry — the schedule could stay stuck in WORKING until the next scheduled run's timeout check. That's a narrower version of the exact staging bug this PR set out to fix.
Would it be worth extending the retry safety net to cover the stale-recovery invocation's own write too, or is that intentionally deferred to the next scheduled run by design?
superset/tasks/scheduler.py:114-148
The new except SoftTimeLimitExceeded: handler on the shared execute() Celery task (incrementing reports.execute.celery_soft_timeout, calling self.update_state(state="FAILURE")) fires for both ALERT and REPORT schedules, since it wraps the whole AsyncExecuteReportScheduleCommand.run() call unconditionally. The PR description and config.py comments frame the soft-timeout metric/state change as report-specific, but this looks like it now applies to alerts too — pre-PR, an alert soft-timeout had no such handler and fell through to Celery's generic failure signal. The numeric soft/hard limits for alerts are untouched, just these side effects.
Could we either scope the handler to report-type schedules only, or update the docs/description to note this is a shared behavior change, and add a test that exercises an alert schedule through this path?
tests/unit_tests/commands/report/execute_test.py (test_working_timeout_replay_promotes_original_execution_without_duplicate_log, test_new_report_execution_does_not_deliver_during_stale_recovery)
These two mock out update_report_schedule_and_log entirely — the function that actually performs the promotion/mutation being tested — so they only prove next() calls it once, not that the promotion/non-mutation behavior itself is correct. The integration-level tests do exercise the real path, so this is more a redundancy/clarity nit than a coverage gap.
Totally optional — could rename these to reflect what they actually assert, or drop them if the integration coverage is considered sufficient on its own?
superset/utils/screenshot_utils.py (take_tiled_screenshot, tile-combine-failure path)
When allow_partial_fallback=False and combine fails, the raised exception is caught by the generic except Exception at the bottom of the function and logged via logger.exception("Tiled screenshot failed...") before converting to None / re-raising as PlaywrightTimeout upstream. Since this is an intentional reject (no-partial-fallback-for-reports), the logger.exception call will read as a spurious error in logs/alerting even though nothing unexpected happened.
Small suggestion — could this specific case log at warning/info instead, to avoid noise in error-rate alerting?
PR description headline mentions a "900-second" default budget, but the shipped default (ALERT_REPORTS_EXECUTION_BUDGET_SECONDS) is 3600s/1hr, consistent with UPDATING.md and the docs. 900 now only shows up as an explicit per-schedule override example.
Nit, not required — just flagging in case the description gets used as reference documentation later.
Also noticed the PR is currently open and not marked draft on GitHub, but the description's closing line still says "This PR remains intentionally draft... Do not merge or deploy it from CI." Worth a quick pass to reconcile the description with the actual PR state before merge.
…ed renames - Apply auto-walrus rewrite in resolve_report_execution_budget_seconds (pre-commit hook failure on CI). - Persist the working_timeout override in the scheduler budget test via a query-level UPDATE: the attribute write on the fixture object was not flushed in CI (all three DB backends still derived limits from 3600), so phase two asserted against the default. - Rename the two delegation-only unit tests to reflect what they assert (they mock update_report_schedule_and_log; the real promotion path is covered by integration tests), per review feedback. Co-Authored-By: Claude <noreply@anthropic.com>
|
Thanks for the thorough pass — responses per item. Verification and fixes are on head 1. Timeout-recovery branch lacks an ownership re-check ( 2. Refused-duplicate rows can share a uuid ( 3. Retry net doesn't cover the stale-recovery invocation's own write ( 4. 5. Delegation-only unit tests 6. 7. "900-second" headline / 8. stale draft closing line CI status: your suspicion in the earlier summary was correct — the run on the reviewed head failed exactly on |
| if report_execution_context: | ||
| phase_timeout( | ||
| "screenshot_capture", | ||
| None, | ||
| report_execution_context.post_capture_reserve_seconds, | ||
| ) | ||
| img = element.screenshot_as_png |
There was a problem hiding this comment.
Suggestion: The Selenium capture phase only calls phase_timeout to validate that some budget remains, but does not apply that timeout to element.screenshot_as_png. Selenium's screenshot command has no timeout inherited from set_page_load_timeout, so it can block past the report deadline and consume the delivery/cleanup reserve. Use a bounded capture mechanism or enforce the deadline around the screenshot operation. [logic error]
Severity Level: Major ⚠️
- ❌ Selenium-backed reports can exceed their configured execution deadline.
- ⚠️ Blocked capture can consume delivery and cleanup reserves.
- ⚠️ Celery may terminate workers before report failure cleanup completes.(Use Cmd/Ctrl + Click for best experience)
Prompt for AI Agent 🤖
This is a comment left during a code review.
**Path:** superset/utils/webdriver.py
**Line:** 1455:1461
**Comment:**
*Logic Error: The Selenium capture phase only calls `phase_timeout` to validate that some budget remains, but does not apply that timeout to `element.screenshot_as_png`. Selenium's screenshot command has no timeout inherited from `set_page_load_timeout`, so it can block past the report deadline and consume the delivery/cleanup reserve. Use a bounded capture mechanism or enforce the deadline around the screenshot operation.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fixThere was a problem hiding this comment.
Real limitation, acknowledged — Selenium's synchronous screenshot command cannot take a timeout, so it isn't cooperatively bounded (this predates the PR; Selenium capture has never been bounded). The layered design covers it: the phase_timeout gate stops a capture from starting without budget, and if the capture itself blocks past the deadline, the Celery soft limit (aligned to the budget) interrupts the task and the command-boundary retry persists the terminal ERROR inside the 30s hard grace — so the 'terminates before cleanup completes' consequence doesn't hold. Bounding the blocking WebDriver HTTP call would need thread/signal wrappers, which we're deliberately not adding here; Playwright (the primary driver, and the only tiled path) passes an explicit capture timeout. Note the Bito reply below quotes the Playwright tiled path as the resolution — different driver; that code is already bounded.
There was a problem hiding this comment.
✅ Customized review instruction saved!
Instruction:
For Selenium screenshot capture, use a phase-timeout gate before starting the synchronous screenshot call; do not require additional thread/signal wrappers to bound the WebDriver call, as Celery soft limits and command-boundary retry handle captures that block past the deadline.
Applied to:
superset/utils/webdriver.py
💡 To manage or update this instruction, visit: CodeAnt AI Settings
| # Celery's hard limit leaves this additional window for terminal cleanup after | ||
| # the 15-minute soft limit. ALERT_REPORTS_WORKING_TIME_OUT_KILL controls these | ||
| # Celery limits; disabling it does not disable the application deadline above. | ||
| ALERT_REPORTS_EXECUTION_HARD_TIMEOUT_GRACE_SECONDS = 30 |
There was a problem hiding this comment.
Suggestion: This comment says the hard-limit grace follows a 15-minute soft limit, but the documented default execution budget and resulting soft limit are 3600 seconds (one hour). This gives operators an incorrect timeout expectation; describe the grace as following the resolved execution budget instead of naming 15 minutes. [comment mismatch]
Severity Level: Minor 🧹
- ⚠️ Operators receive incorrect Celery timeout guidance.
- ⚠️ Deployment timeout sizing may use the wrong soft-limit assumption.(Use Cmd/Ctrl + Click for best experience)
Prompt for AI Agent 🤖
This is a comment left during a code review.
**Path:** docs/admin_docs/configuration/alerts-reports.mdx
**Line:** 262:265
**Comment:**
*Comment Mismatch: This comment says the hard-limit grace follows a 15-minute soft limit, but the documented default execution budget and resulting soft limit are 3600 seconds (one hour). This gives operators an incorrect timeout expectation; describe the grace as following the resolved execution budget instead of naming 15 minutes.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fixThere was a problem hiding this comment.
Correct — stale leftover from the 900s draft. Fixed in ba7777d94a: the comment now describes the grace as following the resolved execution budget (configured budget capped by working_timeout) instead of naming 15 minutes.
| latest_working_log = ( | ||
| db.session.query(ReportExecutionLog) | ||
| .filter( | ||
| ReportExecutionLog.report_schedule_id == report_schedule_id, | ||
| ReportExecutionLog.state == ReportState.WORKING, | ||
| ReportExecutionLog.error_message.is_(None), | ||
| ) | ||
| .order_by(ReportExecutionLog.end_dttm.desc()) | ||
| .first() | ||
| ) | ||
| report_schedule = working_log.report_schedule | ||
| owns_schedule_state = ( | ||
| report_schedule.last_state == ReportState.WORKING | ||
| and latest_working_log is not None | ||
| and latest_working_log.uuid == execution_id | ||
| ) |
There was a problem hiding this comment.
Suggestion: The ownership check is a non-atomic read-modify-write: after latest_working_log and report_schedule.last_state are read, a newer execution can start and establish a different active WORKING row before this commit. The stale retry can then overwrite report_schedule.last_state with ERROR, incorrectly aborting the newer execution. Perform the ownership check and terminal update under a row lock or use a compare-and-swap update that verifies the active execution UUID. [race condition]
Severity Level: Major ⚠️
- ❌ Newer scheduled execution can be marked ERROR by stale worker.
- ⚠️ Schedule remains blocked or requires timeout recovery.
- ⚠️ Execution history can report an incorrect terminal owner.(Use Cmd/Ctrl + Click for best experience)
Prompt for AI Agent 🤖
This is a comment left during a code review.
**Path:** superset/commands/report/execute.py
**Line:** 198:213
**Comment:**
*Race Condition: The ownership check is a non-atomic read-modify-write: after `latest_working_log` and `report_schedule.last_state` are read, a newer execution can start and establish a different active WORKING row before this commit. The stale retry can then overwrite `report_schedule.last_state` with ERROR, incorrectly aborting the newer execution. Perform the ownership check and terminal update under a row lock or use a compare-and-swap update that verifies the active execution UUID.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fixThere was a problem hiding this comment.
Agreed the compare-then-commit is not atomic — this is the known TOCTOU family discussed in the human review above (items 1–2). Without a unique constraint on ReportExecutionLog.uuid or row locking, tightening the compare only narrows the window, and the schema/locking work (unique index + migration + backfill for existing duplicate uuids, guarded writes) is scoped as a tracked follow-up rather than folded into this PR. Worth noting the check that exists here is already strictly stronger than pre-PR master, which had no ownership compare at all on any terminal write path.
| self._phase_timeout( | ||
| "pdf_generation", | ||
| reserve_seconds=reserve_seconds, | ||
| ) | ||
| pdf = build_pdf_from_screenshots(screenshots) | ||
| self._phase_timeout( | ||
| "pdf_generation", | ||
| reserve_seconds=reserve_seconds, | ||
| ) |
There was a problem hiding this comment.
Suggestion: The deadline is checked only before and after the synchronous build_pdf_from_screenshots call, so PDF conversion itself has no cooperative timeout. A large or pathological screenshot set can consume the remaining budget and cleanup reserve before the second check runs, allowing the Celery hard limit to terminate the task before terminal state persistence. PDF generation needs a bounded operation or periodic deadline checks that preserve the cleanup window. [possible bug]
Severity Level: Major ⚠️
- ❌ Large PDF reports can exceed Celery hard limits.
- ⚠️ Terminal ERROR persistence may not execute.
- ⚠️ Timed-out executions can remain WORKING until recovery.(Use Cmd/Ctrl + Click for best experience)
Prompt for AI Agent 🤖
This is a comment left during a code review.
**Path:** superset/commands/report/execute.py
**Line:** 864:872
**Comment:**
*Possible Bug: The deadline is checked only before and after the synchronous `build_pdf_from_screenshots` call, so PDF conversion itself has no cooperative timeout. A large or pathological screenshot set can consume the remaining budget and cleanup reserve before the second check runs, allowing the Celery hard limit to terminate the task before terminal state persistence. PDF generation needs a bounded operation or periodic deadline checks that preserve the cleanup window.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fixThere was a problem hiding this comment.
By design: the deadline is cooperative at operation granularity, and build_pdf_from_screenshots is a synchronous CPU-bound call we intentionally don't interrupt internally. The backstop is the Celery soft limit, which equals the budget — if PDF generation blows through the remaining budget, SoftTimeLimitExceeded interrupts the task and the command boundary persists the terminal ERROR within the 30s hard grace. So 'terminal state persistence may not execute' doesn't hold: that persistence path is exactly what the soft-limit envelope exists for. Adding periodic deadline checks inside PIL-level PDF assembly isn't practical.
There was a problem hiding this comment.
✅ Customized review instruction saved!
Instruction:
Treat synchronous CPU-bound PDF generation as cooperative only at operation boundaries; do not require internal periodic deadline checks when the Celery soft-limit handler and command boundary provide terminal-state persistence.
Applied to:
superset/commands/report/execute.py
💡 To manage or update this instruction, visit: CodeAnt AI Settings
| log_report_delivery_phase( | ||
| report_context, | ||
| getattr(recipient, "type", None), | ||
| "start", | ||
| enforce_budget=True, | ||
| ) |
There was a problem hiding this comment.
Suggestion: The delivery gate reserves only cleanup_reserve_seconds; it does not reserve delivery_reserve_seconds, and notification.send() receives no deadline-derived timeout. A slow email, Slack, or webhook delivery can therefore consume the delivery and cleanup windows, leaving insufficient time to persist the terminal report state despite the shared budget. Reserve both delivery and cleanup capacity and pass a bounded timeout to notification implementations where supported. [possible bug]
Severity Level: Major ⚠️
- ❌ Slow notification can exhaust terminal cleanup time.
- ⚠️ Report ERROR or SUCCESS state may not persist.
- ⚠️ Sequential recipients amplify delivery overrun risk.(Use Cmd/Ctrl + Click for best experience)
Prompt for AI Agent 🤖
This is a comment left during a code review.
**Path:** superset/commands/report/execute.py
**Line:** 1322:1327
**Comment:**
*Possible Bug: The delivery gate reserves only `cleanup_reserve_seconds`; it does not reserve `delivery_reserve_seconds`, and `notification.send()` receives no deadline-derived timeout. A slow email, Slack, or webhook delivery can therefore consume the delivery and cleanup windows, leaving insufficient time to persist the terminal report state despite the shared budget. Reserve both delivery and cleanup capacity and pass a bounded timeout to notification implementations where supported.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fixThere was a problem hiding this comment.
This one misreads the reserve semantics: reserves protect later phases, so the delivery gate reserving only cleanup_reserve_seconds is correct — the delivery reserve is the capacity delivery itself is meant to spend (it's held back from the earlier readiness/capture phases, see readiness_reserve_seconds/post_capture_reserve_seconds). Reserving delivery capacity at the delivery gate would double-count it and starve delivery. An unbounded notification.send() is backstopped the same way as capture: the per-recipient gate stops sends from starting without budget, and the Celery soft limit + command-boundary persistence covers a send that blocks.
There was a problem hiding this comment.
✅ Customized review instruction saved!
Instruction:
Do not flag the report delivery gate for reserving only cleanup capacity; delivery_reserve_seconds is the budget intended for delivery itself, while the gate protects later cleanup phases and should not double-count delivery capacity.
Applied to:
superset/commands/report/execute.py
💡 To manage or update this instruction, visit: CodeAnt AI Settings
| if budget < (min_viable := reserves_total + MIN_REPORT_EXECUTION_WORK_SECONDS): | ||
| logger.warning( | ||
| "Report working_timeout=%s is below the minimum viable execution " | ||
| "budget (%.0fs phase reserves + %.0fs working allowance); " | ||
| "flooring the effective budget at %.0fs.", | ||
| working_timeout, | ||
| reserves_total, | ||
| MIN_REPORT_EXECUTION_WORK_SECONDS, | ||
| min_viable, | ||
| ) | ||
| return min_viable |
There was a problem hiding this comment.
Suggestion: When a schedule's working_timeout is smaller than the phase reserves, this floors the effective budget above the user-configured limit. The resulting value is used for Celery's soft and hard limits and stale-working detection, so a schedule configured to stop after a short interval can continue for reserves_total + 30 seconds instead of honoring its configured timeout. Preserve the configured cap and fail the execution cleanly when it cannot accommodate the reserves. [logic error]
Severity Level: Major ⚠️
- ❌ Report schedules can exceed their configured working timeout.
- ⚠️ Celery soft and hard limits no longer honor short schedules.
- ⚠️ Stale WORKING recovery is delayed beyond user configuration.(Use Cmd/Ctrl + Click for best experience)
Prompt for AI Agent 🤖
This is a comment left during a code review.
**Path:** superset/utils/report_execution.py
**Line:** 86:96
**Comment:**
*Logic Error: When a schedule's `working_timeout` is smaller than the phase reserves, this floors the effective budget above the user-configured limit. The resulting value is used for Celery's soft and hard limits and stale-working detection, so a schedule configured to stop after a short interval can continue for `reserves_total + 30` seconds instead of honoring its configured timeout. Preserve the configured cap and fail the execution cleanly when it cannot accommodate the reserves.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fixThere was a problem hiding this comment.
Intentional and documented (UPDATING.md and the resolver docstring, with a runtime warning when the floor engages). The alternative — strictly honoring a sub-viable working_timeout — means every execution of that schedule fails deterministically, since the timeout can't fit capture + delivery + cleanup. Pre-PR, such a schedule was killed mid-capture every run and never delivered anything, so the floor (reserves + 30s ≈ 240s with defaults) converts a permanently-broken configuration into a working one while logging the discrepancy. A schedule owner who genuinely wants sub-4-minute kills has no deliverable report either way.
There was a problem hiding this comment.
✅ Customized review instruction saved!
Instruction:
Do not flag flooring sub-viable report working timeouts to the minimum execution budget; this is intentional to allow capture, delivery, and cleanup to complete, with a runtime warning documenting the discrepancy.
Applied to:
superset/utils/report_execution.py
💡 To manage or update this instruction, visit: CodeAnt AI Settings
|
The flagged issue is correct. The Selenium To resolve this, ensure the capture_timeout = (
_timeout_seconds(
"screenshot_capture",
reserve_seconds=(
report_execution_context.post_capture_reserve_seconds
if report_execution_context
else 0.0
),
)
if report_execution_context or task_budget is not None
else None
)
tile_screenshot = page.screenshot(
type="png",
clip=clip,
**(
{"timeout": capture_timeout * 1000}
if capture_timeout is not None
else {}
),
)Would you like me to fetch all other comments on this PR to validate and implement fixes for them as well? superset/utils/webdriver.py |
The hard-timeout grace comment predated the 3600s default; the soft limit is the resolved execution budget, not a fixed 15 minutes. Co-Authored-By: Claude <noreply@anthropic.com>
| elif report_execution_context: | ||
| readiness_predicate = REPORT_CHART_HOLDERS_READY_JS |
There was a problem hiding this comment.
Suggestion: The report readiness predicate only verifies chart holders intersecting the current viewport, but the subsequent standard screenshot captures the entire dashboard element, including below-the-fold content. As a result, lower charts can still be unmounted, loading, or blank when the full-page screenshot is delivered. Either ensure the full dashboard is progressively scrolled and each viewport is validated, or use the tiled capture path for reports that require full-dashboard readiness. [incorrect condition logic]
Severity Level: Major ⚠️
- ❌ Scheduled dashboard reports can omit lower charts.
- ⚠️ Standard captures may deliver partially rendered dashboards.
- ⚠️ Virtualized dashboard content is not validated before delivery.(Use Cmd/Ctrl + Click for best experience)
Prompt for AI Agent 🤖
This is a comment left during a code review.
**Path:** superset/utils/webdriver.py
**Line:** 445:446
**Comment:**
*Incorrect Condition Logic: The report readiness predicate only verifies chart holders intersecting the current viewport, but the subsequent standard screenshot captures the entire dashboard element, including below-the-fold content. As a result, lower charts can still be unmounted, loading, or blank when the full-page screenshot is delivered. Either ensure the full dashboard is progressively scrolled and each viewport is validated, or use the tiled capture path for reports that require full-dashboard readiness.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fixThere was a problem hiding this comment.
Real trade-off, deliberately carried over rather than introduced here — the viewport-scoped predicate came from #42153/#42253's production-safe readiness work, because requiring below-the-fold holders deadlocks on virtualized dashboards (off-screen holders never render by design). Full-dashboard readiness is exactly what the tiled path provides: it scrolls tile by tile and validates each viewport before capture, and the thresholds (SCREENSHOT_TILED_CHART_THRESHOLD, SCREENSHOT_TILED_HEIGHT_THRESHOLD) route chart-heavy/tall dashboards there. The residual window is a dashboard taller than the browser window but under both tiling thresholds; operators can close it by lowering SCREENSHOT_TILED_HEIGHT_THRESHOLD toward the viewport height. Auto-tiling whenever element height exceeds the viewport is a reasonable future tightening, but it changes capture behavior for a class of currently-working dashboards, so it's out of scope for this PR.
There was a problem hiding this comment.
✅ Customized review instruction saved!
Instruction:
Do not require below-the-fold chart holders for the standard screenshot readiness predicate, since virtualized dashboards may never render off-screen holders; rely on the tiled capture path for full-dashboard readiness.
Applied to:
superset/utils/webdriver.py
💡 To manage or update this instruction, visit: CodeAnt AI Settings
| return False | ||
|
|
||
| latest_working_log = ( | ||
| db.session.query(ReportExecutionLog) | ||
| .filter( | ||
| ReportExecutionLog.report_schedule_id == report_schedule_id, | ||
| ReportExecutionLog.state == ReportState.WORKING, | ||
| ReportExecutionLog.error_message.is_(None), | ||
| ) | ||
| .order_by(ReportExecutionLog.end_dttm.desc()) | ||
| .first() | ||
| ) | ||
| report_schedule = working_log.report_schedule | ||
| owns_schedule_state = ( | ||
| report_schedule.last_state == ReportState.WORKING | ||
| and latest_working_log is not None | ||
| and latest_working_log.uuid == execution_id | ||
| ) | ||
| ended_at = datetime.now(timezone.utc).replace(tzinfo=None) | ||
| working_log.state = ReportState.ERROR | ||
| working_log.error_message = error_message | ||
| working_log.end_dttm = ended_at | ||
| if owns_schedule_state: | ||
| report_schedule.last_state = ReportState.ERROR | ||
| report_schedule.last_eval_dttm = ended_at | ||
|
|
||
| db.session.commit() # pylint: disable=consider-using-transaction |
There was a problem hiding this comment.
Suggestion: Race condition: the latest WORKING log and report_schedule.last_state are read without a row lock or conditional update, then the schedule is committed later. A newer execution can become WORKING after this check but before the commit, allowing an older worker's retry to set the schedule to ERROR and overwrite the newer execution's state. Lock the schedule/latest log row or make the terminal update conditional on the execution UUID in the same transaction. [race condition]
Severity Level: Major ⚠️
- ❌ Newer report execution can lose WORKING schedule state.
- ⚠️ Stale recovery may run against the wrong execution.
- ⚠️ Concurrent report scheduling can produce inconsistent audit state.(Use Cmd/Ctrl + Click for best experience)
Prompt for AI Agent 🤖
This is a comment left during a code review.
**Path:** superset/commands/report/execute.py
**Line:** 196:222
**Comment:**
*Race Condition: Race condition: the latest WORKING log and `report_schedule.last_state` are read without a row lock or conditional update, then the schedule is committed later. A newer execution can become WORKING after this check but before the commit, allowing an older worker's retry to set the schedule to ERROR and overwrite the newer execution's state. Lock the schedule/latest log row or make the terminal update conditional on the execution UUID in the same transaction.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fixThere was a problem hiding this comment.
Same finding as the thread on line 213 (and items 1–2 of the human review above): agreed the compare-then-commit isn't atomic, and a real fix needs a unique constraint on ReportExecutionLog.uuid plus a guarded/locked write — schema work that's scoped as a tracked follow-up. The compare here is strictly stronger than pre-PR master, which wrote terminal state with no ownership check at all.
| total_seconds = resolve_report_execution_budget_seconds( | ||
| app.config, | ||
| working_timeout=self._model.working_timeout, | ||
| ) |
There was a problem hiding this comment.
Suggestion: The resolved budget no longer honors the schedule's configured working_timeout when that value is below the reserve floor. For example, a schedule with the valid configured timeout of one second is assigned at least the summed reserves plus 30 seconds, and the Celery limits and deadline therefore permit execution well beyond the owner's timeout. Either reject such configurations or preserve the per-schedule timeout as the effective cap instead of flooring it above that value. [api mismatch]
Severity Level: Major ⚠️
- ⚠️ REPORT schedules can exceed configured execution limits.
- ⚠️ Celery workers remain occupied beyond owner expectations.
- ⚠️ Slow reports delay subsequent scheduled executions.(Use Cmd/Ctrl + Click for best experience)
Prompt for AI Agent 🤖
This is a comment left during a code review.
**Path:** superset/commands/report/execute.py
**Line:** 1827:1830
**Comment:**
*Api Mismatch: The resolved budget no longer honors the schedule's configured `working_timeout` when that value is below the reserve floor. For example, a schedule with the valid configured timeout of one second is assigned at least the summed reserves plus 30 seconds, and the Celery limits and deadline therefore permit execution well beyond the owner's timeout. Either reject such configurations or preserve the per-schedule timeout as the effective cap instead of flooring it above that value.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fixThere was a problem hiding this comment.
Same as the resolver thread in report_execution.py (a review instruction was saved there): the floor is intentional and documented in UPDATING.md, with a runtime warning when it engages. Strictly honoring a sub-viable working_timeout (one below capture+delivery+cleanup reserves) means that schedule fails deterministically on every run — pre-PR it was killed mid-capture and never delivered anything. The floor converts a permanently-broken configuration into a working one; rejecting such configs at save time is a fair alternative but is an API/validation change beyond this PR.
There was a problem hiding this comment.
✅ Customized review instruction saved!
Instruction:
Preserve the minimum execution-budget floor when a configured working_timeout is below the capture, delivery, and cleanup reserves, and emit a runtime warning when the floor is applied; do not require strict adherence to sub-viable timeouts.
Applied to:
superset/commands/report/execute.py
💡 To manage or update this instruction, visit: CodeAnt AI Settings
| async_options = { | ||
| "eta": schedule, | ||
| **get_report_task_timeout_options( | ||
| is_report=active_schedule.type == ReportScheduleType.REPORT, | ||
| working_timeout=active_schedule.working_timeout, | ||
| config=current_app.config, | ||
| ), | ||
| } | ||
| execute.apply_async((active_schedule.id,), **async_options) |
There was a problem hiding this comment.
Suggestion: Passing every report's working_timeout through get_report_task_timeout_options can produce a Celery timeout longer than the schedule's configured cap: the helper floors values below the reserve-plus-working minimum instead of preserving the requested cap. For example, a report configured with a 10-second working_timeout receives a 210-second soft limit, so stalled executions can run well past the user-configured timeout. The report timeout calculation must never exceed the schedule's working_timeout. [logic error]
Severity Level: Major ⚠️
- ❌ Stalled scheduled reports exceed configured execution limits.
- ⚠️ Celery workers remain occupied beyond schedule settings.
- ⚠️ Manual report execution shares the same timeout mismatch.(Use Cmd/Ctrl + Click for best experience)
Prompt for AI Agent 🤖
This is a comment left during a code review.
**Path:** superset/tasks/scheduler.py
**Line:** 103:111
**Comment:**
*Logic Error: Passing every report's `working_timeout` through `get_report_task_timeout_options` can produce a Celery timeout longer than the schedule's configured cap: the helper floors values below the reserve-plus-working minimum instead of preserving the requested cap. For example, a report configured with a 10-second `working_timeout` receives a 210-second soft limit, so stalled executions can run well past the user-configured timeout. The report timeout calculation must never exceed the schedule's `working_timeout`.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fixThere was a problem hiding this comment.
Duplicate of the resolver-floor finding (see the report_execution.py thread) — same intentional behavior, same UPDATING.md documentation, evaluated once in resolve_report_execution_budget_seconds and consumed consistently by both the Celery limit derivation here and the in-process deadline, so the two never disagree.
There was a problem hiding this comment.
✅ Customized review instruction saved!
Instruction:
Do not flag the resolver-floor behavior in report execution timeout calculations; it is intentional, documented, and consistently applied to both Celery limits and in-process deadlines.
Applied to:
superset/tasks/scheduler.py
💡 To manage or update this instruction, visit: CodeAnt AI Settings
| # derived from a fresh remaining value instead of a stale one | ||
| # that would let each tile overrun the budget by up to one settle | ||
| # interval (_timeout_seconds also recomputes at call time). | ||
| _raise_if_budget_exhausted() |
There was a problem hiding this comment.
Suggestion: The budget check occurs only after the fixed page.wait_for_timeout(SCROLL_SETTLE_TIMEOUT_MS) sleep, so a report with less than one second remaining can exceed its monotonic deadline before this check runs. Cap the scroll-settle wait using the remaining deadline, or check and fail before sleeping when the remaining time is shorter than the settle interval. [possible bug]
Severity Level: Major ⚠️
- ❌ Near-deadline tiled reports exceed their authoritative execution budget.
- ⚠️ Up to one second of capture time consumes reserved phases.
- ⚠️ Large dashboards repeat this delay once per tile.(Use Cmd/Ctrl + Click for best experience)
Prompt for AI Agent 🤖
This is a comment left during a code review.
**Path:** superset/utils/screenshot_utils.py
**Line:** 523:523
**Comment:**
*Possible Bug: The budget check occurs only after the fixed `page.wait_for_timeout(SCROLL_SETTLE_TIMEOUT_MS)` sleep, so a report with less than one second remaining can exceed its monotonic deadline before this check runs. Cap the scroll-settle wait using the remaining deadline, or check and fail before sleeping when the remaining time is shorter than the settle interval.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fixThere was a problem hiding this comment.
The premise isn't quite right: there's a budget check immediately before the scroll (_raise_if_budget_exhausted() above the scrollTo), and a second check immediately after the settle sleep — added precisely so the per-tile readiness timeout derives from a fresh remaining value (see the comment above that second check). The worst case is one settle interval (1s) of overshoot past the deadline before the re-check raises, which is absorbed by the 30s cleanup reserve plus the 30s Celery hard grace. Gating the sleep itself on remaining time would save at most that 1s in an execution that is already failing.
There was a problem hiding this comment.
✅ Customized review instruction saved!
Instruction:
Do not require capping the scroll-settle sleep when budget checks occur immediately before scrolling and after the sleep, since the bounded overshoot is intentionally covered by the cleanup reserve and hard grace period.
Applied to:
superset/utils/screenshot_utils.py
💡 To manage or update this instruction, visit: CodeAnt AI Settings
There was a problem hiding this comment.
Code Review Agent Run #807e67
Actionable Suggestions - 4
-
superset/commands/report/execute.py - 2
- Missing unit tests for new function · Line 134-161
- Missing exception path test · Line 162-262
-
superset/utils/webdriver.py - 2
- Dead code: conditional guard on non-null deadline · Line 1313-1313
- Dead code: conditional guard on non-null deadline · Line 1345-1345
Additional Suggestions - 7
-
tests/unit_tests/utils/test_screenshot_utils.py - 1
-
Dropped assertions reduce test coverage · Line 935-940Removing all 7 positional assertions from `warning_args` leaves a coverage gap. The docstring says 'Budget exhaustion is a customer chart-loading issue, not a Superset system fault, so it must log at WARNING (not ERROR)', but the removed assertions verify the warning's structured data fields (tile index, counts, elapsed/budget seconds, log-context suffix) — not just the message template. The subsequent `test_budget_exhausted_warning_includes_log_context` test only covers the `warning_args[-1]` log-context field (line 978), not the 6 positional arguments (tile index, tiles captured/total, elapsed, budget values). If the warning format changed, these assertions should reflect the new format; if they didn't change, they should remain.
-
-
superset/utils/report_execution.py - 1
-
Missing negative-value input guards · Line 149-176`available_seconds` at line 152 uses `max(0.0, reserve_seconds)` which silently discards any negative caller value; `timeout_seconds` at line 174 applies `or requested_seconds <= 0` for the same purpose. Both guards hide caller bugs and produce zero when a caller passes a negative timeout. Add explicit ValueError guards so callers discover the bug immediately rather than receiving an unexpectedly unbounded or zero timeout.
-
-
superset/commands/report/execute.py - 2
-
Missing retry complete log · Line 1358-1358When the SlackV2 retry succeeds, `notification.send()` (line 1358) exits the inner `try` without hitting the `log_report_delivery_phase(..., "complete", ...)` that the normal path has on line 1338. Add the missing call so the retry path also records a "complete" phase rather than silently returning.
-
Redundant _phase_timeout call · Line 869-872Duplicate `_phase_timeout` call with identical arguments appears before and after `build_pdf_from_screenshots`. The second call after the operation completes is redundant.
-
-
superset/utils/webdriver.py - 2
-
Dead code: unused timeout variable · Line 1455-1460Assign the result of `phase_timeout` to a timeout variable and apply it to the screenshot capture to enforce the deadline, following the existing pattern at lines 901–916.
-
Inconsistent expected_holders value for chart capture · Line 1307-1307In the chart-container branch, `expected_chart_count` from `report_execution_context` reflects the dashboard chart count, but the screenshot targets one chart. Use `1` to avoid misleading log output.
-
-
superset/utils/screenshot_utils.py - 1
-
Missing test for CHART_HOLDERS_MOUNTED_JS · Line 226-226The `CHART_HOLDERS_MOUNTED_JS` constant is used at line 421 to wait for chart holders to appear before readiness checks begin, but has no unit test. The sibling constants `REPORT_CHART_HOLDERS_READY_JS` and `CHART_HOLDERS_READY_JS` are tested at lines 654-692 of test_screenshot_utils.py. Adding a corresponding assertion would complete the coverage.
-
Filtered by Review Rules
Bito filtered these suggestions based on rules created automatically for your feedback. Manage rules.
-
superset/utils/webdriver.py - 1
- Semantic duplication of elapsed computation · Line 343-347
-
tests/unit_tests/utils/test_report_execution.py - 2
- Missing docstring on helper · Line 32-44
- Missing docstrings and type annotations · Line 47-209
-
tests/unit_tests/utils/test_screenshot_utils.py - 1
- Unreachable mock return value · Line 628-630
-
superset/utils/report_execution.py - 1
- Log field name mismatch · Line 218-218
Review Details
-
Files reviewed - 20 · Commit Range:
55e5d18..8feae1a- docs/admin_docs/configuration/alerts-reports.mdx
- superset/commands/report/execute.py
- superset/commands/report/execute_now.py
- superset/config.py
- superset/initialization/__init__.py
- superset/mcp_service/screenshot/pooled_screenshot.py
- superset/tasks/scheduler.py
- superset/utils/report_execution.py
- superset/utils/screenshot_utils.py
- superset/utils/screenshots.py
- superset/utils/webdriver.py
- tests/integration_tests/reports/commands_tests.py
- tests/integration_tests/reports/scheduler_tests.py
- tests/unit_tests/commands/report/execute_test.py
- tests/unit_tests/commands/report/test_execute_now.py
- tests/unit_tests/initialization_test.py
- tests/unit_tests/tasks/test_scheduler_soft_timeout.py
- tests/unit_tests/utils/test_report_execution.py
- tests/unit_tests/utils/test_screenshot_utils.py
- tests/unit_tests/utils/webdriver_test.py
-
Files skipped - 1
- UPDATING.md - Reason: Filter setting
-
Tools
- MyPy (Static Code Analysis) - ✔︎ Successful
- Astral Ruff (Static Code Analysis) - ✔︎ Successful
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
Bito Usage Guide
Commands
Type the following command in the pull request comment and save the comment.
-
/review- Manually triggers a full AI review. -
/pause- Pauses automatic reviews on this pull request. -
/resume- Resumes automatic reviews. -
/resolve- Marks all Bito-posted review comments as resolved. -
/abort- Cancels all in-progress reviews.
Refer to the documentation for additional commands.
Configuration
This repository uses Superset You can customize the agent settings here or contact your Bito workspace admin at evan@preset.io.
Documentation & Help
| def log_report_delivery_phase( | ||
| report_context: ReportExecutionContext | None, | ||
| recipient_type: ReportRecipientType | None, | ||
| phase: str, | ||
| *, | ||
| enforce_budget: bool, | ||
| ) -> None: | ||
| """Enforce and log a notification phase when executing a report.""" | ||
|
|
||
| if report_context is None: | ||
| return | ||
| deadline = report_context.deadline | ||
| if enforce_budget: | ||
| deadline.timeout_seconds( | ||
| "notification_delivery", | ||
| reserve_seconds=report_context.cleanup_reserve_seconds, | ||
| ) | ||
| logger.info( | ||
| "report_delivery_%s %s recipient_type=%s elapsed_seconds=%.2f " | ||
| "remaining_seconds=%.2f", | ||
| phase, | ||
| report_context.log_context, | ||
| recipient_type, | ||
| deadline.elapsed_seconds, | ||
| deadline.remaining_seconds, | ||
| ) | ||
|
|
||
|
|
There was a problem hiding this comment.
log_report_delivery_phase is called at lines 1322, 1338, and 1352 but has no dedicated unit test. This function implements important execution-logging logic with conditional budget enforcement—coverage gaps could allow regressions in notification timing to go undetected.
Code Review Run #807e67
Should Bito avoid suggestions like this for future reviews? (Manage Rules)
- Yes, avoid them
There was a problem hiding this comment.
Added in 7e2010fe0d: three unit tests covering the no-context no-op, the enforce_budget=True raise on an exhausted budget, and the enforce_budget=False post-send logging path (which must record the phase rather than raise mid-notification).
There was a problem hiding this comment.
The suggestion to add unit tests for log_report_delivery_phase is appropriate. Adding these tests ensures that the function's logic—specifically the conditional budget enforcement and the logging behavior—is verified and protected against future regressions.
| def persist_owned_report_execution_terminal_error( | ||
| report_schedule_id: int, | ||
| execution_id: UUID, | ||
| error_message: str, | ||
| terminal_reason: str, | ||
| report_context: ReportExecutionContext | None = None, | ||
| ) -> bool: | ||
| """ | ||
| Terminalize this command's WORKING row from its application-owned boundary. | ||
|
|
||
| Report states normally persist their terminal result before re-raising. If | ||
| that first write loses its transaction or database connection, the command | ||
| boundary is the last safe in-process retry: it still has Flask application | ||
| context and knows the execution UUID it owns. A compare against the latest | ||
| active WORKING row prevents an old worker from changing the schedule state | ||
| after a newer execution has started. | ||
| """ | ||
|
|
||
| try: | ||
| # The state-machine transaction has already rolled back on its way to | ||
| # this boundary. Roll back again so a failed terminal flush cannot leave | ||
| # the scoped session unusable for the retry. | ||
| db.session.rollback() # pylint: disable=consider-using-transaction | ||
| working_log = ( | ||
| db.session.query(ReportExecutionLog) | ||
| .filter( | ||
| ReportExecutionLog.report_schedule_id == report_schedule_id, | ||
| ReportExecutionLog.uuid == execution_id, | ||
| ReportExecutionLog.state == ReportState.WORKING, | ||
| ReportExecutionLog.error_message.is_(None), | ||
| ) | ||
| .first() | ||
| ) | ||
| if working_log is None: | ||
| return False | ||
|
|
||
| latest_working_log = ( | ||
| db.session.query(ReportExecutionLog) | ||
| .filter( | ||
| ReportExecutionLog.report_schedule_id == report_schedule_id, | ||
| ReportExecutionLog.state == ReportState.WORKING, | ||
| ReportExecutionLog.error_message.is_(None), | ||
| ) | ||
| .order_by(ReportExecutionLog.end_dttm.desc()) | ||
| .first() | ||
| ) | ||
| report_schedule = working_log.report_schedule | ||
| owns_schedule_state = ( | ||
| report_schedule.last_state == ReportState.WORKING | ||
| and latest_working_log is not None | ||
| and latest_working_log.uuid == execution_id | ||
| ) | ||
| ended_at = datetime.now(timezone.utc).replace(tzinfo=None) | ||
| working_log.state = ReportState.ERROR | ||
| working_log.error_message = error_message | ||
| working_log.end_dttm = ended_at | ||
| if owns_schedule_state: | ||
| report_schedule.last_state = ReportState.ERROR | ||
| report_schedule.last_eval_dttm = ended_at | ||
|
|
||
| db.session.commit() # pylint: disable=consider-using-transaction | ||
| log_context = ( | ||
| report_context.log_context | ||
| if report_context is not None | ||
| else ( | ||
| f"capture_kind=report execution_id={execution_id} " | ||
| f"report_schedule_id={report_schedule_id} " | ||
| f"dashboard_id={report_schedule.dashboard_id} " | ||
| f"chart_id={report_schedule.chart_id}" | ||
| ) | ||
| ) | ||
| elapsed_seconds = ( | ||
| f"{report_context.deadline.elapsed_seconds:.2f}" | ||
| if report_context is not None | ||
| else "unknown" | ||
| ) | ||
| remaining_seconds = ( | ||
| f"{report_context.deadline.remaining_seconds:.2f}" | ||
| if report_context is not None | ||
| else "unknown" | ||
| ) | ||
| logger.info( | ||
| "report_execution_terminal %s state=%s terminal_reason=%s " | ||
| "elapsed_seconds=%s remaining_seconds=%s", | ||
| log_context, | ||
| ReportState.ERROR.value, | ||
| terminal_reason, | ||
| elapsed_seconds, | ||
| remaining_seconds, | ||
| ) | ||
| return True | ||
| except Exception: # noqa: BLE001 # never mask the report's original exception | ||
| db.session.rollback() # pylint: disable=consider-using-transaction | ||
| logger.exception( | ||
| "Failed terminal persistence retry for report execution " | ||
| "capture_kind=report execution_id=%s report_schedule_id=%s " | ||
| "terminal_reason=terminal_persistence_retry_failed", | ||
| execution_id, | ||
| report_schedule_id, | ||
| ) | ||
| return False |
There was a problem hiding this comment.
persist_owned_report_execution_terminal_error has tests for the success and race-condition paths (lines 2719-2763 in test file) but no test for the exception handler at line 253. The except Exception block is the safety net for terminal persistence failures—it should be exercised to verify rollback and logging behavior.
Code Review Run #807e67
Should Bito avoid suggestions like this for future reviews? (Manage Rules)
- Yes, avoid them
There was a problem hiding this comment.
Added in 7e2010fe0d: test_terminal_persistence_retry_survives_database_failure makes the session query raise, and asserts the double rollback (entry + handler), the terminal_persistence_retry_failed exception log, no commit, and the False return that keeps the report's original exception unmasked.
There was a problem hiding this comment.
The suggestion to add a test for the exception handler in persist_owned_report_execution_terminal_error is appropriate. Exercising the except Exception block at line 253 is necessary to verify that the session rollback and the terminal_persistence_retry_failed exception logging occur as expected when terminal persistence fails.
superset/commands/report/execute.py
except Exception: # noqa: BLE001 # never mask the report's original exception
db.session.rollback() # pylint: disable=consider-using-transaction
logger.exception(
"Failed terminal persistence retry for report execution "
"capture_kind=report execution_id=%s report_schedule_id=%s "
"terminal_reason=terminal_persistence_retry_failed",
execution_id,
report_schedule_id,
)
return False
| holder.get("state") in ready_states | ||
| for holder in holder_states | ||
| ), | ||
| (f"{deadline.elapsed_seconds:.2f}" if deadline else None), |
There was a problem hiding this comment.
deadline is unconditionally assigned from report_execution_context.deadline on line 1301; the if deadline else None guard on line 1313 is always-true and masks the actual intent. Replace with direct attribute access.
Code Review Run #807e67
Should Bito avoid suggestions like this for future reviews? (Manage Rules)
- Yes, avoid them
There was a problem hiding this comment.
Fixed in 7e2010fe0d — both this and the second occurrence at line 1345 removed; deadline is assigned unconditionally there and ReportExecutionDeadline is always truthy.
There was a problem hiding this comment.
The suggestion to remove the conditional guard is appropriate. Since deadline is assigned unconditionally from report_execution_context.deadline on line 1301, the if deadline else None check on line 1313 is redundant and can be simplified to direct attribute access.
superset/utils/webdriver.py
(f"{deadline.elapsed_seconds:.2f}" if deadline else None),
| report_execution_context.expected_chart_count, | ||
| len(holder_states), | ||
| ready_holders, | ||
| (f"{deadline.elapsed_seconds:.2f}" if deadline else None), |
There was a problem hiding this comment.
deadline is unconditionally assigned on line 1334; the if deadline else None guard on line 1345 is always-true and is dead code. Replace with direct attribute access.
Code Review Run #807e67
Should Bito avoid suggestions like this for future reviews? (Manage Rules)
- Yes, avoid them
There was a problem hiding this comment.
Fixed in 7e2010fe0d (same commit as the sibling at line 1313).
There was a problem hiding this comment.
The suggestion to remove the if deadline else None guard is correct. Since deadline is unconditionally assigned on line 1334, the conditional check is redundant and can be simplified to direct attribute access.
superset/utils/webdriver.py
(f"{deadline.elapsed_seconds:.2f}" if deadline else None),
(f"{deadline.remaining_seconds:.2f}" if deadline else None),
…drop dead guards Per bot review feedback: - log_report_delivery_phase: no-op without a report context, raises on exhausted budget when enforcing, and still logs post-send phases without raising when enforcement is off. - persist_owned_report_execution_terminal_error: a DB failure inside the retry itself rolls back, logs, and returns False without masking the report's original exception. - Remove two always-true 'if deadline' guards in the Selenium readiness logging (deadline is assigned unconditionally and always truthy). Co-Authored-By: Claude <noreply@anthropic.com>
| log_report_delivery_phase( | ||
| report_context, | ||
| recipient.type, | ||
| "retry", | ||
| enforce_budget=True, | ||
| ) | ||
| notification.send() |
There was a problem hiding this comment.
Suggestion: The Slack fallback sends the notification after upgrading from v1 to v2 but never records a complete delivery phase. Successful fallback deliveries therefore produce only start and retry events, causing phase-level monitoring and delivery audit logic to report an incomplete delivery even though the notification succeeded. Emit the same completion event after the retry send. [incomplete implementation]
Severity Level: Minor 🧹
- ⚠️ Slack fallback telemetry lacks successful completion.
- ⚠️ Delivery diagnostics show an incomplete phase sequence.
- ⚠️ Operators cannot distinguish retry success from interruption.(Use Cmd/Ctrl + Click for best experience)
Prompt for AI Agent 🤖
This is a comment left during a code review.
**Path:** superset/commands/report/execute.py
**Line:** 1352:1358
**Comment:**
*Incomplete Implementation: The Slack fallback sends the notification after upgrading from v1 to v2 but never records a `complete` delivery phase. Successful fallback deliveries therefore produce only `start` and `retry` events, causing phase-level monitoring and delivery audit logic to report an incomplete delivery even though the notification succeeded. Emit the same completion event after the retry send.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
Code Review Agent Run #430e57Actionable Suggestions - 0Review Details
Bito Usage GuideCommands Type the following command in the pull request comment and save the comment.
Refer to the documentation for additional commands. Configuration This repository uses Documentation & Help |
SUMMARY
Fixes scheduled-dashboard report reliability for DOM-heavy dashboards such as dashboard 805.
The immediate root cause was a vacuous dashboard-readiness predicate: when React had mounted zero production chart holders, the browser path warned and continued, allowing blank/spinner screenshots to be delivered. Independent screenshot, Celery, and
working_timeoutvalues also left report executions without one authoritative deadline.This change:
working_timeoutmodel default, and capped at each schedule'sworking_timeout) with capture (60s), delivery (120s), and terminal cleanup (30s) reserves;ALERT_REPORTS_WORKING_TIME_OUT_KILLis enabled: soft limit = resolved budget, hard limit = budget + 30s grace (3600s/3630s with defaults). Alerts retain their per-scheduleworking_timeoutplus existing lag semantics for the numeric limits; note the newSoftTimeLimitExceededhandler in the sharedreports.executetask (metric, warning log, explicit FAILURE state before re-raising) applies to alerts as well as reports — previously the exception propagated uncaught, so for alerts this is an observability-only change. Disabling Celery kill limits does not disable cooperative application-deadline checks;ERRORwithout spending hard-limit grace on an error notification, and incrementsreports.execute.celery_soft_timeoutso operators can alert on the otherwise customer-silent failure;WORKINGrecovery by the report budget. A same-execution_idreplay promotes its originalWORKINGrow toERROR; a distinct recovery invocation records its ownERRORand unblocks the schedule without mutating the old worker's audit row or attempting uncertain delivery;boundary. The retry only promotes the exact execution UUID's active
WORKINGrow and only changes the schedule when that UUID is still the latest active
execution, so a delayed worker cannot overwrite a newer run;
Adversarial scope/recovery review
The follow-up commit deliberately removed two unsafe pieces from the initial draft:
task_failuresignal handler performs metadata-DB cleanup. A hard-killed/lost worker can emit that signal from the Celery parent process, where a Superset Flask/SQLAlchemy transaction is not guaranteed. Immediate lost-worker cleanup therefore needs a separate lease/watchdog design and is out of scope for this correctness PR.The application-owned next invocation is the safe schedule-recovery hook: after the 15-minute bound it records the recovery invocation's
ERROR, or promotes the same execution row in place when theexecution_idis replayed. A distinct oldWORKINGaudit row is deliberately left untouched because Celery hard limits do not preemptsolo/eventlet/geventworkers and the original worker may still be alive. This removes the review-identified lost-update race while still allowing the next schedule to start fromERROR. Durable lost-worker terminalization requires a lease/watchdog plus compare-and-set ownership and is a separate follow-up, not a claim made by this PR.No retry was added. Playwright already creates a fresh browser context per capture; retrying delivery is not safe without provider idempotency. The historical tiling decision guard was also restored: chart count alone does not force tiling when the dashboard is shorter than one tile, preserving report and thumbnail defaults.
The remaining cross-layer scope is intentional: the same deadline must reach scheduler task options, report state, browser navigation/readiness/capture, data/PDF generation, delivery, and terminal persistence. Splitting those pieces would reintroduce conflicting limits or allow a correctness path to bypass the deadline. Review-driven scope narrowing removed cross-execution audit-row mutation and the incidental tiling behavior change rather than layering on a database race fix. More invasive immediate worker-loss detection is explicitly excluded above.
The application deadline is cooperative between synchronous phases. Celery soft/hard limits are the final preemption boundary when the configured pool supports them. In particular,
build_pdf_from_screenshotsis checked immediately before and after the synchronous call but cannot be interrupted from inside that call; the after-check prevents delivery once an overrun returns.BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
Not applicable; this is background report orchestration and browser-capture behavior.
TESTING INSTRUCTIONS
Automated validation completed locally:
pre-commit run --all-fileswas also executed as required. It reached the full repository and exposed unrelated base/environment failures: 14 existing mypy errors in version-restore tests; missing/incomplete frontend dependencies and cache failures in prettier/stylelint; read-only default npm configuration for docs eslint; and existing repo-wide PT004/E402 Ruff findings. Its auto-format noise outside this PR was removed. The changed-file suite above passes.The first draft SHA's failed checks were inspected:
pre-commit (current): branch-caused Ruff formatting inwebdriver_test.py;_sendtests);test-postgres-required: aggregate failure from the Postgres job.All are addressed in the follow-up commit and covered by the targeted tests above.
The next CI iteration exposed and fixed two additional branch-caused deltas:
touched webdriver test differed from the older local binary and were corrected
in a formatting-only commit.
set. Alerts now retain their established format-specific timeout/error
notification behavior, while report soft timeouts still propagate to terminal
cleanup with no error delivery. The report CSV integration expectation and
report-vs-alert unit coverage were updated accordingly.
The review-fix commit adds focused coverage for boot-time config rejection,
the soft-timeout operator metric, distinct-execution recovery without an audit
lost-update, the Selenium seconds/reserve wiring, and the historical short-
dashboard tiling guard.
Final GitHub CI for
830e6cb5d532f470c247b7c6c99712cb388bc6bdreached terminal state: 49 checks passed and 10 expected path-based skips or
neutral Netlify rule checks remain. There are no failed, cancelled, or pending
checks.
Staging defect follow-up: readiness timeout terminal persistence
The exact
v6.0.0.22backport exposed a cleanup-path defect in workspace6970454bforreport_schedule_id=2, dashboard 8, executiond390442c-9539-4343-ad9a-06ec2359e39f. The run started at 05:42:16 and timedout at 05:53:46 at the readiness allocation with
elapsed=689.86,remaining=209.94, and only 12/52 chart containers rendered. Capture correctlystopped, but the execution remained
WORKING: the generic unexpected-error logappeared without a
report_execution_terminal state=Errorevent. A concurrentduplicate (
c56c2434) had correctly been refused while the original was active,so leaving the owner row
WORKINGalso blocked later schedules.The workspace execution-history UI makes both persistence defects definitive:
d39044, scheduled 05:40 and started 05:42:16, still showed thegreen
WORKINGicon, duration00:00:00.000, and a blank error after the689.86-second timeout;
c56c24, scheduled 05:44 and started 05:44:48, showedduration
00:00:00.005and the refusal error but also retained the greenWORKINGicon.The correct history is one terminal
ERRORowner row with its actual end,duration, and capture error, plus one terminal
ERRORrefusal row. The refusalmust not change the active owner's schedule state, and it must not add another
row eligible for
WORKINGtimeout/recovery queries.The capture exception itself is handled by the report state machine:
Playwright
TimeoutErrorbecomesReportScheduleScreenshotFailedError, afterwhich the state attempts to promote its
WORKINGrow toERROR. The escape wasin that terminal-write error handling. A raw SQLAlchemy failure from the write
was not covered by the existing
ReportScheduleUnexpectedErrorguard; thetransaction wrapper could therefore replace the capture exception and reach the
outer generic unexpected-error path without a second persistence attempt.
Reserved cleanup time existed but was unused.
Focused commits
d81ef613da0be73cc4f485a551b92d1722f4d647andf27092c928acd66d90b57ba505c4621264524295, followed by ownership hardening inac3f21ce4716f80adcd9d82f2b7cb03a5224a091and execution-history correction in4ae2324f3f06a1fa99476514dd5429c03a7febe4. The MySQL timestamp-precisiontest correction is
2b745d022dd1dcb239a295859460588c15d2437e:a SQLAlchemy error, rolls back the failed session, and retries from the
in-process command boundary;
error-free
WORKINGrow, promotes that audit row, and only moves the schedulefrom
WORKINGtoERRORwhen the same UUID is still the latest activeexecution;
elapsed/remaining budget;
WORKINGstate before thestate machine runs, and only permits the command-boundary retry for that
owner. This is stronger than checking the eventual exception: a
same-
execution_idreplay cannot mistake the still-active owner's row for itsown failed terminal write even if persisting the duplicate-refusal log itself
loses its database transaction;
ERRORaudit row,with end time and refusal error, while leaving the original owner row and
schedule
WORKING. It no longer creates a second active-lookingWORKINGrow;
delivery retry.
Focused validation:
The new integration test starts from a prior
SUCCESSstate, raises an ordinaryPlaywright timeout, injects a database failure into the first
ERRORwrite,asserts the same execution is durably terminal
ERRORwith a real end time,duration, and error text, with no delivery, and
then proves a distinct next execution reaches
SUCCESS. Existing duplicate-runcoverage is run alongside it, including a same-ID fresh replay that must leave
the active owner and schedule
WORKING, both with a successful refusal writeand with an injected SQLAlchemy failure in that write. The successful-refusal
cases assert exactly one active
WORKINGrow and one terminalERRORrefusalrow with
end_dttm, preventing execution-history accumulation. Mypy reports the same two pre-existingSlackChannelSchemaerrors inexecute.pyat the parent SHA and this SHA; thechanged-line delta is clean.
CI on
4ae2324f3f06a1fa99476514dd5429c03a7febe4found one branch-caused testportability issue: MySQL stores these metadata timestamps with one-second
precision, so the subsecond mocked execution correctly persisted both
timestamps but failed a strict
end_dttm > start_dttmassertion. Commit2b745d022dd1dcb239a295859460588c15d2437euses>=for that fast-path test;the 689.86-second staging execution will retain a non-zero stored duration. The
same run's SQLite job was unrelated: GitHub Actions timed out three times while
pulling
redis:7-alpinefrom Docker Hub, before checkout or tests.GitHub CI for the final SHA reached terminal state with all checks passing or
expected path-based skips/neutral results. Code, unit, integration (including
MySQL/Postgres/SQLite), E2E, pre-commit, CodeQL, and the repository's delayed
🎪 Superset Showtimesync are terminal; there are no failures, cancellations,or pending checks.
Staging evidence: dashboard 10
Staging dashboard 10 has 52 charts and produced a 7,504px tiled report. Two
scheduled executions both completed capture and delivered:
The first successful capture alone exceeded 300 seconds, and its total execution
was 305.81 seconds. A 300-second end-to-end deadline would therefore terminate a
report that this workload can successfully capture and deliver. The second run
also leaves little capacity under a 300-second limit. This is direct staging
evidence for the unified 15-minute budget rather than another 300s screenshot or
task assumption.
The 30/60-minute late starts are not part of those capture/execution
durations. They are separate queue, worker-capacity, beat, or scheduler latency
to investigate independently. They do not demonstrate slow DOM readiness, and
they must not be used to explain the chunk error.
Delivery is pipeline success, not proof that all 52 charts rendered
semantically. Readiness treats rendered, empty, and explicit-error holders as
terminal so that an error panel can be captured instead of spinning forever.
This PR retains that semantic-success policy: it prevents zero-holder,
nothing-mounted, and spinner capture, but it does not fail the entire report
merely because a chart reached an explicit error or empty terminal state. The
user-reported chunk error in run 1 therefore needs a per-holder audit; delivery
alone neither classifies the error nor proves full-chart success.
The staging audit must record, per run and deduplicated by chart ID across
tiles,
rendered_holders,empty_holders, andexplicit_error_holders, alongwith expected/mounted totals. If existing holder-state diagnostics cannot
reconstruct those counts reliably, record that as an observability gap rather
than reporting all 52 charts as rendered.
Staged validation plan on reproducer
c5c287cac5c287cadashboard 805 run as the control, then build this draft branch in staging only. Configure the default 900/60/120/30/30-second values explicitly.pivot_table_v2, 19table) as PDF. Confirmexpected_holders=52, holder counts progress from zero to a positive mounted set, every viewport-visible mounted holder reaches a terminal state, capture occurs only afterward, and the execution reachesSUCCESSwith one delivery inside the budget.ERRORwithend_dttm, duration, and the refusal error while the owner's row and schedule remainWORKING; there must still be exactly one activeWORKINGrow. Then confirm owner timeout propagation, anERRORterminal reason, a real owner end/duration/error, cleanup capacity remaining, no incomplete capture or delivery, no remaining activeWORKINGrow, and that a distinct following schedule is admitted. Confirm the terminal event carries the same execution UUID and elapsed/remaining budget rather than only the generic unexpected-error log..chart-containerterminal readiness rather than the dashboard-holder predicate. Run empty dashboard thumbnails and confirm their legacy zero-holder/fallback behavior remains unchanged.ERRORbefore the 930s hard limit, incrementsreports.execute.celery_soft_timeout, and does not attempt an in-band customer error notification.WORKING. Do not expect unsafe signal-side DB cleanup. On the first distinct invocation after the 15-minute stale bound, confirm the recovery invocation recordsERROR, the old audit row is not mutated by a potentially racing worker, no delivery occurs in recovery, and a following new schedule can proceed. Track lease/watchdog terminalization of the old row as a separate follow-up.execution_id; confirm the original row is promoted in place and no duplicate log/delivery is produced.execution_id,report_schedule_id,dashboard_id,chart_id,url,expected_holders,mounted_holders,ready_holders,elapsed_seconds,remaining_seconds,attempt, andterminal_reason. Measure scheduled/queued/worker-start timestamps separately from execution-start/readiness/capture timestamps so scheduler latency is not folded into the report budget.ADDITIONAL INFORMATION
The staged validation plan above has been executed against the staging
reproducer; the terminal-state defect it surfaced is fixed and regression-covered.
This PR is ready for review.