[SIP] Anomaly Detection for Timeseries Charts

[DeepFriedYeti]

Please make sure you are familiar with the SIP process documented
here. The SIP will be numbered by a committer upon acceptance.

[SIP] Proposal for Anomaly Detection for Timeseries Charts

Motivation

Superset has forecasting but no way to visually flag anomalies. Users must export data to detect outliers externally.

On management dashboards, anomalies need to be obvious at a glance — a revenue drop or traffic spike should stand out immediately without manual inspection. This feature adds that: red scatter points overlaid on timeseries charts, highlighting statistical outliers directly where users are already looking.

It's a purely visual feature (not for alerts/reports), reducing time-to-insight on any dashboard with timeseries data.

Proposed Change

Add anomaly detection as a post-processing operation for all 6 ECharts Timeseries chart types (Line, Bar, Area, Scatter, Smooth Line, Step). The implementation follows the exact same architectural pattern as the existing Prophet forecasting feature:

1. Control panel section — "Anomaly Detection" section with enable checkbox and method-specific parameters, placed after the existing Forecast section
2. Frontend operator — Generates an anomaly_detection entry in the post_processing array
3. Backend post-processing function — Receives the DataFrame after SQL execution (and after Prophet if enabled), computes anomalies, appends {column}__anomaly columns
4. Frontend rendering — Recognizes __anomaly suffix columns (same pattern as __yhat, __yhat_lower, __yhat_upper) and renders them as red scatter points

Detection Methods

Method	Algorithm	Best for
Z-Score	Rolling mean/std deviation, flags points where |z| > threshold	General purpose, fast
MAD	Rolling Median Absolute Deviation.	Data with existing outliers (more robust)
Prophet	Fits Prophet model, flags points outside confidence interval	Seasonal data with trends

Forecast Integration

When both forecast and anomaly detection are enabled, anomaly detection automatically runs on the forecast prediction line (__yhat columns). This happens without user configuration because:

• Prophet's post-processing extends the DataFrame with future periods, introducing NaN in original columns for those dates
• Anomaly detection skips columns with NaN values and skips confidence bounds (__yhat_lower, __yhat_upper)
• The __yhat column has complete data for all dates and is processed automatically

Visual Rendering

• Red scatter points (#FF0000) overlaid on the original series
• Minimum symbol size of 10px for clear visibility
• Excluded from legend to avoid clutter
• Tooltip indicator — ⚠ anomaly shown when hovering over anomaly points
• Value labels suppressed on anomaly points

New or Changed Public Interfaces

Backend

• New post-processing operation: anomaly_detection added to pandas_postprocessing module
• Column naming: Appends {column}__anomaly columns to the DataFrame (follows existing __yhat convention)

Frontend

• New form data fields on EchartsTimeseriesFormData:
  • anomalyDetectionEnabled (boolean)
  • anomalyDetectionMethod (string: 'zscore' | 'mad' | 'prophet')
  • anomalyDetectionRollingWindow (number, for zscore/mad)
  • anomalyDetectionSensitivity (number, for zscore/mad)
  • anomalyDetectionConfidenceInterval (number, for prophet)
  • anomalyDetectionSeasonalityYearly/Weekly/Daily (boolean | number | null, for prophet)
• New enum value: Anomaly = '__anomaly' added to ForecastSeriesEnum
• New type field: anomaly?: number added to ForecastValue
• New TypeScript type: PostProcessingAnomalyDetection added to PostProcessingRule union

No changes to:

• REST API endpoints
• Database models or configuration
• CLI tools
• Existing saved dashboards/charts (feature is opt-in)

New dependencies

None. Prophet is already an existing optional dependency used by the forecast feature. The Z-Score and MAD methods use only pandas and numpy, which are core dependencies.

Migration Plan and Compatibility

• No database migration required — no new models or schema changes
• Fully backward compatible — anomaly detection is disabled by default; existing charts and dashboards are unaffected
• Saved charts — existing saved charts will not have anomaly detection fields in their form data, which defaults to disabled (the ANOMALY_DEFAULT_DATA provides all defaults)
• Feature coexistence — works independently of or alongside the existing forecast feature

Rejected Alternatives

1. Frontend-only detection (JavaScript)

Rejected because:

• Limited to data visible in the browser (post-pagination/sampling)
• Cannot leverage Prophet for seasonality-aware detection
• Would create inconsistency with the forecast feature which uses backend post-processing

2. Separate "Anomaly Chart" visualization type

Rejected because:

• Anomalies are most useful as an overlay on existing timeseries charts
• A separate chart type would force users to duplicate their chart configuration
• The overlay approach matches how Prophet forecast is already rendered

3. Alert/Report integration

Not pursued in this proposal because:

• Anomaly detection's primary value is visual — instant recognition on dashboards
• Alert integration would require additional infrastructure (thresholds, notification channels, scheduling)
• Can be added as a follow-up enhancement without changing this implementation

Samples

[DeepFriedYeti]

I have created a draft pull request with the code changes required: #40523