Repository navigation
Show how old a dashboard's source data is ("Data as of"), configured per dataset #46027
Brian-Kariu
started this conversation in
Ideas
Replies: 1 comment
|
When I've had a similar need I use a workaround: a Handlebars chart that queries the max of the desired column and shows it as text. I like your proposal! To answer your questions from Slack, I don't think there's a way to do this currently, and I think this is a good candidate for a SIP. Maybe try taking it to a Superset Townhall meeting, or SIP office hours, if those work for your timezone + schedule. You have it as a Feature Flag above; I think it would be more appropriate as a config flag (see the recent discussion at #44512), we are trying to move away from feature flags as the default, they are more for features that are still being refined over several releases. |
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Motivation
Superset tells users when a query last ran ("Cached 5 minutes ago", and "Last queried at" from #36934). It doesn't tell them how old the data behind the query is. Most of our dashboards read warehouse tables that a pipeline rebuilds every hour or two, so a chart queried two minutes ago can still show data from six hours ago. Users can't tell the two apart, and "is this up to date?" is the question we get most often.
Most teams already know the answer, because their pipeline records it: a
_loaded_atcolumn on each table, a run log, or dbt source freshness. What's missing is a place in Superset to say where that timestamp lives, and somewhere to show it.Proposal
Make the timestamp a dataset setting, roll it up to the dashboard, and show it beside the existing cache-age text.
1. Global config (
superset_config.py)2. Dataset setting (Edit dataset → Settings), stored in the dataset's
extraJSONThis follows the same pattern as
certificationandwarning_markdown. Each dataset gets one of two options:{ "freshness": { "column": "_loaded_at", "warn_after_minutes": 120 } }or, when the timestamp lives somewhere else:
{ "freshness": { "sql": "SELECT MAX(loaded_at) FROM analytics.table_freshness WHERE table_name = 'sales.invoices'", "warn_after_minutes": 120 } }column→ Superset runsSELECT MAX(<column>)on the dataset.sql→ Superset runs the query on the dataset's database and expects a single timestamp back. Jinja is allowed, as in other dataset SQL.warn_after_minutesis optional. Once the data is older than this, the label switches to a warning style, the same idea as dbt'swarn_after.3. Dashboard setting (Dashboard properties), stored in
json_metadata{ "show_data_freshness": true }The dashboard's "data as of" is the oldest freshness value across the datasets its charts use. A dashboard is only as current as its stalest input. Datasets with no freshness setting are skipped.
4. UI
warn_after_minutes.5. API and embedded SDK
GET /api/v1/dashboard/<id_or_slug>/freshnessreturns:{ "data_as_of": "2026-10-29T16:28:00Z", "datasets": [ { "id": 12, "name": "invoices", "data_as_of": "2026-10-29T16:28:00Z", "stale": false } ] }Embedded SDK:
dashboardUiConfig: { showDataFreshness: true }, so host apps can turn the header label on or off likehideTitle.The same endpoint lets a host app read the value and show it in its own layout.
Example
Our pipeline writes one row per warehouse table into a freshness table: the table name and the time of the newest source data it holds. With this proposal, we would set
freshness.sqlonce on each dataset, pointing at its row. Every dashboard built on those datasets would then show the right "data as of" with no per-dashboard work, including dashboards added later.Alternatives considered
datasetalso gets applied to the freshness table, so the chart fails for those users.Open questions
sqlquery could still read data the user can't. Running it with RLS is the safe default. The cost is that a guest RLS rule with nodataset(see above) would break it, so those rules may need to skip freshness queries.I'm happy to help with a SIP or a PR if there's interest.
All reactions