Unified storage migration silently deletes all migrated dashboards post-migration (`dashboard-service` runs unscoped search, treats entire result set as deletion candidates) Synology 918+

[lampshade29]

What happened?

On upgrade from Grafana 12.x to 13.0.2, the unified-storage folders/dashboards migration completes successfully and reports a clean validation (legacy_count=23 unified_count=23 rejected=0) — but within ~3.5 minutes after migration completes, every single migrated dashboard is deleted by Grafana itself. The instance ends up fully "healthy" (HTTP 200, API responsive, folders intact) but with zero dashboards. This is reproducible, deterministic, and independent of SQLite locking mode (rollback-journal vs WAL) or filesystem COW settings — three separate test runs under three different configurations produced the exact same row-level signature down to the integer.

Debug-level logging (GF_LOG_LEVEL=debug) exposes the actual mechanism: a post-migration search runs against the dashboards.dashboard.grafana.app resource with no query/filter (query=), naturally returns the entire migrated set, and dashboard-service interprets "search returned everything" as "found N dashboards to be deleted" — then deletes them all via real, successful DELETE calls through the k8s-style apiserver.

What did you expect to happen?

Expected behavior

Migration completes and all 23 dashboards remain present and queryable.

Actual behavior

Migration completes successfully, then a background process deletes every dashboard it just migrated. The instance reports healthy throughout — this is silent data loss with no error surfaced to the user; the only symptom is an empty dashboard list.

grafana13_debug_test_evidence.md
grafana13_wal_test_evidence.md

• Used isolated throwaway cp -a clones of a 23-dashboard / 3-folder / ~3-months-history SQLite data directory — fully reproducible, zero risk to source data.
• Source legacy dashboard SQL table remains fully intact throughout (26 rows = 23 dashboards + 3 folders) — only the unified-storage migration's output is destroyed; nothing touches the legacy table.
• Loggers to watch when reproducing: dashboard-service (the decision), resource-search (the unfiltered query feeding it), grafana-apiserver (the DELETE execution chain — Going to delete object from registry → About to delete object from database → Object deleted from database).
• Happy to provide the full debug-level log captures (continuous docker logs -f from container start) and/or the throwaway data directory if useful for your own reproduction.

Did this work before?

old Version 12.xx

How do we reproduce it?

1. Take an existing Grafana 12.x SQLite data directory with a non-trivial number of dashboards/folders and accumulated dashboard-version history.
2. Boot grafana/grafana:13.0.2 against that data directory (fresh container, no prior 13.x state).
3. Let the unified-storage migration run to completion — it will report success.
4. Wait ~3-4 minutes after "Unified storage migrations completed successfully".
5. Query /api/search?type=dash-db — returns []. All dashboards are gone. /apis/dashboard.grafana.app/v0alpha1/namespaces/default/dashboards returns items: [].

Whatever cleanup/reconciliation routine dashboard-service runs after a unified-storage migration needs an actual scoping filter on its search — something that distinguishes "stale/orphaned/duplicate records that legitimately need removal" from "the complete set of records that currently and correctly exist." Right now the query has no filter at all (query=), so on a freshly (and correctly) migrated instance, "everything" and "everything that needs deleting" are indistinguishable, and the latter wins.

Is the bug inside a dashboard panel?

The mechanism (debug-level log, verbatim)

This is the core of the report — these three consecutive log lines, ~3 minutes 21 seconds after migration completion, show Grafana naming its own bug:

logger=bleve-backend     namespace=default group=dashboard.grafana.app resource=dashboards size=23 reason=search level=info  msg="Finished building index" elapsed=16.951334309s listRV=1780942591972008
logger=resource-search                                                                                          level=debug msg="Search stats" operation=Search elapsedTime=28.474602412s indexBuildTime=28.468618868s totalHits=23 returnedDocuments=23 namespace=default group=dashboard.grafana.app resource=dashboards query=
logger=dashboard-service                                                                                        level=debug msg="Found dashboards to be deleted" orgId=1 count=23

Read in order: a search runs against the dashboards resource with query= — literally empty, no scoping criteria — naturally matches every dashboard that exists (totalHits=23 returnedDocuments=23, the entire migrated set), and dashboard-service interprets "the search returned everything" as "found 23 dashboards that need deleting."

It then issues 23 real, successful DELETE calls through the apiserver (note the kubernetes/$Format user-agent — this is Grafana's own internal client):

logger=grafana-apiserver level=debug msg=Request verb=DELETE url=https://127.0.0.1:3000/apis/dashboard.grafana.app/v0alpha1/namespaces/default/dashboards/jchmRiqUfXgTM headers="...User-Agent: grafana/v0.0.0 (linux/amd64) kubernetes/$Format\n"
logger=grafana-apiserver level=debug msg="grafana-apiserver: DELETE \"/apis/dashboard.grafana.app/v0alpha1/namespaces/default/dashboards/jchmRiqUfXgTM\" satisfied by gorestful with webservice /apis/dashboard.grafana.app/v0alpha1"
logger=grafana-apiserver level=debug msg="Going to delete object from registry" object=default/jchmRiqUfXgTM
logger=resource-server   level=debug msg="Server. Streaming Event" type=DELETED previousRV=1772363900000079 group=dashboard.grafana.app namespace=default resource=dashboards name=jchmRiqUfXgTM

Full server-side trace for one (representative — all 23 follow this shape):

Trace[1046125123]: "Delete" ... name:BHhxFeZRz ... verb:DELETE (total time: 7939ms):
Trace[1046125123]: ---"About to delete object from database" 0ms
Trace[1046125123]: ---"Object deleted from database" 7938ms
Trace[1046125123]: [7.939383637s] [7.939383637s] END

Important: these are not failed/retried operations colliding under lock contention — they are successful deletes (slowed to ~8 seconds each by SQLite contention on this particular host, but completing "Object deleted from database" every single time). The bug is not that concurrency corrupts the migration. The bug is that a cleanup routine decides the entire freshly-migrated set is garbage, and then — slowly but completely successfully — removes it.

Full ledger of dashboard UIDs deleted (traced 19 of the eventual 23; SQL confirms all 23 were removed):

jchmRiqUfXgTM   BHhxFeZRz    WopVO_mgz   TSmNYvRRk   -pkIkhmRz
jchmDbInfo      zm7wN6Zgz    _7WkNSyWk   Y8upc6ZRk   lBIoQIggk
fu4SiQgWz       be2m9kga7b8q ZzhF-aRWz   NjtMTFggz   riqUfXgRz
xo4BNRkZz       1EZnXszMk    IiC07mgWz   RG_DxSmgk

Direct database verification (bypasses live lock via ?mode=ro&immutable=1)

select action, count(*) from resource_history
  where "group"='dashboard.grafana.app' and resource='dashboards' group by action;

action=1 (create) : 23
action=2 (update) : 75
action=3 (delete) : 23

select count(*) from resource where "group"='dashboard.grafana.app' and resource='dashboards';
→ 0

select count(*) from resource_history where "group"='dashboard.grafana.app' and resource='dashboards';
→ 121   (= 23 + 75 + 23 ✓ internally consistent)

The 75 "updates" are not part of the bug — debug-level migration logs show each dashboard being written through several internal revisions ([v: 1] → [v: 3] → [v: 4] → [v: 5]) as part of the migration's own normal operation, before dashboard-service ever runs. The migration itself is not at fault; what runs ~3 minutes after it is.

Why this is a deterministic code defect, not a contention/timing issue

I reproduced this three times under three meaningfully different configurations:

Run	SQLite mode	NOCOW	Index workers tuned	Debug logging
1	rollback-journal (default)	yes	no	no
2	WAL (GF_DATABASE_WAL=true)	yes	yes (index_workers=2, index_rebuild_workers=1)	no
3	WAL	yes	yes	yes

WAL and rollback-journal are fundamentally different SQLite locking models — WAL removes reader-vs-writer blocking entirely, leaving only writer-vs-writer contention. A timing-sensitive race condition would be expected to produce different outcomes under these different regimes: different survivor counts, different update/delete tallies, perhaps full success on a lucky run. It did not move by a single row. All three runs produced:

23 created → 75 updated → 23 deleted → 0 survivors → 121 total history rows

to the exact integer. That is the signature of a deterministic decision in code — specifically, the unscoped search shown above, which will always return "everything" and will therefore always be misread as "everything is garbage," regardless of how fast or slow the underlying storage resolves locks. Storage latency only changes how long the deletion takes to complete (milliseconds vs. minutes) — not whether it happens.

Environment (with versions)?

• Grafana version: grafana/grafana:13.0.2 (Docker)
• Previous version: 12.4.0 (clean, stable, unaffected)
• Storage backend: SQLite (default), tested under both rollback-journal and WAL modes
• Host: Synology DS918+, Btrfs on mdadm RAID5 (spinning disks), tested both with and without NOCOW on the data directory
• Dataset: 23 dashboards, 3 folders, ~3 months of accumulated dashboard history (TeslaMate stack)
• All testing done on isolated throwaway cp -a clones of the production SQLite data directory — zero risk to live data, live instance verified healthy and serving normally throughout every test

Grafana platform?

None

Datasource(s)?

No response

[RafaelPaulovic]

Hi @lampshade29 , thanks for reporting this. I believe this a misconfiguration issue.

If I understand the report correctly, the TeslaMate stack wrote the dashboards (via provisioning), then you booted the bare grafana/grafana:13.0.2 image against that same data.

The bare image has no provisioning configs, so the cleanup happens.

I.e. if you configure the same provisioning settings when launching the bare image, it shouldn't delete any dashboards.

[RafaelPaulovic]

Closing the issue for now. Please re-open if you have more questions.

[lampshade29]

@RafaelPaulovic
I tested your theory, and you're right, but there's a second issue underneath it on this particular host.

Setup: Synology DS918+, Btrfs on mdadm RAID5 (spinning disks, no SSD cache) — the same box from my earlier reports. Cloned the live teslamate/grafana data dir to a throwaway copy, applied the alerting kv_store fix from my earlier
blocker #2, and booted teslamate/grafana:latest (NOT the bare grafana/grafana:13.0.2 I used in all prior cycles) on a separate port/container, zero risk to the live instance.

Result 1: your misconfiguration theory is confirmed

With teslamate/grafana:latest — which bakes in GF_FEATURE_TOGGLES_provisioning=false plus TeslaMate's actual provisioning configs and dashboard JSON — the unified storage migration completed cleanly:

"Count validation" legacy_count=23 unified_count=23 migration_summary_count=23 rejected=0 history=0

No "Found dashboards to be deleted" log line. Direct SQL check on resource / resource_history confirms 23 dashboards + 3 folders all present, every history row action=1 (created), zero action=3 (deleted). This is the exact
opposite of the 23-created/75-updated/23-deleted/0-survivors signature I got every time with the bare image. So: bare grafana/grafana:13.0.2 against TeslaMate-provisioned data + no provisioning config = dashboard-service treats
the migrated set as orphaned-provisioned and deletes it. Confirmed.

Result 2: separate blocker — provisioning module SQLITE_BUSY (crash or hang)

Even with the dashboard-deletion bug avoided, the container never reached a working state on this hardware:

1st boot — fatal crash during provisioning module startup:
logger=secrets level=error msg="Failed to get current data key" error="database is locked (5) (SQLITE_BUSY)" label=...secretKey.v1
logger=sqlstore.transactions msg="Database locked, sleeping then retrying" error="database is locked (5) (SQLITE_BUSY)" retry=0..4
logger=provisioning level=error msg="Failed to provision data sources" error="Datasource provisioning error: database is locked (5) (SQLITE_BUSY)"
Error: ✗ invalid service state: Failed, expected: Terminated, failure: invalid service state: Failed, expected: Running, failure: not healthy, 0 terminated, 1 failed: [starting module provisioning: invalid service state: Failed,
expected: Running, failure: Datasource provisioning error: database is locked (5) (SQLITE_BUSY)]
5 retries exhausted in ~15s, process exits entirely.

2nd boot (restart — migration already persisted, skipped this time) — survived the secrets/datasource SQLITE_BUSY retries, alerting provisioned fine ("finished to provision alerting"), then hung indefinitely:
logger=provisioning.dashboard msg="starting to provision dashboards"
3+ minutes, zero further log output, /api/health connection refused/reset throughout. Never recovered.

Why v12.4.0 works fine on the exact same disk

12.4.0 has no unified storage layer — no resource/resource_history migration, no bleve search index build, no concurrent job-drivers/history-pruner. Its provisioning step does a handful of sequential SQLite reads/writes at startup
and finishes in seconds, comfortably inside the default 7.5s busy_timeout even on slow Btrfs/RAID5.

13.x adds, all at startup, all hitting the same single SQLite file: the unified-storage migration (23 dashboards + 3 folders + 121 history rows), a bleve index build (14–24s observed here), 3 concurrent job-drivers + a history pruner,
and the legacy provisioning module's secrets/datasource/dashboard provisioning. On fast storage this is invisible. On this box, the combined lock contention blows past 5 retries / 7.5s busy_timeout — sometimes fatally (crash),
sometimes it just never clears (hang).

Net conclusion

Your fix resolves the dashboard-deletion bug — confirmed with real evidence. But on slow-storage hosts (Btrfs-on-spinning-RAID5, possibly any single-disk/SQLite Grafana install under similar I/O latency), v13's startup does enough
additional concurrent SQLite I/O vs v12 that the provisioning module's SQLITE_BUSY retries (5x, short backoff) aren't enough — leading to either a fatal crash or an indefinite hang with zero progress. I'm staying on 12.4.0 for now.
Happy to provide more logs/goroutine dumps if useful — this might warrant a separate issue for the provisioning-module retry/backoff behavior on slow disks.