Upgrade from 9.3.1 to 9.4.3 corrupted the datasource UIDs in 10% of dashboards

[UnrealCraig]

What happened:
Many dashboards stopped working, when comparing the JSON models I discovered the cause is datasource references changed from e.g.:

      "datasource": {
        "type": "elasticsearch",
        "uid": "000000004"
      },

to:

      "datasource": {
        "uid": "Elasticsearch Logs"
      },

and others changed from:

      "datasource": {
        "type": "stackdriver",
        "uid": "000000006"
      },

to:

          "datasource": {
            "uid": "Google Cloud Stackdriver"
          },

This seems to break entire dashboard and not just the Elasticsearch panels; when clicking Edit on any panel on a broken dashboard it shows the default datasource as the source, regardless of the correct source. If changing the datasource selection to correct datasource then all configuration (including queries etc) is lost. On the broken dashboards it seems every datasource on them has been edited and on the remaining dashboards every datasource still has correct format & UID.

What you expected to happen:
Datasource references should not have been edited by the upgrade

How to reproduce it (as minimally and precisely as possible):
Upgrade a large Grafana instance from 9.3.x to 9.4.x
We have over 200 dashboards in 8 organizations with up to 200 panels on each dashboard and up to 19 datasources in each organization (although we only have 4 different types of datasource in total)

The issue wasn't seen when testing on smaller instances (which use equivalent set of datasources)

Anything else we need to know?:
Datasources we use: Prometheus, Elasticsearch, Google StackDriver, Postgres

Environment:

• Grafana version: 9.4.3
• Data source type & version: Multiple/All
• OS Grafana is installed on: Ubuntu (Docker image grafana/grafana:9.4.3-ubuntu)
• User OS & Browser: Not relevant
• Grafana plugins: Only official "By Grafana Labs" plugins for the datasources
• Others:

[mellieA]

@UnrealCraig this looks like a duplicate of #63904

The title says it is for the prometheus datasource, but other users have mentioned seeing it with elasticsearch datasource and others. Are you able to see the query editor when you edit a panel?

Here's one of the developers' comments, I'm going to link these two issues:

It seems the query editor state has a different datasource UID than the UID coming from props. So the component keeps waiting for the correct datasource to be fetched.
The one in the state is the correct one, the one coming from props is not correct.

There is a workaround for this, select again the datasource in the picker, and the editor appears again. Obviously not ideal, but just for you to keep moving.

I'll try to reproduce this, deterministically, and see why we receive from props a different datasource UID.

[UnrealCraig]

The user-facing symptoms seem the same, but our underlying JSON isn't matching.
The upgrade we applied removed the "type" and changed the "uid" whereas in that other ticket the example seems to have both type & uid for all but the screenshot suggests the UI is changing the uid?

[ivanortegaalba]

Hi, here is Ivan from Dashboard,

There are 2 issues here:

1. The described issue here is a migration issue in which we lose the UID and have the name instead. This is the root cause of issue 2.
2. The editor is not rendered, because the UID cannot be found in the user data source. The behavior should be to fall back to the default. And this is what we're trying to solve here QueryEditor: Not rendered when data source not found #64011.

So, the big problem is we're corrupting the UIDs of the data sources.

Can you share with us a dashboard that gets corrupted when migrating? I'd like to reproduce it locally.

[dprokop]

@UnrealCraig friendly ping :)

[grafanabot]

We've closed this issue since it needs more information and hasn't had any activity recently. We can re-open it after you you add more information. To avoid having your issue closed in the future, please read our CONTRIBUTING guidelines.

Happy graphing!