Make Grafana more friendly to read-only MySQL database

[azhiltsov]

What Grafana version are you using?

5.2.2

What datasource are you using?

Mixed (irrelevant to my question)

What OS are you running grafana on?

Linux

What did you do?

Grafana is one of the main tools for troubleshooting during outages and we are looking to a way to make it as much reliable as possible.
We have a big installation of Grafana (around 90 instances in the same Domain/Org)
We are using MySQL to store both Grafana config and Sessions.
We are using MySQL Master->many_Slaves replication chain managed by Orchestrator https://github.com/github/orchestrator
Since Grafana can utilize only single MySQL server(Master) for it internal needs #399 #3676 we are providing service discovery by using short living DNS records pointing to a current Master, updated by Orchestrator for redundancy. This setup work fine until recently.
The recent DNS outage proved that this DNS dependency is better to be avoided, as If DNS fails then we do not have a way to discover MySQL and Grafana fails to perform any operations without MySQL available. Essentially if your DNS is broken, then everything else is broken.

We were thinking about adding some extra fail-over logic on Grafana boxes where we could pull all available slaves and make them available locally for grafana, for instance via /etc/hosts. This should make grafana available for troubleshooting, however grafana just can't deal with read-only database.

What was the expected result?

I would expect it to keep providing read-only access to all dashboards, search and navigation should work.

What happened instead?

If restarted it can't start
t=2018-09-24T13:43:23+0200 lvl=eror msg="Server shutdown" logger=server reason="Service init failed: Datasource provisioning error: Error 1290: The MySQL server is running with the --read-only option so it cannot execute this statement"

If IP of the MySQL has been changed to RO then it can't search for dashboards and navigation is impossible with "Server error" or "Unknown error"

Proper solution as I can see it:

• Teach Grafana to work with Write / RO connections separately
• Fail gracefully if Write connection unavailable

[stuartnelson3]

Is there interest in adding this feature?

[bergquist]

In theory, I think it makes sense to support this. But I'm not sure at what extent.
In practice, I dont think anyone in the Grafana team will be working on this in the near future.

[bergquist]

The ORM we use in Grafana support this but the documentation is quite lacking and we are not very confident about the lib itself. https://godoc.org/github.com/go-xorm/xorm#NewEngineGroup

[karlism]

Run into similar issue while trying to install HA Grafana setup with PostgreSQL database, with second Grafana instance having replicated PostgreSQL database as it's primary DB (read only access is fine for us on backup Grafana node). Grafana server dies with following error upon startup:

t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create migration_log table"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create user table"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="add unique index user.login"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="add unique index user.email"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="drop index UQE_user_login - v1"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="drop index UQE_user_email - v1"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Rename table user to user_v1 - v1"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create user table v2"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create index UQE_user_login - v2"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create index UQE_user_email - v2"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="copy data_source v1 to v2"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Drop old table user_v1"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Add column help_flags1 to user table"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Update user table charset"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Add last_seen_at column to user"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Add missing user data"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Add is_disabled column to user"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create temp user table v1-7"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create index IDX_temp_user_email - v1-7"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create index IDX_temp_user_org_id - v1-7"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create index IDX_temp_user_code - v1-7"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create index IDX_temp_user_status - v1-7"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Update temp_user table charset"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create star table"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="add unique index star.user_id_dashboard_id"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create org table v1"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create index UQE_org_name - v1"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create org_user table v1"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create index IDX_org_user_org_id - v1"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create index UQE_org_user_org_id_user_id - v1"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Update org table charset"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Update org_user table charset"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Migrate all Read Only Viewers to Viewers"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create dashboard table"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="add index dashboard.account_id"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="add unique index dashboard_account_id_slug"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create dashboard_tag table"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="add unique index dashboard_tag.dasboard_id_term"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="drop index UQE_dashboard_tag_dashboard_id_term - v1"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Rename table dashboard to dashboard_v1 - v1"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create dashboard v2"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create index IDX_dashboard_org_id - v2"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create index UQE_dashboard_org_id_slug - v2"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="copy dashboard v1 to v2"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="drop table dashboard_v1"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="alter dashboard.data to mediumtext v1"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Add column updated_by in dashboard - v2"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Add column created_by in dashboard - v2"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Add column gnetId in dashboard"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Add index for gnetId in dashboard"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Add column plugin_id in dashboard"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Add index for plugin_id in dashboard"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Add index for dashboard_id in dashboard_tag"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Update dashboard table charset"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Update dashboard_tag table charset"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Add column folder_id in dashboard"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Add column isFolder in dashboard"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Add column has_acl in dashboard"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Add column uid in dashboard"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Update uid column values in dashboard"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Add unique index dashboard_org_id_uid"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Remove unique index org_id_slug"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Update dashboard title length"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Add unique index for dashboard_org_id_title_folder_id"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create dashboard_provisioning"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create dashboard_provisioning v2"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id - v2"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id_name - v2"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="copy dashboard_provisioning v1 to v2"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="drop dashboard_provisioning_tmp_qwerty"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Add check_sum column"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create data_source table"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="add index data_source.account_id"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="add unique index data_source.account_id_name"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="drop index IDX_data_source_account_id - v1"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="drop index UQE_data_source_account_id_name - v1"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Rename table data_source to data_source_v1 - v1"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create data_source table v2"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create index IDX_data_source_org_id - v2"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create index UQE_data_source_org_id_name - v2"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="copy data_source v1 to v2"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Drop old table data_source_v1 #2"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Add column with_credentials"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Add secure json data column"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Update data_source table charset"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Update initial version to 1"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Add read_only data column"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Migrate logging ds to loki ds"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Update json_data with nulls"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create api_key table"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="add index api_key.account_id"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="add index api_key.key"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="add index api_key.account_id_name"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="drop index IDX_api_key_account_id - v1"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="drop index UQE_api_key_key - v1"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="drop index UQE_api_key_account_id_name - v1"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Rename table api_key to api_key_v1 - v1"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create api_key table v2"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create index IDX_api_key_org_id - v2"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create index UQE_api_key_key - v2"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create index UQE_api_key_org_id_name - v2"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="copy api_key v1 to v2"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Drop old table api_key_v1"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Update api_key table charset"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Add expires to api_key table"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create dashboard_snapshot table v4"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="drop table dashboard_snapshot_v4 #1"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create dashboard_snapshot table v5 #2"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create index UQE_dashboard_snapshot_key - v5"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create index UQE_dashboard_snapshot_delete_key - v5"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create index IDX_dashboard_snapshot_user_id - v5"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="alter dashboard_snapshot to mediumtext v2"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Update dashboard_snapshot table charset"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Add column external_delete_url to dashboard_snapshots table"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create quota table v1"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create index UQE_quota_org_id_user_id_target - v1"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Update quota table charset"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create plugin_setting table"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create index UQE_plugin_setting_org_id_plugin_id - v1"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Add column plugin_version to plugin_settings"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Update plugin_setting table charset"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create session table"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Drop old table playlist table"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Drop old table playlist_item table"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create playlist table v2"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create playlist item table v2"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Update playlist table charset"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Update playlist_item table charset"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="drop preferences table v2"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="drop preferences table v3"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create preferences table v3"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Update preferences table charset"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Add column team_id in preferences"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Update team_id column values in preferences"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create alert table v1"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="add index alert org_id & id "
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="add index alert state"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="add index alert dashboard_id"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Create alert_rule_tag table v1"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Add unique index alert_rule_tag.alert_id_tag_id"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create alert_notification table v1"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Add column is_default"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Add column frequency"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Add column send_reminder"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Add column disable_resolve_message"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="add index alert_notification org_id & name"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Update alert table charset"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Update alert_notification table charset"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create notification_journal table v1"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="add index notification_journal org_id & alert_id & notifier_id"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="drop alert_notification_journal"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create alert_notification_state table v1"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="add index alert_notification_state org_id & alert_id & notifier_id"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Add for to alert table"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Add column uid in alert_notification"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Update uid column values in alert_notification"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Add unique index alert_notification_org_id_uid"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Remove unique index org_id_name"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Drop old annotation table v4"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create annotation table v5"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="add index annotation 0 v3"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="add index annotation 1 v3"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="add index annotation 2 v3"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="add index annotation 3 v3"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="add index annotation 4 v3"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Update annotation table charset"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Add column region_id to annotation table"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Drop category_id index"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Add column tags to annotation table"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Create annotation_tag table v2"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Add unique index annotation_tag.annotation_id_tag_id"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Update alert annotations and set TEXT to empty"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Add created time to annotation table"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Add updated time to annotation table"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Add index for created in annotation table"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Add index for updated in annotation table"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Convert existing annotations from seconds to milliseconds"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Add epoch_end column"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Add index for epoch_end"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Make epoch_end the same as epoch"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Move region to single row"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create test_data table"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create dashboard_version table v1"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="add index dashboard_version.dashboard_id"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="add unique index dashboard_version.dashboard_id and dashboard_version.version"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Set dashboard version to 1 where 0"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="save existing dashboard data in dashboard_version table v1"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="alter dashboard_version.data to mediumtext v1"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create team table"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="add index team.org_id"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="add unique index team_org_id_name"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create team member table"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="add index team_member.org_id"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="add unique index team_member_org_id_team_id_user_id"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Add column email to team table"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Add column external to team_member table"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Add column permission to team_member table"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create dashboard acl table"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="add index dashboard_acl_dashboard_id"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="add unique index dashboard_acl_dashboard_id_user_id"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="add unique index dashboard_acl_dashboard_id_team_id"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="save default acl rules in dashboard_acl table"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create tag table"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="add index tag.key_value"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create login attempt table"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="add index login_attempt.username"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="drop index IDX_login_attempt_username - v1"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Rename table login_attempt to login_attempt_tmp_qwerty - v1"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create login_attempt v2"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create index IDX_login_attempt_username - v2"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="copy login_attempt v1 to v2"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="drop login_attempt_tmp_qwerty"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create user auth table"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create index IDX_user_auth_auth_module_auth_id - v1"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="alter user_auth.auth_id to length 190"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Add OAuth access token to user_auth"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Add OAuth refresh token to user_auth"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Add OAuth token type to user_auth"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Add OAuth expiry to user_auth"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="Add index to user_id column in user_auth"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create server_lock table"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="add index server_lock.operation_uid"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create user auth token table"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="add unique index user_auth_token.auth_token"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="add unique index user_auth_token.prev_auth_token"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="create cache_data table"
t=2019-10-15T15:35:05+0000 lvl=dbug msg="Skipping migration: Already executed" logger=migrator id="add unique index cache_data.cache_key"
t=2019-10-15T15:35:05+0000 lvl=eror msg="Server shutdown" logger=server reason="Service init failed: pq: cannot set transaction read-write mode during recovery"

# grafana-server -v       
Version 6.4.2 (commit: 443a0ba, branch: HEAD)
# /usr/pgsql-12/bin/postmaster --version
postgres (PostgreSQL) 12.0

Grafana remote cache type is set to memcached.

[nkondratyk]

Hi

Do you think that we can proceed with it somehow? It`s really important feature when you have Grafana HA

[bergquist]

Any implementation suggestions or pull requests would be more than welcome.

[Antanukas]

+1 for being able to operate with read only connection.
Couple of things I have noticed that breaks:

• Provisioning step at startup. Its possible to bypass it by not having any dashboards, datasources, plugins in /etc/grafana/provisioning
• Remote Cache. Dashboards seem to stop loading when cache key expires for user sessions because code tries to update it.

HA guide advises to configure database for high availability. One of possible techniques can be to fallback to read only replica in case master goes down. But the problem is that Grafana has no official support for read only connections and it's unclear if future version will not break somehow. The fact that currently its kinda usable can change any time.