Alert fails on dashboard using opensearch datasource with TLS client auth enabled

[devesh2k7]

When testing an alert created using Opensearch as datasource for a dashboard panel, we get an error that occurs while testing the alert rule that is configured:

tsdb.HandleRequest() error rpc error: code = Unknown desc = invalid character 'U' looking for beginning of value

We have configured the Opensearch datasource used in Grafana as below.

datasources:
- name: opensearch-ds
type: grafana-opensearch-datasource
version: 2
access: proxy
orgId: 1
url: {{ .Values.opensearch.url }}
jsonData:
interval: Daily
timeField: "@timestamp"
timeInterval: "10m"
tlsAuth: true
tlsAuthWithCACert: true
database: "[opensearch-ds-]YYYY.MM.DD"
version: "1.0.0"
tlsSkipVerify: true
secureJsonData:
tlsCACert: $ENV_VAR_1
tlsClientCert: $ENV_VAR_2
tlsClientKey: $ENV_VAR_3

Please help us in getting this issue fixed at the earliest.

Grafana version: 7.5.17
Opensearch Datasource Plugin version: 1.1.2
Opensearch version: 2.4.1

Note: The alert does not work and it fails with same error even with the latest Grafana version 9.3.6 and the latest Opensearch datasource plugin version 2.0.5. 

Please note although that we want the fix for the older version of Grafana (v7.5.17) and Opensearch datasource plugin (v1.1.2) as mentioned above in the issue description.

[devesh2k7]

Just to also mention, the alert does not work and it fails with same error even with the latest Grafana version 9.3.6 and the latest Opensearch datasource plugin version 2.0.5.

Please note although that we want the fix for the older version of Grafana (v7.5.17) and Opensearch datasource plugin (v1.1.2) as mentioned above in the issue description.

[sarahzinger]

@devesh2k7 thanks for that! A few more clarifying questions for when you get a chance:

• do you know if this error only happens when TLS is enabled?
• do queries load successfully and it's just alerting that fails?
• do you get the same error for any kind of opensearch alert that you're trying to make or is it limited to a particular query?
• did this used to work for you at some point and suddenly stop working? (ex after upgrading something?)

and to clarify are these the versions you're seeing the issue on?
opensearch 1.2 https://opensearch.org/docs/1.2/
with the grafana opensearch plugin on 2.1.0 https://github.com/grafana/opensearch-datasource/releases/tag/v2.1.0
and grafana 9.3.6 https://github.com/grafana/grafana/releases/tag/v9.3.6

[devesh2k7]

@sarahzinger Thanks for acknowledging! Please find the responses inline (highlighted in bold):

• do you know if this error only happens when TLS is enabled?
  Yes, this error occurs only when TLS Client Auth is enabled and skipTlsVerify is enabled as well.

• do queries load successfully and it's just alerting that fails?
  Yes, queries load successfully in query section and data is also populated in dashboard panel, and its just alerting that fails

• do you get the same error for any kind of opensearch alert that you're trying to make or is it limited to a particular query?
  This error is not limited to a particular query. There is same error in alerting for all opensearch dashboards.

• did this used to work for you at some point and suddenly stop working? (ex after upgrading something?)
  This has not worked ever after we have switched the datasource from elasticsearch to opensearch.

• and to clarify are these the versions you're seeing the issue on?
  opensearch 1.2 https://opensearch.org/docs/1.2/
  with the grafana opensearch plugin on 2.1.0 https://github.com/grafana/opensearch-datasource/releases/tag/v2.1.0
  and grafana 9.3.6 https://github.com/grafana/grafana/releases/tag/v9.3.6

  We have seen the same issue on both the versions that we mentioned earlier. Please find the versions below having the same issue:

  Grafana version: 7.5.17
  Opensearch Datasource Plugin version: 1.1.2
  Opensearch version: 2.4.1

  Grafana version: 9.3.6
  Opensearch Datasource Plugin version: 2.0.5
  Opensearch version: 2.4.1

  Please note that we are currently using Grafana v7.5.17, Opensearch Datasource Plugin version: 1.1.2, as mentioned above
  in the first section, so we want the fix for this Grafana version.

[fridgepoet]

Hi @devesh2k7, thanks for all the responses.

• What was your exact process to trigger the error in 9.3.6, please?
• Do you know if the error will occur when you use basic auth as well?

[devesh2k7]

Hi @fridgepoet, please find the responses to the queries below:

• What was your exact process to trigger the error in 9.3.6, please?

  We have tested on a dashboard panel linked to a custom datasource of type opensearch with TLS Client Auth configured as the authentication method as shown in the first screenshot below.

  The query section on the panel shows the query normally that allows to create alerts (shown in second screenshot). We have then created an alert (from Alert tab) with the required parameters and upon testing the alert rule, the same error is triggered (shown in the third and fourth screenshots below).

• Do you know if the error will occur when you use basic auth as well?

  The error will not occur when using Basic Auth. We have tested this part as well, and found that alert is triggered without any error.

[fridgepoet]

Hi @devesh2k7 thanks for reporting this issue and providing all the information!
I believe the keys and certificates were not being transferred in the alert queries. The fix should be in the next release (estimated this week).