Support data source health check endpoint introduced with Grafana 9

[amotl]

@jangaraj mentioned at #20 (comment), that Grafana 9 features a dedicated data source health check endpoint ¹. This patch implements the corresponding client support.

Footnotes

1. https://grafana.com/docs/grafana/latest/developers/http_api/data_source/#check-data-source-health ↩

[amotl]

Report

When running the example, I am receiving this outcome. Apparently, the /health still responds with 404 Not Found on my end.

$ python examples/datasource-health.py --type=prometheus --url=http://host.docker.internal:9090
2022-06-20 15:39:20,230 [urllib3.connectionpool             ] DEBUG  : Starting new HTTP connection (1): localhost:3000
2022-06-20 15:39:20,233 [urllib3.connectionpool             ] DEBUG  : http://localhost:3000 "GET /api/health HTTP/1.1" 200 70
2022-06-20 15:39:20,233 [__main__                           ] INFO   : Connected to Grafana version 9.0.0 at http://localhost:3000
2022-06-20 15:39:20,234 [__main__                           ] INFO   : Creating data source 'probe-prometheus'
2022-06-20 15:39:20,248 [urllib3.connectionpool             ] DEBUG  : http://localhost:3000 "POST /api/datasources HTTP/1.1" 409 104
2022-06-20 15:39:20,248 [__main__                           ] INFO   : Data source already exists: {'message': 'data source with the same name already exists', 'traceID': '00000000000000000000000000000000'}. Updating.
2022-06-20 15:39:20,262 [urllib3.connectionpool             ] DEBUG  : http://localhost:3000 "GET /api/datasources/name/probe-prometheus HTTP/1.1" 200 326
2022-06-20 15:39:20,294 [urllib3.connectionpool             ] DEBUG  : http://localhost:3000 "PUT /api/datasources/4 HTTP/1.1" 200 405
2022-06-20 15:39:20,310 [urllib3.connectionpool             ] DEBUG  : http://localhost:3000 "GET /api/datasources/4 HTTP/1.1" 200 326
2022-06-20 15:39:20,326 [urllib3.connectionpool             ] DEBUG  : http://localhost:3000 "GET /api/datasources/uid/NJoLtZq7k/health HTTP/1.1" 404 68
2022-06-20 15:39:20,327 [__main__                           ] ERROR  : Data source health check for uid=NJoLtZq7k failed: Client Error 404: Not found. Response: {'message': 'Not found', 'traceID': '00000000000000000000000000000000'}

Traceback (most recent call last):
  File "/Users/amo/dev/panodata/sources/grafana-client/examples/datasource-health.py", line 159, in <module>
    run_healthcheck(grafana_client, grafana_version=grafana_version)
  File "/Users/amo/dev/panodata/sources/grafana-client/examples/datasource-health.py", line 135, in run_healthcheck
    health_info = health_inquiry(grafana, datasource, grafana_version=grafana_version)
  File "/Users/amo/dev/panodata/sources/grafana-client/examples/datasource-health.py", line 82, in health_inquiry
    health = grafana.datasource.health(datasource_uid=datasource_uid)
  File "/Users/amo/dev/panodata/sources/grafana-client/grafana_client/elements/datasource.py", line 31, in health
    r = self.client.GET(path)
  File "/Users/amo/dev/panodata/sources/grafana-client/grafana_client/client.py", line 131, in __request_runnner
    raise GrafanaClientError(
grafana_client.client.GrafanaClientError: Client Error 404: Not found

Evaluation

http://localhost:3000/api/datasources/uid/NJoLtZq7k/health looks sane, requesting http://localhost:3000/api/datasources/uid/NJoLtZq7k actually works.

Through this log message, I am sure that I am on Grafana 9, running locally through Docker. The version string is inquired from the instance.

INFO   : Connected to Grafana version 9.0.0 at http://localhost:3000

Thoughts

Am I missing something on this matter? Would this endpoint need to be explicitly enabled in the configuration? Is it enterprise only?

[jangaraj]

I would say doc is not correct - grafana/grafana@6bb843b#diff-5e6dbad9a85fb156660edecabd16951c3ebbf9f026c063a6504a79b82fbdaccfR607 - it looks like GET /api/datasources/:datasourceId/health was introduced and also deprecated at the same time. Anyway, Grafana starts to use uids everywhere, so GET /api/datasources/uid/:uid/health seems to be better API.

[amotl]

Thanks for your quick response. What is implemented right now is the /uid/:uid/health route, which responds with 404 Not Found. I also tried the /:datasourceId/health route, but the result was the same.

I must be doing something wrong.

[amotl]

I've just confirmed the behavior also with the current main image, like

export GRAFANA_VERSION=main

docker run --rm -it \
    --publish=3000:3000 \
    --env='GF_SECURITY_ADMIN_PASSWORD=admin' \
    grafana/grafana:${GRAFANA_VERSION}

Do those endpoints actually work on your end, @jangaraj?

[jangaraj]

Nope, but I'm trying to figure out how to configure that. It works here - https://play.grafana.org/api/datasources/uid/000000098/health

[amotl]

I also thought about checking play.grafana.org, thank you for making a start. From a few samples I made, the outcome is 50/50. Both 200 OK and 400 Bad Request responses are probably intended, but the 404 Not Found responses also happening over there let me conclude that this endpoint might not be thoroughly implemented for all data source types yet.

Works

200 OK

• TwinMaker: https://play.grafana.org/api/datasources/uid/qenjJQtnk/health

400 Bad Request

• Cloudwatch: https://play.grafana.org/api/datasources/uid/000000098/health

Not implemented yet?

404 Not Found

• Testdata: https://play.grafana.org/api/datasources/uid/000000051/health
• Prometheus: https://play.grafana.org/api/datasources/uid/000000008/health

[amotl]

[...] the 404 Not Found responses also happening over there let me conclude that this endpoint might not be thoroughly implemented for all data source types yet.

25ead33 adds a small adjustment to the example program in order to compensate for this. Now, when hitting 404 Not Found on the native data source health check endpoint, the procedure will fall back to the alternative, client-side implementation. We will need it for Grafana <9 anyway.