Grafana Explorer not working with OCI plugin

[rjanci]

I've deployed OCI plugin into Grafana Cloud and unfortunately the Explorer functionality is not working with the error attached.
This is limiting us to fully use OCI in our environment.
I would like to ask you to fix this issue.
Thank you.

[rjanci]

Just adding that I tested the issue not only on Grafana Cloud but also on standalone installed instance with the same behaviour.

[mamorett]

Hi. I tried with three different installations of Grafana, standalone, in kubernetes and Grafana cloud and the Explore worked as expected. My suggestions then are: please check if you have the latest version of the plugin installed., you should have v 5.0.3. In case you are running an older version, please try to update and then close the browser, clean browser cache and retry. Also, please double check your datasource configuration, if it is configured correctly. As said, I tested Explore in different installations and using single and multitenancy mode and it worked.

[rjanci]

Hi

I see some issue in our OCI policy config.
We used documentation ( https://github.com/oracle/oci-grafana-metrics/blob/master/docs/linux.md ) and following what is stated there is probably not enough.
• allow group grafana to read metrics in tenancy
• allow group grafana to read compartments in tenancy

I’ve added
• Allow group grafana_grp to inspect metrics in tenancy

It didn’t help.
Then I added
• Allow group grafana_grp to read metrics in compartment id ocid1.tenancy.oc1..aaaaaaaa........
• Allow group grafana_grp to read compartments in compartment id ocid1.tenancy.oc1..aaaaaaaa........
• Allow group grafana_grp to inspect metrics in compartment id ocid1.tenancy.oc1..aaaaaaaa........

It helped but only for my Always Free tenancy but not for our production one.
Please can you share what you use exactly in your policy for your grafana user/group?

[rjanci]

Adding part of grafana logs when trying to access explorer
logger=context userId=0 orgId=0 uname= t=2024-02-10T08:08:57.237788536Z level=info msg="Request Completed" method=GET path=/api/live/ws status=401 remote_addr=[::1] time_ms=0 duration=609.87µs size=105 referer= handler=/api/live/ws logger=plugin.oci-metrics-datasource t=2024-02-10T08:09:01.880758183Z level=error msg=plugin QueryData=oci-metrics-datasource logger=plugin.oci-metrics-datasource t=2024-02-10T08:09:01.880913343Z level=error msg=plugin.query query="query initiated for A" logger=plugin.oci-metrics-datasource t=2024-02-10T08:09:01.881014202Z level=error msg=client GetMetricDataPoints="fetching the metrics datapoints under compartment '' for query 'select metric[1m].avg()'" logger=plugin.oci-metrics-datasource t=2024-02-10T08:09:01.881108929Z level=error msg=GetTenancyAccessKey Validtakey=DEFAULT/ logger=plugin.oci-metrics-datasource t=2024-02-10T08:09:01.881374527Z level=error msg=client GetMetricDataPoints="host is invalid. parse \"https://telemetry.select region.oraclecloud.com\": invalid character \" \" in host name"

[kz4d1x]

Hi, the verb "read" in the policy statement includes the "inspect" permissions. Please see the details for monitoring: https://docs.oracle.com/en-us/iaas/Content/Identity/Reference/monitoringpolicyreference.htm
Are there additional restrictions applied to the "grafana_grp" group?

[mamorett]

in the log you provided I read this : https://telemetry.select region.oraclecloud.com": invalid character " " in host name". That means the region was not chosen and/or the datasource setup was not completed. Again, please double check your datasource configuration because there are probably some missing or wrong parameters there.

[Maros112358]

Hello, we have found a workaround. Plugin worked without selecting a tenancy at all, or by selecting a multi tenancy mode. Below I provide relevant logs from grafana backend and configuration/error screenshots.

No tenancy mode selected works fine

Datasource configuration

Grafana backend logs

logger=plugin.oci-metrics-datasource t=2024-02-21T13:02:12.619959923Z level=error msg=plugin NewOCIDatasource=8
logger=plugin.oci-metrics-datasource t=2024-02-21T13:02:12.620079808Z level=error msg=plugin dsSettings.Environment="dsSettings.Environment: local"
logger=plugin.oci-metrics-datasource t=2024-02-21T13:02:12.620116397Z level=error msg=plugin dsSettings.TenancyMode="dsSettings.TenancyMode: "
logger=plugin.oci-metrics-datasource t=2024-02-21T13:02:12.621333094Z level=error msg=plugin CheckHealth=oci-metrics-datasource
logger=plugin.oci-metrics-datasource t=2024-02-21T13:02:12.621585288Z level=error msg=client TestConnectivity="testing the OCI connectivity"
logger=plugin.oci-metrics-datasource t=2024-02-21T13:02:12.621826351Z level=error msg=TestConnectivity ConfigKey=DEFAULT/ TestingTenancyOCID=xxx
logger=plugin.oci-metrics-datasource t=2024-02-21T13:02:13.435487548Z level=error msg=TestConnectivity OK=200 ConfigKey=DEFAULT/
logger=authn.service t=2024-02-21T13:02:13.624262642Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=context userId=0 orgId=0 uname= t=2024-02-21T13:02:13.624342412Z level=warn msg=Unauthorized error="user token not found" remote_addr=[::1] traceID=
logger=context userId=0 orgId=0 uname= t=2024-02-21T13:02:13.624394499Z level=info msg="Request Completed" method=GET path=/api/live/ws status=401 remote_addr=[::1] time_ms=0 duration=635.705µs size=40 referer= handler=/api/live/ws
logger=authn.service t=2024-02-21T13:02:17.454758034Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=context userId=0 orgId=0 uname= t=2024-02-21T13:02:17.455131325Z level=warn msg=Unauthorized error="user token not found" remote_addr=[::1] traceID=
logger=context userId=0 orgId=0 uname= t=2024-02-21T13:02:17.455217727Z level=info msg="Request Completed" method=GET path=/api/live/ws status=401 remote_addr=[::1] time_ms=0 duration=940.517µs size=40 referer= handler=/api/live/ws
logger=authn.service t=2024-02-21T13:02:25.461950528Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=context userId=0 orgId=0 uname= t=2024-02-21T13:02:25.462034736Z level=warn msg=Unauthorized error="user token not found" remote_addr=[::1] traceID=
logger=context userId=0 orgId=0 uname= t=2024-02-21T13:02:25.462088237Z level=info msg="Request Completed" method=GET path=/api/live/ws status=401 remote_addr=[::1] time_ms=0 duration=619.454µs size=40 referer= handler=/api/live/ws
logger=plugin.oci-metrics-datasource t=2024-02-21T13:02:29.925156544Z level=error msg=client GetSubscribedRegions="fetching the subscribed region for tenancy: "
logger=plugin.oci-metrics-datasource t=2024-02-21T13:02:29.925249208Z level=error msg=GetTenancyAccessKey Validtakey=DEFAULT/
logger=plugin.oci-metrics-datasource t=2024-02-21T13:02:29.925295515Z level=error msg=client GetSubscribedRegionstakey="fetching the subscribed region for tenancy OCID: xxxa"
logger=plugin.oci-metrics-datasource t=2024-02-21T13:02:30.327525602Z level=error msg=client GetSubscribedRegionstakey="fetching the subscribed region for regioname: eu-amsterdam-1"
logger=plugin.oci-metrics-datasource t=2024-02-21T13:02:30.327748821Z level=error msg=client GetSubscribedRegionstakey="fetching the subscribed region for regioname: eu-frankfurt-1"
logger=plugin.oci-metrics-datasource t=2024-02-21T13:02:32.314570194Z level=error msg=client GetCompartments="fetching the sub-compartments for tenancy: "
logger=plugin.oci-metrics-datasource t=2024-02-21T13:02:32.314754491Z level=error msg=GetTenancyAccessKey Validtakey=DEFAULT/
logger=plugin.oci-metrics-datasource t=2024-02-21T13:02:33.820717243Z level=error msg=client GetNamespaceWithMetricNames="fetching the metric names along with namespaces under compartment: xxx"
logger=plugin.oci-metrics-datasource t=2024-02-21T13:02:33.82162505Z level=error msg=GetTenancyAccessKey Validtakey=DEFAULT/
logger=plugin.oci-metrics-datasource t=2024-02-21T13:02:33.821844863Z level=error msg=client.utils listMetricsMetadataPerRegion="Data fetch start by calling list metrics API for a particular regions"
logger=authn.service t=2024-02-21T13:02:34.036493382Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=context userId=0 orgId=0 uname= t=2024-02-21T13:02:34.036573723Z level=warn msg=Unauthorized error="user token not found" remote_addr=[::1] traceID=
logger=context userId=0 orgId=0 uname= t=2024-02-21T13:02:34.036626552Z level=info msg="Request Completed" method=GET path=/api/live/ws status=401 remote_addr=[::1] time_ms=0 duration=699.013µs size=40 referer= handler=/api/live/ws
logger=plugin.oci-metrics-datasource t=2024-02-21T13:02:35.895554048Z level=error msg=client GetResourceGroups="fetching the resource groups under compartment 'xxx' for namespace 'oracle_oci_database'"
logger=plugin.oci-metrics-datasource t=2024-02-21T13:02:35.895646732Z level=error msg=GetTenancyAccessKey Validtakey=DEFAULT/
logger=plugin.oci-metrics-datasource t=2024-02-21T13:02:35.895700393Z level=error msg=client.utils listMetricsMetadataPerRegion="Data fetch start by calling list metrics API for a particular regions"
logger=plugin.oci-metrics-datasource t=2024-02-21T13:02:37.879928931Z level=error msg=client GetResourceGroups="fetching the resource groups under compartment 'xxx' for namespace 'oracle_oci_database'"
logger=plugin.oci-metrics-datasource t=2024-02-21T13:02:37.880053934Z level=warn msg=client GetResourceGroups="getting the data from cache"
logger=authn.service t=2024-02-21T13:02:42.029672149Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=context userId=0 orgId=0 uname= t=2024-02-21T13:02:42.029884398Z level=warn msg=Unauthorized error="user token not found" remote_addr=[::1] traceID=
logger=context userId=0 orgId=0 uname= t=2024-02-21T13:02:42.030005906Z level=info msg="Request Completed" method=GET path=/api/live/ws status=401 remote_addr=[::1] time_ms=0 duration=884.602µs size=40 referer= handler=/api/live/ws
logger=plugin.oci-metrics-datasource t=2024-02-21T13:02:43.079331909Z level=error msg=plugin QueryData=oci-metrics-datasource-oci-user-with-no-tenancy-mode-selected
logger=plugin.oci-metrics-datasource t=2024-02-21T13:02:43.079911367Z level=error msg=plugin.query query="query initiated for A"
logger=plugin.oci-metrics-datasource t=2024-02-21T13:02:43.080121061Z level=error msg=client GetMetricDataPoints="fetching the metrics datapoints under compartment 'xxx' for query 'CpuUtilization[1m].avg()'"
logger=plugin.oci-metrics-datasource t=2024-02-21T13:02:43.080314084Z level=error msg=GetTenancyAccessKey Validtakey=DEFAULT/
logger=plugin.oci-metrics-datasource t=2024-02-21T13:02:43.125196407Z level=error msg=client fetchFromCache="fetching from cache"
logger=plugin.oci-metrics-datasource t=2024-02-21T13:02:43.125523973Z level=error msg=client GetTags="fetching the tags for namespace 'oracle_oci_database'"
logger=plugin.oci-metrics-datasource t=2024-02-21T13:02:43.125720143Z level=info msg=client getResourceTags="Resource tags got for region-eu-frankfurt-1"
logger=plugin.oci-metrics-datasource t=2024-02-21T13:02:43.125774275Z level=info msg=client GetMetricDataPoints="Metric datapoints got for region-eu-frankfurt-1"

Single tenancy causes error

Datasource configuration

Error in Grafana Explorer

Grafana backend logs

logger=authn.service t=2024-02-21T13:09:40.303530977Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=context userId=0 orgId=0 uname= t=2024-02-21T13:09:40.303624563Z level=warn msg=Unauthorized error="user token not found" remote_addr=[::1] traceID=
logger=context userId=0 orgId=0 uname= t=2024-02-21T13:09:40.303696107Z level=info msg="Request Completed" method=GET path=/api/live/ws status=401 remote_addr=[::1] time_ms=0 duration=680.789µs size=40 referer= handler=/api/live/ws
logger=context userId=1 orgId=1 uname=admin t=2024-02-21T13:09:43.626722023Z level=info msg="Request Completed" method=GET path=/api/live/ws status=-1 remote_addr=127.0.0.1 time_ms=1 duration=1.861148ms size=0 referer= handler=/api/live/ws
logger=plugin.oci-metrics-datasource t=2024-02-21T13:09:43.901343412Z level=error msg=plugin QueryData=oci-metrics-datasource-oci-user-with-single-tenancy-mode
logger=plugin.oci-metrics-datasource t=2024-02-21T13:09:43.901414305Z level=error msg=plugin.query query="query initiated for A"
logger=plugin.oci-metrics-datasource t=2024-02-21T13:09:43.901447097Z level=error msg=client GetMetricDataPoints="fetching the metrics datapoints under compartment '' for query 'select metric[1m].avg()'"
logger=plugin.oci-metrics-datasource t=2024-02-21T13:09:43.901866015Z level=error msg=GetTenancyAccessKey Validtakey=DEFAULT/
logger=plugin.oci-metrics-datasource t=2024-02-21T13:09:43.902280344Z level=error msg=client GetMetricDataPoints="host is invalid. parse \"https://telemetry.select region.oraclecloud.com\": invalid character \" \" in host name"

Multitenancy mode works fine

Datasource configuration

Grafana backend logs

logger=authn.service t=2024-02-21T13:14:20.705648243Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=context userId=0 orgId=0 uname= t=2024-02-21T13:14:20.705857035Z level=warn msg=Unauthorized error="user token not found" remote_addr=[::1] traceID=
logger=context userId=0 orgId=0 uname= t=2024-02-21T13:14:20.705969146Z level=info msg="Request Completed" method=GET path=/api/live/ws status=401 remote_addr=[::1] time_ms=0 duration=821.462µs size=40 referer= handler=/api/live/ws
logger=context userId=1 orgId=1 uname=admin t=2024-02-21T13:14:21.223887036Z level=info msg="Request Completed" method=GET path=/api/live/ws status=-1 remote_addr=127.0.0.1 time_ms=2 duration=2.025053ms size=0 referer= handler=/api/live/ws

[mamorett]

Hi guys. I did the test, and I confirm it happens also in my lab when using Grafana 10.3.3 ! So they definitively changed something in that version. For now I would suggest to install Grafana 10.1.x using docker or bare metal, until we clarify with Grafana what makes this error happens

[mamorett]

We have a PR which should fix this issue for v 10.2 and 10.3 of Grafana. The PR is currently under review: #260

[mamorett]

Version 5.1.0 of the plugin fixes the issue.

[rjanci]

Great, we will test and let know, Thx From: Mattia Moretti ***@***.***> Sent: Friday, March 8, 2024 11:29 AM To: oracle/oci-grafana-metrics ***@***.***> Cc: Rastislav Janci ***@***.***>; Author ***@***.***> Subject: Re: [oracle/oci-grafana-metrics] Grafana Explorer not working with OCI plugin (Issue #253) Closed #253<#253> as completed. - Reply to this email directly, view it on GitHub<#253 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BGBE2AOCGG44NHYMEYUDRWTYXGHFXAVCNFSM6AAAAABDBIC5KGVHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJSGA2TGOJTGUZTMOA>. You are receiving this because you authored the thread.Message ID: ***@***.******@***.***>>

[Maros112358]

Hello, we tested the new plugin and confirm that the single tenancy mode works. Thank you.