Opensearch plugin doesn't support using Grafana configured time zone for ES queries

[thislyc]

While using Grafana for data visualization, I noticed that the Opensearch plugin doesn't support using the time zone configured in Grafana for ES queries. Regardless of how I change the time zone settings in Grafana, the Opensearch plugin always uses the UTC time zone for queries.

Expected result: The Opensearch plugin should support using the time zone configured in Grafana for ES queries to ensure data accuracy and consistency.

Steps to reproduce:

Set different time zones in Grafana, such as UTC+8, UTC-5, etc.
Create a panel based on the Opensearch data source and set the time zone to "auto".
Observe the time range and display of the query results.
Actual result: Regardless of the time zone set in Grafana, the Opensearch plugin always uses the UTC time zone for queries, resulting in inaccurate or inconsistent data.

Suggested solution: Add support for using the time zone configured in Grafana for ES queries to the Opensearch plugin to ensure data accuracy and consistency. An option can be added in the plugin settings to allow users to choose whether to use the time zone configured in Grafana for ES queries. Additionally, updating the plugin documentation to explicitly state that the plugin doesn't support using the time zone configured in Grafana for ES queries can be considered.

[tonypowa]

👋 Thanks for creating this issue, @

I’m forwarding this issue to the observability squad for review

[sarahzinger]

@thislyc thanks for reporting your issue, we're trying to recreate it on our end. Do you know:

• What version of Grafana you're running
• What version of the open search plugin you're running?
• What sort of timerange you have selected (ex: relative time of last 5 min vs a specific time selection)

I looked in main grafana and noticed there does seem to be an open issue around relative timezones, so it's possible you're running into that?: grafana/grafana#59665

I believe this plugin works the same as all grafana plugins in which it:

• takes the selected time range in the user's timezone and converts that to the same time selection but in the utc equivalent
• a query is then made with that utc selection
• the results are then shown to the user in the user's timezone

Can you explain more how converting a query to UTC and back would result in inaccurate or inconsistent data? I would expect the opposite?

[thislyc]

• What version of Grafana you're running
  v9.2.5

• What version of the open search plugin you're running?
  2.1.0

• What sort of timerange you have selected (ex: relative time of last 5 min vs a specific time selection)

last 6 hour

I am in UTC+8 time zone, no matter how I modify the time zone settings on the dashboard
I find that there is always a delay in the data
But the same dashboard uses the ElasticSearch plug-in in grafana 8.5, and there is no time delay

[thislyc]

[sarahzinger]

@thislyc thanks for that! Unfortunately I still haven't been able to reproduce this, so a few more questions:

• how are you running grafana? Is it self hosted, on Grafana Cloud etc?
• can you talk me through your timezone settings for server, organization, team and personal: https://grafana.com/docs/grafana/latest/administration/organization-preferences/#organization-preferences
• Does this seem to happen for all open search queries or a particular type of query?
• What version of opensearch are you running?

[thislyc]

• how are you running grafana? Is it self hosted, on Grafana Cloud etc?

It self hosted

• can you talk me through your timezone settings for server, organization, team and personal:

• Does this seem to happen for all open search queries or a particular type of query?

all open search queries

• What version of opensearch are you running?

In fact I don't use opensearch, but ElasticSearch6.5

[sarahzinger]

Thanks for that @thislyc one more clarifying q are you running Elastic Search 6.5 through Amazon's Open Search Service or is it self hosted?

[thislyc]

Thanks for that @thislyc one more clarifying q are you running Elastic Search 6.5 through Amazon's Open Search Service or is it self hosted?

Self hosted

[kevinwcyu]

Hi @thislyc, we haven't been able to reproduce this yet.

Create a panel based on the Opensearch data source and set the time zone to "auto".

Do you have a screenshot of setting the time zone to "auto"? For the time zone picker, there's only a "Default" option.

Can you share the dashboard JSON? You can do so as follows:

1. Open the dashboard you want to export.
2. Click the Share icon.
3. Click Export.
4. Toggle on the "Export for sharing externally" option
5. Click Save to file.

[sarahzinger]

Going to close this since we haven't been able to reproduce it and it's been a while. But if you're still running into issues please feel free to make a new issue with the dashboard json as @kevinwcyu has suggested and we will take another look!