Exemplars: won't visualize unless timestamps are second-level precision

[juliaweingart]

What happened?

When Grafana calls /query_exemplars API to retrieve exemplars and visualize in a panel, the data points will only be visualized if the timestamps are second-level precision (10 digits to the left of the decimal point). When timestamps are returned in millisecond-level precision or nanosecond-level precision (13 or 16 digits to the left of the decimal), Grafana doesn't visualize. Bug is that Grafana should be able plot / interpret exemplars with any unix timestamp regardless of precision-level as long as the timestamp is within the query range.

WORKS:

[
        {
            "seriesLabels": {
                "__name__": "test_exemplar_metric_total",
                "instance": "localhost:8090",
                "job": "prometheus",
                "service": "bar"
            },
            "exemplars": [
                {
                    "labels": {
                        "id1": "EpTxMJ40fUus7aGY"
                    },
                    "value": "6",
                    "timestamp": 1600096945.479
                }
            ]
       }
]

DOESN'T WORK:

[
        {
            "seriesLabels": {
                "__name__": "test_exemplar_metric_total",
                "instance": "localhost:8090",
                "job": "prometheus",
                "service": "bar"
            },
            "exemplars": [
                {
                    "labels": {
                        "id1": "EpTxMJ40fUus7aGY"
                    },
                    "value": "6",
                    "timestamp": 1600096945479000
                }
            ]
       }
]

What did you expect to happen?

Exemplars to be plotted even with nanosecond unix timestamps

Did this work before?

I don't think so, but I haven't tried with earlier versions.

How do we reproduce it?

1. See above in "what happened" but mock a /exemplars response where timestamps are nanosecond-level with 16 digits.
2. Try to plot exemplars in a chart.

Is the bug inside a dashboard panel?

No response

Environment (with versions)?

Grafana: Grafana v9.5.6
OS: Windows
Browser: Edge

Grafana platform?

A downloaded binary

Datasource(s)?

No response

[usmangt]

Hi @juliaweingart,

Thanks for opening this issue.

Grafana's current latest stable version is 10.0.3. Can you please try to reproduce your case on the latest version and let us know?

We will be waiting for your feedback.

[juliaweingart]

@usmangt I just tried on Grafana 10.0.3 and it has the same issue.

[usmangt]

Thanks for the quick testing.

Can you please tell me which Data source you are using and find out about this issue?

Also, is it limited to a specific Data source or Visual panel or independent? (provide some examples if any).

[juliaweingart]

@usmangt I am using a Prometheus datasource and am working on a custom implementation of Prometheus API / exemplars storage. I found out about the issue as, by default, I was returning timestamps in 16 digits and wasn't seeing any exemplars visualized in Grafana. I have been mostly trying via explore tab or just a simple "Time Series" chart in a dashboard.

[bohandley]

Need to sync up with Prometheus to point us to an official response. Looks like from this issue that OpenTelemetry is moving towards Prometheus which indicates only supporting seconds. We will investigate to get a more clear response. @bboreham do you have any context on the seconds/nanoseconds issue?

[bboreham]

Prometheus timestamps are held in milliseconds. Generally when the unit is seconds, a time like “0.01” will work, meaning 10ms.