Odd query differences when selecting "by 1 minute" on the dashboard

[dazinator]

I have two charts on a dashboard, side by side.
They both have the same signal - essentially showing requests made to a particular URL.
They both select the count(*) of events from that signal.
One groups by a value found in the RequestPath. RequestPath is a property of the log. The value is always found at a fixed postition in the RequestPath and is a GUID so is a fixed length. I use "Substring(RequestPath, 34, 32)" to grab this value.
The other chart just groups by a different property of the log called "Tenant."
When these charts are placed side by side, I can select different combinations of time period for example:-

• "Last 8 hours by 5 mins"
• "Last 8 hours by 15 mins"
  Both charts show the same "total" number of events irrespective of which combinations I chose for time period. This is the "correct" / "expected" behaviour that I desire.
  .. Except for when I chose "by minute". When I chose by "minute" the charts disagree. The one that groups by the "Tenant" log property shows a higher total count than the one that groups by the portion of the URL.

Here is a log event from the signal

Here is the two charts agreeing for example when selecting by 5 mins ("Job" label is the GUID extracted from the URL that I am grouping on)

here they are disagreeing when changing to 1 minute:

Am I missing something?

Here is the json definiton of the two charts:

 {
      "Id": "chart-101",
      "Title": "by Job",
      "SignalExpression": {
        "Kind": "Signal",
        "SignalId": "signal-97"
      },
      "Queries": [
        {
          "Id": "chartquery-102",
          "Measurements": [
            {
              "Value": "count(*)",
              "Label": "count"
            }
          ],
          "Where": null,
          "SignalExpression": null,
          "GroupBy": [
            "Substring(RequestPath, 34, 32)"
          ],
          "DisplayStyle": {
            "Type": "Bar",
            "LineFillToZeroY": false,
            "LineShowMarkers": true,
            "BarOverlaySum": true,
            "SuppressLegend": false,
            "Palette": "Default"
          },
          "Alerts": [],
          "Having": null,
          "OrderBy": [],
          "Limit": null
        }
      ],
      "DisplayStyle": {
        "WidthColumns": 6,
        "HeightRows": 2
      }
    },
    {
      "Id": "chart-118",
      "Title": "by Tenant",
      "SignalExpression": {
        "Kind": "Signal",
        "SignalId": "signal-97"
      },
      "Queries": [
        {
          "Id": "chartquery-119",
          "Measurements": [
            {
              "Value": "count(*)",
              "Label": "count"
            }
          ],
          "Where": null,
          "SignalExpression": null,
          "GroupBy": [
            "Tenant"
          ],
          "DisplayStyle": {
            "Type": "Bar",
            "LineFillToZeroY": false,
            "LineShowMarkers": true,
            "BarOverlaySum": true,
            "SuppressLegend": false,
            "Palette": "Default"
          },
          "Alerts": [],
          "Having": null,
          "OrderBy": [
            "Tenant"
          ],
          "Limit": null
        }
      ],
      "DisplayStyle": {
        "WidthColumns": 6,
        "HeightRows": 2
      }
    }

[nblumhardt]

Hi @dazinator - thanks for the heads-up.

My suspicion is that it's a timing issue - my first guess is that since the charts aren't rendered in the same request/transaction, they might show different data because of changes occurring on the server between the two requests.

Do the one-minute charts ever show the same value, if you refresh them every 10s for a minute or so?

And if you do the same for the five-minute charts, do they ever disagree?

Best regards,
Nick

[dazinator]

Do the one-minute charts ever show the same value, if you refresh them every 10s for a minute or so?

I observe that refreshing the charts continuously for for a few minutes, if they agreed to begin with, they continue to agree.
Refreshing charts where they disagree to begin with, for a few minutes, they continue to disagree.

And if you do the same for the five-minute charts, do they ever disagree?

Same holds as above, if they disagree to begin with they continue to disagree.

Here are some additional findings:

If I select "last 2 days", and just vary the "By" selection to any of the largest values (least granular) values:

• Day
• 8 hours
• hour
• 30 mins

Both charts agree.

But when I increase the granularity to "by 15" mins there is now a slight difference emerging:

.. and increase the granularity even further to "by 5 mins" there is now a larger difference emerging:

This pattern seems to emerge, irrespective of what I select for "last x" as long as the following is true:

1. There is a decent volume of events in the given selection:-

• e.g for "last 8 hours" I can go to max granularity of "by minute" - and in this case both charts do agree but there are only 25 events total in this case.
• The granularity i.e "by X" is relatively high (or the most granular).

[dazinator]

Its interesting to note that when changing the "by x" selection from 15 mins, to 5 mins" but keeping the "last X" as 2 days, the left hand chart in the images shown above drops its total count by approx 15k.
I can continue to switch backwards and forwards between "by 15 mins" and "by 5 mins" and reproduce this.

[nblumhardt]

Thanks for all of the info - we'll dig in deeper. Are you running 2021.x?

[dazinator]

Yep - 2021.1

[nblumhardt]

We've tracked this down and raised #1173 to address it - thanks for bringing this to our attention, @dazinator 🙏

[dazinator]

thanks!