Different values produced by quantilesDoublesSketchToQuantile with the same query

[RuochenLI]

Different values produced by quantilesDoublesSketchToQuantile with the same query

Affected Version

0.22.1

Description

I have a datasource that I'm trying to query using quantiles to get P60. But every time it produced different values

Sample json query:

{
  "dataSource": "datasource_name",
  "dimensions": [
    {
      "dimension": "__time",
      "outputName": "date",
      "outputType": "LONG",
      "type": "default"
    },
    {
      "dimension": "dimension1",
      "outputName": "dimension1",
      "outputType": "STRING",
      "type": "default"
    },
    {
      "dimension": "dimension2",
      "outputName": "dimension2",
      "outputType": "STRING",
      "type": "default"
    }
  ],
  "granularity": {
    "type": "all"
  },
  "filter": {
    "fields": [
      {
        "dimension": "dimension1",
        "values": [
          "dimension1_filter"
        ],
        "type": "in"
      },
      {
        "dimension": "dimension2",
        "values": [
          "dimension2_filter"
        ],
        "type": "in"
      }
    ],
    "type": "and"
  },
    "aggregations": [
        {
            "name": "quantile_quantity",
            "fieldName": "quantity",
            "type": "quantilesDoublesSketch"
        }
    ],
    "postAggregations": [
        {
            "name": "aggregated_quantity",
            "field": {
                "name": "aggregated_quantity",
                "fieldName": "quantile_quantity",
                "type": "fieldAccess"
            },
            "fraction": 0.6,
            "type": "quantilesDoublesSketchToQuantile"
        }
    ],
    "intervals": [
        "2022-07-10/2022-07-17"
    ],
  "queryType": "groupBy"
}

Sample of result

{
        "aggregated_quantity": 0.8879,
        "quantile_quantity": 7040
    }

{
        "aggregated_quantity": 0.9017,
        "quantile_quantity": 7040
    }

{
        "aggregated_quantity": 0.8984,
        "quantile_quantity": 7040
    }

{
        "aggregated_quantity": 0.905,
        "quantile_quantity": 7040
    }

It is quite confusing. Is there some configuration I'm missing or is it a known issue?

Thank you!
Ruochen

[LakshSingla]

As per my understanding, quantilesDoublesSketchToQuantile produces approximate results since it utilizes the Datasketches library to compute the result. From the results that you have posted above, it seems that they are not far off from each other and in an acceptable variability range.
From the doc for the function, I think setting and increasing the k should reduce the variability b/w the runs (at the expense of performance).

[gianm]

My understanding agrees with @LakshSingla. It is an approximate algorithm that may return slightly different numbers. Nevertheless, all numbers returned by the algorithm fall within the expected error bounds. Like @LakshSingla mentioned you can raise k if you need tighter error bounds. It will improve accuracy, at the cost of using more memory and processing time.

If you encounter further issues, please feel free to reopen this, or join us on Slack: https://druid.apache.org/community/