Add columnMappings to explain plan output by zachjsh · Pull Request #14187 · apache/druid

zachjsh · 2023-04-29T03:24:49Z

This change adds columnMappings to the native explain query mode explain plan output. This field is a list containing the mapping of queryColumn name to outputColumn name.

As an example for this rollup query (as given as an example in the MSQ docs)

INSERT INTO "kttm_rollup"

WITH kttm_data AS (
SELECT * FROM TABLE(
  EXTERN(
    '{"type":"http","uris":["https://static.imply.io/example-data/kttm-v2/kttm-v2-2019-08-25.json.gz"]}',
    '{"type":"json"}',
    '[{"name":"timestamp","type":"string"},{"name":"agent_category","type":"string"},{"name":"agent_type","type":"string"},{"name":"browser","type":"string"},{"name":"browser_version","type":"string"},{"name":"city","type":"string"},{"name":"continent","type":"string"},{"name":"country","type":"string"},{"name":"version","type":"string"},{"name":"event_type","type":"string"},{"name":"event_subtype","type":"string"},{"name":"loaded_image","type":"string"},{"name":"adblock_list","type":"string"},{"name":"forwarded_for","type":"string"},{"name":"language","type":"string"},{"name":"number","type":"long"},{"name":"os","type":"string"},{"name":"path","type":"string"},{"name":"platform","type":"string"},{"name":"referrer","type":"string"},{"name":"referrer_host","type":"string"},{"name":"region","type":"string"},{"name":"remote_address","type":"string"},{"name":"screen","type":"string"},{"name":"session","type":"string"},{"name":"session_length","type":"long"},{"name":"timezone","type":"string"},{"name":"timezone_offset","type":"long"},{"name":"window","type":"string"}]'
  )
))

SELECT
  FLOOR(TIME_PARSE("timestamp") TO MINUTE) AS __time,
  session,
  agent_category,
  agent_type,
  browser,
  browser_version,
  MV_TO_ARRAY("language") AS "language", -- Multi-value string dimension
  os,
  city,
  country,
  forwarded_for AS ip_address,

  COUNT(*) AS "cnt",
  SUM(session_length) AS session_length,
  APPROX_COUNT_DISTINCT_DS_HLL(event_type) AS unique_event_types
FROM kttm_data
WHERE os = 'iOS'
GROUP BY 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11
PARTITIONED BY HOUR
CLUSTERED BY browser, session

The plan with added columnMappings field is given as:

[
  {
    "query": {
      "queryType": "groupBy",
      "dataSource": {
        "type": "external",
        "inputSource": {
          "type": "http",
          "uris": [
            "https://static.imply.io/example-data/kttm-v2/kttm-v2-2019-08-25.json.gz"
          ]
        },
        "inputFormat": {
          "type": "json",
          "keepNullColumns": false,
          "assumeNewlineDelimited": false,
          "useJsonNodeReader": false
        },
        "signature": [
          {
            "name": "timestamp",
            "type": "STRING"
          },
          {
            "name": "agent_category",
            "type": "STRING"
          },
          {
            "name": "agent_type",
            "type": "STRING"
          },
          {
            "name": "browser",
            "type": "STRING"
          },
          {
            "name": "browser_version",
            "type": "STRING"
          },
          {
            "name": "city",
            "type": "STRING"
          },
          {
            "name": "continent",
            "type": "STRING"
          },
          {
            "name": "country",
            "type": "STRING"
          },
          {
            "name": "version",
            "type": "STRING"
          },
          {
            "name": "event_type",
            "type": "STRING"
          },
          {
            "name": "event_subtype",
            "type": "STRING"
          },
          {
            "name": "loaded_image",
            "type": "STRING"
          },
          {
            "name": "adblock_list",
            "type": "STRING"
          },
          {
            "name": "forwarded_for",
            "type": "STRING"
          },
          {
            "name": "language",
            "type": "STRING"
          },
          {
            "name": "number",
            "type": "LONG"
          },
          {
            "name": "os",
            "type": "STRING"
          },
          {
            "name": "path",
            "type": "STRING"
          },
          {
            "name": "platform",
            "type": "STRING"
          },
          {
            "name": "referrer",
            "type": "STRING"
          },
          {
            "name": "referrer_host",
            "type": "STRING"
          },
          {
            "name": "region",
            "type": "STRING"
          },
          {
            "name": "remote_address",
            "type": "STRING"
          },
          {
            "name": "screen",
            "type": "STRING"
          },
          {
            "name": "session",
            "type": "STRING"
          },
          {
            "name": "session_length",
            "type": "LONG"
          },
          {
            "name": "timezone",
            "type": "STRING"
          },
          {
            "name": "timezone_offset",
            "type": "LONG"
          },
          {
            "name": "window",
            "type": "STRING"
          }
        ]
      },
      "intervals": {
        "type": "intervals",
        "intervals": [
          "-146136543-09-08T08:23:32.096Z/146140482-04-24T15:36:27.903Z"
        ]
      },
      "virtualColumns": [
        {
          "type": "expression",
          "name": "v0",
          "expression": "timestamp_floor(timestamp_parse(\"timestamp\",null,'UTC'),'PT1M',null,'UTC')",
          "outputType": "LONG"
        },
        {
          "type": "expression",
          "name": "v1",
          "expression": "mv_to_array(\"language\")",
          "outputType": "ARRAY<STRING>"
        },
        {
          "type": "expression",
          "name": "v2",
          "expression": "'iOS'",
          "outputType": "STRING"
        }
      ],
      "filter": {
        "type": "selector",
        "dimension": "os",
        "value": "iOS"
      },
      "granularity": {
        "type": "all"
      },
      "dimensions": [
        {
          "type": "default",
          "dimension": "v0",
          "outputName": "d0",
          "outputType": "LONG"
        },
        {
          "type": "default",
          "dimension": "session",
          "outputName": "d1",
          "outputType": "STRING"
        },
        {
          "type": "default",
          "dimension": "agent_category",
          "outputName": "d2",
          "outputType": "STRING"
        },
        {
          "type": "default",
          "dimension": "agent_type",
          "outputName": "d3",
          "outputType": "STRING"
        },
        {
          "type": "default",
          "dimension": "browser",
          "outputName": "d4",
          "outputType": "STRING"
        },
        {
          "type": "default",
          "dimension": "browser_version",
          "outputName": "d5",
          "outputType": "STRING"
        },
        {
          "type": "default",
          "dimension": "v1",
          "outputName": "d6",
          "outputType": "ARRAY<STRING>"
        },
        {
          "type": "default",
          "dimension": "v2",
          "outputName": "d7",
          "outputType": "STRING"
        },
        {
          "type": "default",
          "dimension": "city",
          "outputName": "d8",
          "outputType": "STRING"
        },
        {
          "type": "default",
          "dimension": "country",
          "outputName": "d9",
          "outputType": "STRING"
        },
        {
          "type": "default",
          "dimension": "forwarded_for",
          "outputName": "d10",
          "outputType": "STRING"
        }
      ],
      "aggregations": [
        {
          "type": "count",
          "name": "a0"
        },
        {
          "type": "longSum",
          "name": "a1",
          "fieldName": "session_length"
        },
        {
          "type": "HLLSketchBuild",
          "name": "a2",
          "fieldName": "event_type",
          "lgK": 12,
          "tgtHllType": "HLL_4",
          "round": true
        }
      ],
      "limitSpec": {
        "type": "default",
        "columns": [
          {
            "dimension": "d4",
            "direction": "ascending",
            "dimensionOrder": {
              "type": "lexicographic"
            }
          },
          {
            "dimension": "d1",
            "direction": "ascending",
            "dimensionOrder": {
              "type": "lexicographic"
            }
          }
        ]
      },
      "context": {
        "finalizeAggregations": false,
        "groupByEnableMultiValueUnnesting": false,
        "queryId": "665106f9-325a-4799-85a8-cc16855bca10",
        "sqlInsertSegmentGranularity": "\"HOUR\"",
        "sqlQueryId": "665106f9-325a-4799-85a8-cc16855bca10"
      }
    },
    "signature": [
      {
        "name": "d0",
        "type": "LONG"
      },
      {
        "name": "d1",
        "type": "STRING"
      },
      {
        "name": "d2",
        "type": "STRING"
      },
      {
        "name": "d3",
        "type": "STRING"
      },
      {
        "name": "d4",
        "type": "STRING"
      },
      {
        "name": "d5",
        "type": "STRING"
      },
      {
        "name": "d6",
        "type": "ARRAY<STRING>"
      },
      {
        "name": "d7",
        "type": "STRING"
      },
      {
        "name": "d8",
        "type": "STRING"
      },
      {
        "name": "d9",
        "type": "STRING"
      },
      {
        "name": "d10",
        "type": "STRING"
      },
      {
        "name": "a0",
        "type": "LONG"
      },
      {
        "name": "a1",
        "type": "LONG"
      },
      {
        "name": "a2",
        "type": "LONG"
      }
    ],
    "columnMappings": [
      {
        "queryColumn": "d0",
        "outputColumn": "__time"
      },
      {
        "queryColumn": "d1",
        "outputColumn": "session"
      },
      {
        "queryColumn": "d2",
        "outputColumn": "agent_category"
      },
      {
        "queryColumn": "d3",
        "outputColumn": "agent_type"
      },
      {
        "queryColumn": "d4",
        "outputColumn": "browser"
      },
      {
        "queryColumn": "d5",
        "outputColumn": "browser_version"
      },
      {
        "queryColumn": "d6",
        "outputColumn": "language"
      },
      {
        "queryColumn": "d7",
        "outputColumn": "os"
      },
      {
        "queryColumn": "d8",
        "outputColumn": "city"
      },
      {
        "queryColumn": "d9",
        "outputColumn": "country"
      },
      {
        "queryColumn": "d10",
        "outputColumn": "ip_address"
      },
      {
        "queryColumn": "a0",
        "outputColumn": "cnt"
      },
      {
        "queryColumn": "a1",
        "outputColumn": "session_length"
      },
      {
        "queryColumn": "a2",
        "outputColumn": "unique_event_types"
      }
    ]
  }
]

This PR has:

sql/src/main/java/org/apache/druid/sql/calcite/planner/QueryHandler.java

+      objectNode.put(
+          "columnMappings",
+          jsonMapper.convertValue(QueryUtils.buildColumnMappings(relRoot.fields, druidQuery), ArrayNode.class));


* add tests

abhishekrb19

LGTM overall. A few questions and comments:

Will the new columnMappings be backwards compatible with existing clients? Currently, this info is added as part of PLAN column along with queries and signature -- I think this should be ok as it's a new map entry, but would like to double check.
Can we update the EXPLAIN PLAN sample output in the following README to include the new info: https://github.com/apache/druid/blob/master/docs/querying/sql-translation.md#interpreting-explain-plan-output?
Since columnMappings info is only available in the native explain query mode (and not in the legacy Calcite mode), it'd be worth clarifying that in the PR summary or title, so it's evident in the next release notes.

abhishekrb19 · 2023-05-02T23:31:51Z

.github/scripts/unit_tests_script.sh

  --type jacoco \
-  --line-coverage 50 \
-  --branch-coverage 50 \
+  --line-coverage 1 \


Is this change intentional?

Please do revert this back to 50, 50.

yeah was intentional to get around the code coverage blocking the ITs from running, I will revert back for sure.

If this change is to get around the coverage failure and run everything, I believe you could achieve that by running workflows directly from the source repo's branch, i.e., your fork

sql/src/main/java/org/apache/druid/sql/calcite/planner/QueryHandler.java

sql/src/main/java/org/apache/druid/sql/calcite/planner/QueryUtils.java

gianm

The basic idea LGTM. I generally agree with @abhishekrb19's review comments.

gianm · 2023-05-03T06:44:38Z

.github/scripts/unit_tests_script.sh

  --type jacoco \
-  --line-coverage 50 \
-  --branch-coverage 50 \
+  --line-coverage 1 \


Please do revert this back to 50, 50.

zachjsh · 2023-05-03T19:23:21Z

LGTM overall. A few questions and comments:

1. Will the new `columnMappings` be backwards compatible with existing clients? Currently, this info is added as part of `PLAN` column along with `queries` and `signature` -- I think this should be ok as it's a new map entry, but would like to double check.

2. Can we update the `EXPLAIN PLAN` sample output in the following `README` to include the new info: https://github.com/apache/druid/blob/master/docs/querying/sql-translation.md#interpreting-explain-plan-output?

3. Since `columnMappings` info is only available in the native explain query mode (and not in the legacy Calcite mode),  it'd be worth clarifying that in the PR summary or title, so it's evident in the next release notes.

thanks @abhishekrb19 , I addressed your concerns here. Adding a new property to existing returned object should not break existing clients, unless they are bad clients 👿

abhishekrb19

LGTM, thanks!

zachjsh · 2023-05-03T22:06:47Z

dont merge until I revert the code coverage threshold. Will do so when all tests pass.

Add columnMappings to explain plan output

9b06f43

zachjsh requested a review from gianm April 29, 2023 03:24

github-advanced-security bot found potential problems Apr 29, 2023

View reviewed changes

* fix checkstyle

ca8b83a

* add tests

zachjsh marked this pull request as ready for review May 1, 2023 17:46

zachjsh added 7 commits May 1, 2023 12:15

* improve test coverage

5ced00b

* temporarily remove unit-test need to run ITs

f5bc2da

* depend on build

0266ab3

* temporarily lower unit test threshold

b07ce6e

* add back dependency on unit-tests

80bd9c6

* add license headers

cdb6689

* fix header order

d21c3f9

abhishekrb19 reviewed May 3, 2023

View reviewed changes

gianm reviewed May 3, 2023

View reviewed changes

* review comments

8e2df15

github-actions bot added the Area - Documentation label May 3, 2023

* fix intellij inspection errors

a0ff3c2

abhishekrb19 approved these changes May 3, 2023

View reviewed changes

* revert code coverage change

20ef0e8

zachjsh merged commit 48cde23 into apache:master May 4, 2023

zachjsh deleted the explain-plan-column-mappings branch May 4, 2023 17:36

zachjsh restored the explain-plan-column-mappings branch May 4, 2023 18:03

zachjsh deleted the explain-plan-column-mappings branch May 4, 2023 18:27

abhishekagarwal87 added this to the 27.0 milestone Jul 19, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add columnMappings to explain plan output#14187

Add columnMappings to explain plan output#14187
zachjsh merged 12 commits intoapache:masterfrom
zachjsh:explain-plan-column-mappings

zachjsh commented Apr 29, 2023 •

edited

Loading

Uh oh!

Check notice

abhishekrb19 left a comment

Uh oh!

abhishekrb19 May 2, 2023

Uh oh!

gianm May 3, 2023

Uh oh!

zachjsh May 3, 2023

Uh oh!

abhishekrb19 May 3, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gianm left a comment

Uh oh!

gianm May 3, 2023

Uh oh!

zachjsh commented May 3, 2023

Uh oh!

abhishekrb19 left a comment

Uh oh!

zachjsh commented May 3, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

zachjsh commented Apr 29, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Check notice

abhishekrb19 left a comment

Choose a reason for hiding this comment

Uh oh!

abhishekrb19 May 2, 2023

Choose a reason for hiding this comment

Uh oh!

gianm May 3, 2023

Choose a reason for hiding this comment

Uh oh!

zachjsh May 3, 2023

Choose a reason for hiding this comment

Uh oh!

abhishekrb19 May 3, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gianm left a comment

Choose a reason for hiding this comment

Uh oh!

gianm May 3, 2023

Choose a reason for hiding this comment

Uh oh!

zachjsh commented May 3, 2023

Uh oh!

abhishekrb19 left a comment

Choose a reason for hiding this comment

Uh oh!

zachjsh commented May 3, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

zachjsh commented Apr 29, 2023 •

edited

Loading