Add support for Pivot/Unpivot statements #6387

Mytherin · 2023-02-20T20:28:53Z

Implements #2061

This PR adds support for the pivot/unpivot statements in DuckDB. We add both the "SQL standard" manner of pivoting/unpivoting (supported by Snowflake, BigQuery, DataBricks, SQL Server, others?) and a pivot/unpivot statement that is (hopefully) easier to use/understand.

In the examples below we use this example data set:

CREATE TABLE Cities(Country VARCHAR, Name VARCHAR, Year INT, Population INT);
INSERT INTO Cities VALUES ('NL', 'Amsterdam', 2000, 1005);
INSERT INTO Cities VALUES ('NL', 'Amsterdam', 2010, 1065);
INSERT INTO Cities VALUES ('NL', 'Amsterdam', 2020, 1158);
INSERT INTO Cities VALUES ('US', 'Seattle', 2000, 564);
INSERT INTO Cities VALUES ('US', 'Seattle', 2010, 608);
INSERT INTO Cities VALUES ('US', 'Seattle', 2020, 738);
INSERT INTO Cities VALUES ('US', 'New York City', 2000, 8015);
INSERT INTO Cities VALUES ('US', 'New York City', 2010, 8175);
INSERT INTO Cities VALUES ('US', 'New York City', 2020, 8772);

Or for the non-database engines out there:

┌─────────┬───────────────┬───────┬────────────┐
│ Country │     Name      │ Year  │ Population │
│ varchar │    varchar    │ int32 │   int32    │
├─────────┼───────────────┼───────┼────────────┤
│ NL      │ Amsterdam     │  2000 │       1005 │
│ NL      │ Amsterdam     │  2010 │       1065 │
│ NL      │ Amsterdam     │  2020 │       1158 │
│ US      │ Seattle       │  2000 │        564 │
│ US      │ Seattle       │  2010 │        608 │
│ US      │ Seattle       │  2020 │        738 │
│ US      │ New York City │  2000 │       8015 │
│ US      │ New York City │  2010 │       8175 │
│ US      │ New York City │  2020 │       8772 │
└─────────┴───────────────┴───────┴────────────┘

Pivot

Pivot, also known as PIVOT_WIDER, takes the values of a column and turns them into separate columns.

For example, in the above table, if we want to move the values from the "year" and "population" column into separate columns (pivoted over name) we can do that as follows:

PIVOT Cities ON Year USING SUM(Population);
┌─────────┬───────────────┬────────┬────────┬────────┐
│ Country │     Name      │  2000  │  2010  │  2020  │
│ varchar │    varchar    │ int128 │ int128 │ int128 │
├─────────┼───────────────┼────────┼────────┼────────┤
│ NL      │ Amsterdam     │   1005 │   1065 │   1158 │
│ US      │ Seattle       │    564 │    608 │    738 │
│ US      │ New York City │   8015 │   8175 │   8772 │
└─────────┴───────────────┴────────┴────────┴────────┘

And using the SQL Standard pivot syntax:

FROM Cities PIVOT (SUM(Population) FOR Year IN (2000, 2010, 2020));
┌─────────┬───────────────┬────────┬────────┬────────┐
│ Country │     Name      │  2000  │  2010  │  2020  │
│ varchar │    varchar    │ int128 │ int128 │ int128 │
├─────────┼───────────────┼────────┼────────┼────────┤
│ NL      │ Amsterdam     │   1005 │   1065 │   1158 │
│ US      │ Seattle       │    564 │    608 │    738 │
│ US      │ New York City │   8015 │   8175 │   8772 │
└─────────┴───────────────┴────────┴────────┴────────┘

As part of the syntax, we need to specify:

The column to pivot on (i.e. from which we take the names for the new columns)
The aggregate used to fill in the values of the columns (this must be an aggregate because we might have multiple matches)
(Optionally) the columns to group by - by default this is all columns not mentioned in the above clauses

Note that when using the new Pivot syntax we do not need to specify the values of the columns - they are automatically discovered from the table. We can still specify the columns, however, in case we want to pivot only a subset of values. Any values not specified will be ignored.

PIVOT Cities ON Year IN (2000, 2020) USING SUM(Population);
┌─────────┬───────────────┬────────┬────────┐
│ Country │     Name      │  2000  │  2020  │
│ varchar │    varchar    │ int128 │ int128 │
├─────────┼───────────────┼────────┼────────┤
│ NL      │ Amsterdam     │   1005 │   1158 │
│ US      │ Seattle       │    564 │    738 │
│ US      │ New York City │   8015 │   8772 │
└─────────┴───────────────┴────────┴────────┘

We can also specify an optional GROUP BY clause, in case we want to group by only a subset of the remaining columns. For example:

PIVOT Cities ON Year USING SUM(Population) GROUP BY Country;
┌─────────┬────────┬────────┬────────┐
│ Country │  2000  │  2010  │  2020  │
│ varchar │ int128 │ int128 │ int128 │
├─────────┼────────┼────────┼────────┤
│ NL      │   1005 │   1065 │   1158 │
│ US      │   8579 │   8783 │   9510 │
└─────────┴────────┴────────┴────────┘

Unpivot

Unpivot, also known as PIVOT_LONGER, takes the names of a set of columns and turns them into values. For example, if we start with the previously pivoted table:

CREATE TABLE PivotedCities AS PIVOT Cities ON Year USING SUM(Population);
┌─────────┬───────────────┬────────┬────────┬────────┐
│ Country │     Name      │  2000  │  2010  │  2020  │
│ varchar │    varchar    │ int128 │ int128 │ int128 │
├─────────┼───────────────┼────────┼────────┼────────┤
│ NL      │ Amsterdam     │   1005 │   1065 │   1158 │
│ US      │ Seattle       │    564 │    608 │    738 │
│ US      │ New York City │   8015 │   8175 │   8772 │
└─────────┴───────────────┴────────┴────────┴────────┘

We can revert it back to the original table using the following UNPIVOT statement:

UNPIVOT PivotedCities ON 2000, 2010, 2020 INTO NAME Year VALUE Population;

┌─────────┬───────────────┬─────────┬────────────┐
│ Country │     Name      │  Year   │ Population │
│ varchar │    varchar    │ varchar │   int128   │
├─────────┼───────────────┼─────────┼────────────┤
│ NL      │ Amsterdam     │ 2000    │       1005 │
│ NL      │ Amsterdam     │ 2010    │       1065 │
│ NL      │ Amsterdam     │ 2020    │       1158 │
│ US      │ Seattle       │ 2000    │        564 │
│ US      │ Seattle       │ 2010    │        608 │
│ US      │ Seattle       │ 2020    │        738 │
│ US      │ New York City │ 2000    │       8015 │
│ US      │ New York City │ 2010    │       8175 │
│ US      │ New York City │ 2020    │       8772 │
└─────────┴───────────────┴─────────┴────────────┘

Using the SQL standard syntax, that looks like this:

FROM PivotedCities UNPIVOT(Population FOR Year IN (2000, 2010, 2020));

The unpivot clause takes the given set of columns and decomposes them into two columns: a NAME and a VALUE column. The NAME column contains the column names, and the VALUE column contains the column values.

Note that providing names for these columns is optional. We could also run the query without specifying these. In that case, the columns will be called name and value.

UNPIVOT PivotedCities ON 2000, 2010, 2020;

┌─────────┬───────────────┬─────────┬────────┐
│ Country │     Name      │ name    │ value  │
│ varchar │    varchar    │ varchar │ int128 │
├─────────┼───────────────┼─────────┼────────┤
│ NL      │ Amsterdam     │ 2000    │   1005 │
│ NL      │ Amsterdam     │ 2010    │   1065 │
│ NL      │ Amsterdam     │ 2020    │   1158 │
│ US      │ Seattle       │ 2000    │    564 │
│ US      │ Seattle       │ 2010    │    608 │
│ US      │ Seattle       │ 2020    │    738 │
│ US      │ New York City │ 2000    │   8015 │
│ US      │ New York City │ 2010    │   8175 │
│ US      │ New York City │ 2020    │   8772 │
└─────────┴───────────────┴─────────┴────────┘

We also allow COLUMNS and star expressions in the UNPIVOT. For example:

UNPIVOT PivotedCities ON COLUMNS('\d+') INTO NAME Year VALUE Population;
UNPIVOT PivotedCities ON * EXCLUDE (Country, Name) INTO NAME Year VALUE Population;

Internals

Pivoting and unpivoting are implemented entirely as rewrites into SQL queries. PIVOT is implemented as a set of aggregations with FILTER clauses, and UNPIVOT is implemented using lists and UNNEST statement.

Here are the equivalent SQL clauses:

Pivot

PIVOT Cities ON Year USING SUM(Population);

Becomes:

SELECT Country,
       Name,
       SUM(Population) FILTER (Year=2000) AS "2000",
       SUM(Population) FILTER (Year=2010) AS "2010",
       SUM(Population) FILTER (Year=2020) AS "2020"
FROM Cities
GROUP BY ALL;

Both providing the following result:

┌─────────┬───────────────┬────────┬────────┬────────┐
│ Country │     Name      │  2000  │  2010  │  2020  │
│ varchar │    varchar    │ int128 │ int128 │ int128 │
├─────────┼───────────────┼────────┼────────┼────────┤
│ NL      │ Amsterdam     │   1005 │   1065 │   1158 │
│ US      │ Seattle       │    564 │    608 │    738 │
│ US      │ New York City │   8015 │   8175 │   8772 │
└─────────┴───────────────┴────────┴────────┴────────┘

Unpivot

UNPIVOT PivotedCities ON 2000, 2010, 2020 INTO NAME Year VALUE Population;

Becomes:

SELECT Country,
       Name,
       UNNEST(['2000', '2010', '2020']) AS "Year",
       UNNEST(["2000", "2010", "2020"]) AS "Population"
FROM PivotedCities;

Both providing the following result:

┌─────────┬───────────────┬─────────┬────────────┐
│ Country │     Name      │  Year   │ Population │
│ varchar │    varchar    │ varchar │   int128   │
├─────────┼───────────────┼─────────┼────────────┤
│ NL      │ Amsterdam     │ 2000    │       1005 │
│ NL      │ Amsterdam     │ 2010    │       1065 │
│ NL      │ Amsterdam     │ 2020    │       1158 │
│ US      │ Seattle       │ 2000    │        564 │
│ US      │ Seattle       │ 2010    │        608 │
│ US      │ Seattle       │ 2020    │        738 │
│ US      │ New York City │ 2000    │       8015 │
│ US      │ New York City │ 2010    │       8175 │
│ US      │ New York City │ 2020    │       8772 │
└─────────┴───────────────┴─────────┴────────────┘

Pivot without specifying the names

The true magic (or hack - depending on your perspective :)) in this PR is in allowing PIVOT without specifying the column names up front. DuckDB, like most SQL engines, is completely dependent on knowing all the column names and types within a single SQL statement up front. The magic - and what happens here - is that running a PIVOT without specifying the names will result in running multiple SQL statements.

Specifically, we (ab)use the enums for this purpose. Enums hold a set of values that are known at query compilation time, and can be created from a query.

So the following query:

PIVOT Cities ON Year USING SUM(Population);

Actually gets translated into the following queries:

CREATE TEMPORARY TYPE __pivot_enum_0_0 AS ENUM (SELECT DISTINCT Year::VARCHAR FROM Cities ORDER BY Year);
PIVOT Cities ON Year IN __pivot_enum_0_0 USING SUM(Population);

Using multiple pivots is also possible. For example, we can join on the results of a PIVOT statement with another PIVOT statement:

FROM
	(PIVOT Cities ON Year USING SUM(Population) GROUP BY Country)
JOIN
	(PIVOT Cities ON Name USING SUM(Population) GROUP BY Country)
USING (Country);

┌─────────┬────────┬────────┬────────┬───────────┬───────────────┬─────────┐
│ Country │  2000  │  2010  │  2020  │ Amsterdam │ New York City │ Seattle │
│ varchar │ int128 │ int128 │ int128 │  int128   │    int128     │ int128  │
├─────────┼────────┼────────┼────────┼───────────┼───────────────┼─────────┤
│ NL      │   1005 │   1065 │   1158 │      3228 │          NULL │    NULL │
│ US      │   8579 │   8783 │   9510 │      NULL │         24962 │    1910 │
└─────────┴────────┴────────┴────────┴───────────┴───────────────┴─────────┘

…in the binder

…upport for star and COLUMNS expressions in unpivot

jaredlander · 2023-02-21T12:26:10Z

This looks really awesome and is something I've wanted to do for a long time. Since which direction pivot and unpivot move the data, can I ask that this is very explicit and the functions are called just pivot_longer and pivot_wider?

parodayco · 2023-02-21T13:52:45Z

I can't wait to see this in duckdb since we rely on the Snowflake pivot function to deal with large denormalized tables similar to your initial example. Our assignment tables include extraneous columns and numerous categorical values in the 'name' field, so I'm sharing a sample query here for consideration while you design the duckdb solution.

select * 
from users as u 
left join 
	(select userid, userpropertyname, userpropertyvalue from userpropertyassignments)
	pivot(
		max(userpropertyvalue) 
		for userpropertyname in ('Geography', 'Market', 'Business Group', 'Business Unit')
	) 
	as up (user_id, geography, market, business_group, business_unit) 
on up.user_id = u.id

Where pivoting fails is when there are multiple userpropertyvalues for a single userid and userpropertyname. In this case, my query only returns a single "max" value. I know there's manual solutions but aggregated values would be a magnificent addition to any pivot function. No pressure, and thank you for adding pivot to duckdb.

Mytherin · 2023-02-21T14:01:04Z

I can't wait to see this in duckdb since we rely on the Snowflake pivot function to deal with large denormalized tables similar to your initial example. Our assignment tables include extraneous columns and numerous categorical values in the 'name' field, so I'm sharing a sample query here for consideration while you design the duckdb solution.
select * 
from users as u 
left join 
	(select userid, userpropertyname, userpropertyvalue from userpropertyassignments)
	pivot(
		max(userpropertyvalue) 
		for userpropertyname in ('Geography', 'Market', 'Business Group', 'Business Unit')
	) 
	as up (user_id, geography, market, business_group, business_unit) 
on up.user_id = u.id 
Where pivoting fails is when there are multiple userpropertyvalues for a single userid and userpropertyname. In this case, my query only returns a single "max" value. I know there's manual solutions but aggregated values would be a magnificent addition to any pivot function. No pressure, and thank you for adding pivot to duckdb.

You could use a LIST aggregate for this, e.g. with my above example:

D PIVOT Cities ON Year USING LIST(Population) GROUP BY Country;
┌─────────┬─────────────┬─────────────┬─────────────┐
│ Country │    2000     │    2010     │    2020     │
│ varchar │   int32[]   │   int32[]   │   int32[]   │
├─────────┼─────────────┼─────────────┼─────────────┤
│ NL      │ [1005]      │ [1065]      │ [1158]      │
│ US      │ [564, 8015] │ [608, 8175] │ [738, 8772] │
└─────────┴─────────────┴─────────────┴─────────────┘

Mytherin · 2023-02-21T14:10:48Z

This looks really awesome and is something I've wanted to do for a long time. Since which direction pivot and unpivot move the data, can I ask that this is very explicit and the functions are called just pivot_longer and pivot_wider?

Sure, I have added PIVOT_WIDER as an alias for PIVOT and PIVOT_LONGER as an alias for UNPIVOT.

jaredlander · 2023-02-21T14:26:01Z

Thanks! I think this will lead to a lot less confusion and many more happy users.

domoritz · 2023-02-21T20:56:20Z

This is amazing. Now DuckDB has all basic data transformations. Really cool!

Mytherin · 2023-03-01T11:34:40Z

v0.7.1 was a bug-fix release - this will be in v0.8.0

Alex-Monahan · 2023-03-09T15:46:21Z

This is super awesome! I am going to use the heck out of this feature. Thank you!!

I do have one request though @Mytherin... Sorry to even ask! However, it's the last piece that's needed for full Excel pivot table parity.

Is there a way that the high level pivot syntax could support multiple aggregation metrics / measures? It appears that the SQL standard syntax that you implemented can support it based on this test:

# multiple aggregations
query IIIIIIIII
SELECT year, q1_total, q1_avg, q2_total, q2_avg, q3_total, q3_avg, q4_total, q4_avg
    FROM (SELECT year, quarter, sales FROM sales) AS s
    PIVOT (sum(sales) AS total, avg(sales) AS avg
      FOR quarter
      IN (1 AS q1, 2 AS q2, 3 AS q3, 4 AS q4));

Could we have multiple aggregates in the USING clause, maybe something like this?

PIVOT Cities ON Year USING (SUM(Population) as sum_pop, count(population) as count_pop) GROUP BY Country;

or maybe even just without the aliases?

PIVOT Cities ON Year USING (SUM(Population), count(population)) GROUP BY Country;

I added parentheses, but maybe they wouldn't be needed either?

PIVOT Cities ON Year USING SUM(Population), count(population) GROUP BY Country;

Mytherin · 2023-03-09T15:56:01Z

Yes, this is actually trivially implemented as it was already supported. Pushed it in #6644

Alex-Monahan · 2023-03-09T16:04:02Z

GAME ON!! For those of you following along at home, that is a 10 minute turnaround time from the wizard himself, Dr. @Mytherin... Thank you!

Mytherin · 2023-03-09T16:07:24Z

Dr. Mytherin please ;)

Alex-Monahan · 2023-03-09T16:09:56Z

Dr. Mytherin please ;)

Gah!! Sorry! Fixed...

Mytherin · 2023-03-09T16:28:58Z

Hah, no worries :) not a stickler for titles, but if I get an opportunity to reference Austin Powers I will :)

lben · 2023-04-28T19:54:14Z

Hi, this is great! I have a question though and sorry in advance if it comes from ignorance.
My understanding is that DuckDB builds a query plan and performs things like projection pushdown but since this functionality seems to require knowledge about the ON column values (column Year in the example) does that mean that it forces DuckDB to execute its query plan up to that point and then perform these rewritten queries? Or -somehow- the rewritten queries are still part of the query plan and everything will be lazy evaluated until the very end?

I hope I was clear, if not feel free to ask any clarifying questions

Thanks!

Mytherin added 30 commits February 17, 2023 09:58

Pivot v0.1

37931a4

Pivot mostly working somehow

b4a386d

Add support for multiple pivots

6b35e64

Test pivot

c98a020

We can support expressions within the pivot aggregate - so why not

871ead4

Top-level pivot statement working using PRAGMA replacement

02c3ee1

Missing files

4d83523

Top-level PIVOT syntax working as a subquery/select clause

88e0d0e

Clean up PIVOT syntax

fd52e16

Support GROUP BY syntax in standard PIVOT syntax

192f670

Unpivot working

fc51ce1

Add support for supplying aliases to pivots

1041cdd

Add more pivot tests

3fa6df2

Add support for IN lists to the top-level PIVOT statement

edc180e

PIVOT and UNPIVOT with multiple columns working

7320925

Add support for multiple aggregations

d9698dd

INCLUDE/EXCLUDE NULLS working, plus more tests

31780fa

enable_verification

622e38b

Fixes for query results

1d94d88

(Un)pivot working in correlated subqueries

742420f

More pivot tests (generated columns, prepared statements, storage tests)

69ff306

Add missing include

c3d3478

Re-generate serialized plans

562d2d8

Various CI fixes

df5a7ea

More CI fixes

9e05f91

LogicalPlanStatement - when executing correctly set the bound_tables …

2b56b5c

…in the binder

Rework top-level pivot syntax, add top-level unpivot syntax and add s…

0b0c7f6

…upport for star and COLUMNS expressions in unpivot

Add pivot examples, and some more fixes/simplifications in the grammar

315dd69

Fix for joining on pivot statements

5bf79fb

Tidy fix

98ff356

Mytherin added 3 commits February 21, 2023 10:31

Fixes for single-file compilation

a0c254b

More join order fixes (thanks @Tmonster)

d569580

Remove very slow test that does not offer much value

af98772

Add pivot_wider and pivot_longer aliases

9bc7e63

westonpace mentioned this pull request Feb 21, 2023

[C++] Add a pivot_longer node apache/arrow#34266

Closed

Mytherin merged commit 0a4bb30 into duckdb:feature Feb 21, 2023

Mytherin mentioned this pull request Feb 21, 2023

Support PIVOT statement #2061

Closed

This comment was marked as abuse.

Sign in to view

cpcloud mentioned this pull request Mar 5, 2023

feat(table): pivot_longer API ibis-project/ibis#5663

Merged

hadley mentioned this pull request Mar 8, 2023

pivot_wider doesn't seem to be lazy tidyverse/dbplyr#598

Closed

snth mentioned this pull request Mar 12, 2023

Pivot and melt PRQL/prql#644

Open

Mytherin mentioned this pull request Mar 22, 2023

PIVOT does not work #6826

Closed

2 tasks

Mytherin mentioned this pull request Apr 3, 2023

Faster PIVOT statement #6961

Merged

Mytherin deleted the pivot branch April 24, 2023 13:35

eitsupi mentioned this pull request May 4, 2024

Rename melt to unpivot and make parameters consistent with pivot pola-rs/polars#11974

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add support for Pivot/Unpivot statements #6387

Add support for Pivot/Unpivot statements #6387

Uh oh!

Mytherin commented Feb 20, 2023

Uh oh!

jaredlander commented Feb 21, 2023

Uh oh!

parodayco commented Feb 21, 2023

Uh oh!

Mytherin commented Feb 21, 2023

Uh oh!

Mytherin commented Feb 21, 2023

Uh oh!

jaredlander commented Feb 21, 2023

Uh oh!

domoritz commented Feb 21, 2023

Uh oh!

This comment was marked as abuse.

Mytherin commented Mar 1, 2023

Uh oh!

This comment was marked as abuse.

Alex-Monahan commented Mar 9, 2023 •

edited

Loading

Uh oh!

Mytherin commented Mar 9, 2023

Uh oh!

Alex-Monahan commented Mar 9, 2023 •

edited

Loading

Uh oh!

Mytherin commented Mar 9, 2023

Uh oh!

Alex-Monahan commented Mar 9, 2023

Uh oh!

Mytherin commented Mar 9, 2023

Uh oh!

lben commented Apr 28, 2023

Uh oh!

Uh oh!

Add support for Pivot/Unpivot statements #6387

Add support for Pivot/Unpivot statements #6387

Uh oh!

Conversation

Mytherin commented Feb 20, 2023

Pivot

Unpivot

Internals

Pivot

Unpivot

Pivot without specifying the names

Uh oh!

jaredlander commented Feb 21, 2023

Uh oh!

parodayco commented Feb 21, 2023

Uh oh!

Mytherin commented Feb 21, 2023

Uh oh!

Mytherin commented Feb 21, 2023

Uh oh!

jaredlander commented Feb 21, 2023

Uh oh!

domoritz commented Feb 21, 2023

Uh oh!

This comment was marked as abuse.

Mytherin commented Mar 1, 2023

Uh oh!

This comment was marked as abuse.

Alex-Monahan commented Mar 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Mytherin commented Mar 9, 2023

Uh oh!

Alex-Monahan commented Mar 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Mytherin commented Mar 9, 2023

Uh oh!

Alex-Monahan commented Mar 9, 2023

Uh oh!

Mytherin commented Mar 9, 2023

Uh oh!

lben commented Apr 28, 2023

Uh oh!

Uh oh!

Alex-Monahan commented Mar 9, 2023 •

edited

Loading

Alex-Monahan commented Mar 9, 2023 •

edited

Loading