Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sql: support lateral joins #24560

Open
knz opened this issue Apr 6, 2018 · 8 comments

Comments

@knz
Copy link
Member

commented Apr 6, 2018

A join is "lateral" when its right operand depends on the left operand.

Lateral joins are one of the ways a subquery or relational expression can be correlated.

Lateral joins occur:

  • in the regular join syntax with the keyword "LATERAL", e.g. SELECT * FROM a, LATERAL b
  • implicitly when using a SRF in the select render position and one of the SRF argument uses a column name from one of the existing FROM tables, e.g. SELECT generate_series(v) FROM kv
  • also needed for common uses of jsonb_object_keys(), see #26110

Note: LATERAL in the join syntax changes the name resolution rules!

For example, the following two queries have very different behavior:

SELECT
   (SELECT a FROM db1.ab, LATERAL (SELECT * FROM kv WHERE v = b))
          -- v = b refers to "b" in db1.ab
FROM db2.ab

vs.

SELECT
   (SELECT a FROM db1.ab, (SELECT * FROM kv WHERE v = b))
          -- v = b refers to "b" in db2.ab!
FROM db2.ab
@knz knz added this to the 2.1 milestone Apr 6, 2018
@knz knz added the C-enhancement label Apr 6, 2018
craig bot added a commit that referenced this issue Apr 6, 2018
24561: sql: introduce the LATERAL keyword and mark it as unimplemented r=knz a=knz

CockroachDB aims to implement lateral joins. Until then, recognize
the syntax and link the error message to the appropriate support
issue.

This incidentally motivates keeping the keyword "reserved" in the
grammar.

Release note: None

Informs #24489.
Refers to #24560.
cc @petermattis
@knz

This comment has been minimized.

Copy link
Member Author

commented Apr 15, 2018

@benesch found this:

LATERAL can also precede a function-call FROM item, but in this case it is a noise word, because the function expression can refer to earlier FROM items in any case.

https://www.postgresql.org/docs/current/static/sql-select.html

In other words, a join with a SRF is always lateral.

@justinj

This comment has been minimized.

Copy link
Member

commented Mar 18, 2019

I think this is mostly straightforward, save for one minor hiccup, which is that we build our join trees right-to-left:

select * from a, b, c is a join (b join c)

however, the semantics here imply that they're built left-to-right: an expression with lateral (implicit or explicit) must be in a right subtree of an apply join with everything to the left of it as its left subtree:

select * from a, b, lateral c needs to mean (a join b) apply join c

It's not really possible to globally change the direction we build our joins until we have more thorough join ordering (because it can make hand-optimized queries much slower), so I think we would have to conditionally restructure the join tree in the presence of lateral.

justinj added a commit to justinj/cockroach that referenced this issue Apr 8, 2019
Fixes cockroachdb#24676.
Partial fix for cockroachdb#24560.

This commit adds support for LATERAL in the FROM clause of a SELECT.
LATERAL allows a subquery or SRF to refer to columns in tables earlier
in the FROM clause. This is semantically an inner apply join.

A hiccup with this is that we build join trees in a FROM clause
right-deep, but the semantics of LATERAL force the tree to be left-deep.
Changing the default to be left-deep could cause some hand-optimized
queries to become slower, so we only change the order if there is a
LATERAL subquery. Note that SRFs are always implicitly LATERAL.

This commit does *not* add support for LATERAL as a keyword to explicit
`JOIN` expressions.

Release note (sql change): the LATERAL keyword in a FROM clause is now
supported.
justinj added a commit to justinj/cockroach that referenced this issue Apr 15, 2019
Fixes cockroachdb#24676.
Partial fix for cockroachdb#24560.

This commit adds support for LATERAL in the FROM clause of a SELECT.
LATERAL allows a subquery or SRF to refer to columns in tables earlier
in the FROM clause. This is semantically an inner apply join.

A hiccup with this is that we build join trees in a FROM clause
right-deep, but the semantics of LATERAL force the tree to be left-deep.
Changing the default to be left-deep could cause some hand-optimized
queries to become slower, so we only change the order if there is a
LATERAL subquery. Note that SRFs are always implicitly LATERAL.

This commit does *not* add support for LATERAL as a keyword to explicit
`JOIN` expressions.

Release note (sql change): the LATERAL keyword in a FROM clause is now
supported.
justinj added a commit to justinj/cockroach that referenced this issue Apr 16, 2019
Fixes cockroachdb#24676.
Partial fix for cockroachdb#24560.

This commit adds support for LATERAL in the FROM clause of a SELECT.
LATERAL allows a subquery or SRF to refer to columns in tables earlier
in the FROM clause. This is semantically an inner apply join.

A hiccup with this is that we build join trees in a FROM clause
right-deep, but the semantics of LATERAL force the tree to be left-deep.
Changing the default to be left-deep could cause some hand-optimized
queries to become slower, so we only change the order if there is a
LATERAL subquery. Note that SRFs are always implicitly LATERAL.

This commit does *not* add support for LATERAL as a keyword to explicit
`JOIN` expressions.

Release note (sql change): the LATERAL keyword in a FROM clause is now
supported.
craig bot pushed a commit that referenced this issue Apr 16, 2019
36613: opt: add support for LATERAL in FROM clause r=justinj a=justinj

Fixes #24676.
Partial fix for #24560.

This commit adds support for LATERAL in the FROM clause of a SELECT.
LATERAL allows a subquery or SRF to refer to columns in tables earlier
in the FROM clause. This is semantically an inner apply join.

A hiccup with this is that we build join trees in a FROM clause
right-deep, but the semantics of LATERAL force the tree to be left-deep.
Changing the default to be left-deep could cause some hand-optimized
queries to become slower, so we only change the order if there is a
LATERAL subquery. Note that SRFs are always implicitly LATERAL.

This commit does *not* add support for LATERAL as a keyword to explicit
`JOIN` expressions.

Release note (sql change): the LATERAL keyword in a FROM clause is now
supported.

Co-authored-by: Justin Jaffray <justin@cockroachlabs.com>
craig bot pushed a commit that referenced this issue Apr 16, 2019
36613: opt: add support for LATERAL in FROM clause r=justinj a=justinj

Fixes #24676.
Partial fix for #24560.

This commit adds support for LATERAL in the FROM clause of a SELECT.
LATERAL allows a subquery or SRF to refer to columns in tables earlier
in the FROM clause. This is semantically an inner apply join.

A hiccup with this is that we build join trees in a FROM clause
right-deep, but the semantics of LATERAL force the tree to be left-deep.
Changing the default to be left-deep could cause some hand-optimized
queries to become slower, so we only change the order if there is a
LATERAL subquery. Note that SRFs are always implicitly LATERAL.

This commit does *not* add support for LATERAL as a keyword to explicit
`JOIN` expressions.

Release note (sql change): the LATERAL keyword in a FROM clause is now
supported.

Co-authored-by: Justin Jaffray <justin@cockroachlabs.com>
@jordanlewis jordanlewis added this to Triage in SQL Execution Team Apr 23, 2019
@jordanlewis

This comment has been minimized.

Copy link
Member

commented Apr 25, 2019

@justinj what's left to do after #36613?

@justinj

This comment has been minimized.

Copy link
Member

commented Apr 25, 2019

Explicit (using the JOIN keyword) joins that use the LATERAL keyword

@jordanlewis jordanlewis removed this from Feature requests / pie-in-the-skie in (DEPRECATED) SQL Front-end, Lang & Semantics Apr 27, 2019
@jordanlewis jordanlewis removed this from Triage in SQL Execution Team Apr 27, 2019
@awoods187

This comment has been minimized.

Copy link
Contributor

commented May 9, 2019

How big a project is this to complete?

@justinj

This comment has been minimized.

Copy link
Member

commented May 9, 2019

Pretty small, I just need to find the time to wrap it up

@awoods187

This comment has been minimized.

Copy link
Contributor

commented May 9, 2019

That's exciting! Looking forward to this.

@yuzefovich

This comment has been minimized.

Copy link
Contributor

commented Oct 12, 2019

Is this done now that #40945 has been merged? cc @justinj

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
SQL Planning
  
Lower Priority Backlog
6 participants
You can’t perform that action at this time.