Support persistent CTE's #20887

jaystarshot · 2023-09-15T20:28:40Z

Description

Support persistent query-level CTEs

The goal is to allow users to define persistent ctes and support them
Session property cte_materialization_strategy is added which when set to ALL will materialize all ctes to HDFS

This will be the first step for 19744.

Motivation and Context

This change is based on the design. The eventual goal is to materialize or stream the CTEs (either in local memory or remote) using CBO rules and the architecture reflects that.
The temporary tables created for each cte will be partitioned on the first input column
In the current implementation, all CTEs will be evaluated before the query is run.

This change has 5 main commits

Add session property and ast changes

This change adds the session property which allows CTE materialization. This information is also added to the CTEInformation collector.
The relational planner has the main change where a CTEReference Node is added if a CTE is persistent. This reference node will be used by the LogicalCteOptimizer to process.

Planner changes

Main changes
LogicalCteOptimizer - Processes the CTE references into CteConsumers and CteProducers. It also creates a sequence node which is added as a child below the topmost node. The sequence node has a list of leftSources and a right source. The leftSources list has the ctes ordered in a topological order of execution. {a,b,c,d} --> order of execution {d->c->b->a}.

PhysicalCteOptimizer - Processes the CteConsumers and CteProducers into table writes and temporary table scan. The processing code is based on basePlanFragmenter. From logical to physical optimizers, the CteProducer, CteConsumers, and Sequence Node are present to enable further logical optimizations. Physical optimization is applied just before adding exchanges. Post this optimization stage, CteConsumers and CteProducers are no longer present.

Code change in other optimizers and visitors

AddExchanges.java - Handling Sequence Node
HashGenerationOptimizer.java - Handling Sequence Node
StreamPropertyDerivations.java - Handling Sequence Node
PropertyDerivations.java - Handling Sequence Node
PruneUnreferencedOutputs.java - Handling CteConsumer, CteProducer and Sequnce
PushdownSubfields.java - handling CteProducer Node
StreamPropertyDerivations.java - handling sequence
UnaliasSymbolReferences.java - Handling CteConsumer, CteProducer and Sequnce

Scheduling changes

BasePlanFragmenter - Handle and transform sequence nodes. {a,b,c,d} --> order of execution {d->c->b->a}
We will create a chain of different sections of plans {d->c->b->a} and add it as a child of the main query section.

StreamingPlanSection will make sure that the children are scheduled and executed before the parent

Tests

Integration test to test end to end and result correctness
Testing logical cte optimizer

Current wips

Handle UnaliasSymbolReferences.java.
Handle ValidateDependenciesChecker.java
Check if result correctness can be done at a record level
Add parser and planner unit tests

Impact

Test Plan

Production tested with ALL CTEs materialized

Contributor checklist

Please make sure your submission complies with our development, formatting, commit message, and attribution guidelines.
PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced.
Documented new properties (with its default value), SQL syntax, functions, or other functionality.
If release notes are required, they follow the release notes guidelines.
Adequate tests were added if applicable.
CI passed.

Todo in future PRs

Integrate with HBO and cost model
Local auto replacement of cte in the optimizer
Select the top five candidate strategies from the optimizer, execute the planning process for each from the logical to the physical stage within the optimizer, and determine the most optimal one.
Support non partitioned temporary tables

Final Quality ToDos

Fix cases where the bucketing is done on unsupported buckets
Production test again with changes

Release Notes

Please follow release notes guidelines and fill in the release notes below.

== RELEASE NOTES ==

General Changes
* New Feature: Materialization of Common Table Expressions (CTEs) in Queries
* The relevant user connectors must support creating temporary tables 
* By materializing CTEs, users can take advantage of reusing common table expressions within their queries, which can lead to reductions in both storage operations and computational costs

## Recommended Configurations

1. CTE Materialization Strategy:
   - Enable by setting the session property ``cte_materialization_strategy`` or the configuration property ``cte-materialization-strategy`` to ``ALL``.
2. Temporary Table Storage Format:
   - For faster processing, set the configuration ``hive.temporary-table-storage-format`` or the session property ``temporary_table_storage_format`` to ``PAGEFILE``.
3. Temporary Table Schema:
   - It is recommended to set the configuration ``hive.temporary-table-schema`` or the session property ``temporary_table_schema`` to use the user's schema.
 4. Partitioning Provider Catalog
 - The config property ``query.partitioning-provider-catalog`` ``partitioning_provider_catalog`` session property must be set to enable cte materialization. The catalog must support providing a custom partitioning and ability to store temporary tables.

github-actions · 2023-10-24T12:56:00Z

Codenotify: Notifying subscribers in CODENOTIFY files for diff 3b8e8cc...8993b02.

No notifications.

mlyublena · 2023-10-24T22:44:11Z

General comment: I wonder if it would have been possible to achieve the same without introducing new syntax (for example through a session parameter). I know if't not as flexible as allowing specification per CTE but also not as disruptive as new syntax

presto-spi/src/main/java/com/facebook/presto/spi/plan/SequenceNode.java

jaystarshot · 2023-11-27T21:41:19Z

It won't because the canonical hashes won't be generated and properly linked with the execution.
I have 3 follow up prs'

Add these new nodes in cost frameworks - link
Support HBO with these new nodes - link
Locally auto materialize the ctes - link

mlyublena · 2023-11-27T23:17:45Z

It won't because the canonical hashes won't be generated and properly linked with the execution. I have 3 follow up prs'

Add these new nodes in cost frameworks - link

Support HBO with these new nodes - link

Locally auto materialize the ctes - link

makes sense.
what about the opposite: if the CTE is not materialized, would HBO work as before?

jaystarshot · 2023-11-28T00:51:04Z

Yes, the optimizer and entire flow remains unchanged unless the relational planner adds a cteReferenceNode, which only occurs if the specific flag is enabled.

jaystarshot · 2023-11-28T22:55:05Z

Does this PR look good to merge? Will need a committer approval

jaystarshot · 2023-11-29T02:28:46Z

As a follow up PR, I will also add a materialization strategy to only materialize ctes if references > threshold times and have a join or aggregate and maybe reduce data size

tdcmeehan · 2023-11-29T14:52:27Z

Does this PR look good to merge? Will need a committer approval

I'll have a look this week, other committers are free to get to it sooner!

mlyublena · 2023-12-01T18:54:49Z

...o-main/src/main/java/com/facebook/presto/sql/planner/optimizations/PhysicalCteOptimizer.java

+            catch (PrestoException e) {
+                if (e.getErrorCode().equals(NOT_SUPPORTED.toErrorCode())) {
+                    throw new PrestoException(
+                            NOT_SUPPORTED,


should we add this check to is isCteMaterializable?

I think here its just a catch for the metadata.createTemporaryTable() which cannot be invoked in the parser since we do not have partition metadata and other info there

mlyublena

Looks good!

jaystarshot · 2023-12-12T02:33:45Z

Adding a nit (using partitioningProviderCatalog) instead of hardcoded "hive" string in a use case.

tdcmeehan

Please make sure to consistently validate inputs. Please squash the Tests commit.

tdcmeehan · 2023-12-12T15:14:32Z

presto-spi/src/main/java/com/facebook/presto/spi/plan/CteReferenceNode.java

+    @Override
+    public PlanNode replaceChildren(List<PlanNode> newChildren)
+    {
+        checkArgument(newChildren.size() == 1, "expected newChildren to contain 1 node");


Please also validate that the input is not null with Objects.requireNonNull.

tdcmeehan · 2023-12-12T15:14:58Z

presto-main/src/main/java/com/facebook/presto/sql/TemporaryTableUtil.java

+    {
+    }
+
+    public static TableScanNode createTemporaryTableScan(


Here and elsewhere: Please validate inputs are not null

Thanks added in other cases, not adding here because not really needed at this static function.

tdcmeehan · 2023-12-12T15:20:08Z

...to-main/src/main/java/com/facebook/presto/sql/planner/optimizations/LogicalCteOptimizer.java

+
+        public List<PlanNode> getTopologicalOrdering()
+        {
+            List<PlanNode> topSortedCteProducerList = new ArrayList<>();


Use immutable list

tdcmeehan · 2023-12-12T15:20:21Z

...to-main/src/main/java/com/facebook/presto/sql/planner/optimizations/LogicalCteOptimizer.java

+
+        public Map<String, CteProducerNode> getCteProducerMap()
+        {
+            return cteProducerMap;


Can this return an immutable copy

tdcmeehan · 2023-12-12T15:20:56Z

...o-main/src/main/java/com/facebook/presto/sql/planner/optimizations/PhysicalCteOptimizer.java

+    }
+
+    @Override
+    public PlanOptimizerResult optimize(PlanNode plan, Session session, TypeProvider types, VariableAllocator variableAllocator, PlanNodeIdAllocator idAllocator, WarningCollector warningCollector)


Here and elsewhere--please always validate the inputs are not null

tdcmeehan · 2023-12-12T15:21:04Z

...o-main/src/main/java/com/facebook/presto/sql/planner/optimizations/PhysicalCteOptimizer.java

+
+        public CteProducerRewiter(Session session, PlanNodeIdAllocator idAllocator, VariableAllocator variableAllocator)
+        {
+            this.idAllocator = idAllocator;


tdcmeehan · 2023-12-12T15:21:11Z

...o-main/src/main/java/com/facebook/presto/sql/planner/optimizations/PhysicalCteOptimizer.java

+
+        @Override
+        public PlanNode visitCteProducer(CteProducerNode node, RewriteContext<PhysicalCteTransformerContext> context)
+        {


tdcmeehan · 2023-12-12T15:22:14Z

...o-main/src/main/java/com/facebook/presto/sql/planner/optimizations/PhysicalCteOptimizer.java

+                    tempScan.getCurrentConstraint(),
+                    tempScan.getEnforcedConstraint());
+
+            /* The temporary table scan might have columns removed by the UnaliasSymbolReferences and other optimizers (its a plan tree after all),


In general we prefer // comments

tdcmeehan · 2023-12-12T15:23:15Z

...o-main/src/main/java/com/facebook/presto/sql/planner/optimizations/PhysicalCteOptimizer.java

+            public TemporaryTableInfo(TableScanNode tableScanNode, List<VariableReferenceExpression> originalOutputVariables)
+            {
+                this.tableScanNode = tableScanNode;
+                this.originalOutputVariables = originalOutputVariables;


Please check inputs are not null, and create an immutable copy of the original output variables (no cost if the input is immutable as well)

tdcmeehan · 2023-12-12T15:24:27Z

presto-main/src/main/java/com/facebook/presto/sql/planner/BasePlanFragmenter.java

+    {
+        // Since this is topologically sorted by the LogicalCtePlanner, need to make sure that execution order follows
+        // Can be optimized further to avoid non dependents from getting blocked
+        int n = node.getCteProducers().size();


Please do not use single letter variables. cteProducerCount or something similar

tdcmeehan · 2023-12-12T15:25:36Z

Please look into the merge conflicts.

tdcmeehan

Just one last nit, and I think good to merge.

presto-spi/src/main/java/com/facebook/presto/spi/plan/CteReferenceNode.java

rschlussel

Left a few comments for Presto style and best practices that are likely to come up elsewhere in the future.

rschlussel · 2024-03-14T15:39:15Z

presto-spi/src/main/java/com/facebook/presto/spi/plan/CteReferenceNode.java

+            String cteName)
+    {
+        super(sourceLocation, id, statsEquivalentPlanNode);
+        this.cteName = cteName;


Always check requireNonNull() when setting Object fields in a constructor. (sometimes other argument checks are appropriate as well, but we almost never allow nulls. Generally we enforce this in constructors, and then elsewhere assume that nothing is null.)

rschlussel · 2024-03-14T15:41:04Z

presto-main/src/main/java/com/facebook/presto/sql/planner/SqlPlannerContext.java

+    public class NestedCteStack
+    {
+        @VisibleForTesting
+        public static final String delimiter = "_*%$_";


naming convention note: static final fields should be in all caps (so DELIMETER)

rschlussel · 2024-03-14T15:41:52Z

presto-main/src/main/java/com/facebook/presto/sql/planner/SqlPlannerContext.java

+
+        public NestedCteStack()
+        {
+            this.cteStack = new Stack<>();


initialize these where they are declared. They don't need anything from the constructor.

rschlussel · 2024-03-14T15:43:22Z

presto-main/src/main/java/com/facebook/presto/sql/planner/SqlPlannerContext.java

+
+        public void push(String cteName, Query query)
+        {
+            this.cteStack.push(cteName);


style note: we generally only use the "this" prefix in the constructor or setters. Otherwise, we refer to fields without the "this" prefix as long as there is no conflict requiring it. (not sure why, i think just to be concise).

rschlussel · 2024-03-14T15:49:34Z

presto-spi/src/main/java/com/facebook/presto/spi/plan/SequenceNode.java

+            PlanNode primarySource)
+    {
+        super(sourceLocation, planNodeId, statsEquivalentPlanNode);
+        this.cteProducers = leftList;


when setting a field to a collection passed in as input, make a defensive copy using ImmutableList.copyOf(). If it's already an immutable list this will be a no-op. (these should have requireNonNullChecks too)

rschlussel · 2024-03-14T15:53:48Z

presto-spi/src/main/java/com/facebook/presto/spi/plan/SequenceNode.java

+    {
+        List<PlanNode> children = new ArrayList<>(cteProducers);
+        children.add(primarySource);
+        return children;


use immutable collections whenever possible. It prevents bugs caused by things being modified unexpectedly. mutable collections should only be used when you plan something to be modified, and should generally stay local to wherever they are.

Here you could do

return ImmutableList.builder() .addAll(cteProducers); .add(primarySource) .build();

jaystarshot force-pushed the presto-cte branch 2 times, most recently from a226e08 to 61c61cb Compare September 24, 2023 23:15

jaystarshot force-pushed the presto-cte branch from 61c61cb to ed19859 Compare October 10, 2023 03:02

jaystarshot force-pushed the presto-cte branch from ed19859 to 2b45bcf Compare October 17, 2023 20:07

ClarenceThreepwood self-requested a review October 18, 2023 17:58

jaystarshot force-pushed the presto-cte branch 4 times, most recently from deb4054 to 49dd17c Compare October 20, 2023 09:12

tdcmeehan self-assigned this Oct 20, 2023

jaystarshot force-pushed the presto-cte branch 13 times, most recently from 85ae4bd to 6d1c216 Compare October 24, 2023 11:52

jaystarshot marked this pull request as ready for review October 24, 2023 12:54

jaystarshot requested a review from a team as a code owner October 24, 2023 12:54

jaystarshot requested a review from presto-oss October 24, 2023 12:54

jaystarshot requested a review from mlyublena October 24, 2023 12:58

mlyublena reviewed Oct 24, 2023

View reviewed changes

presto-spi/src/main/java/com/facebook/presto/spi/plan/SequenceNode.java Outdated Show resolved Hide resolved

jaystarshot force-pushed the presto-cte branch from 305b9bc to 9852616 Compare November 27, 2023 21:35

jaystarshot requested a review from mlyublena November 27, 2023 21:53

mlyublena reviewed Dec 1, 2023

View reviewed changes

mlyublena approved these changes Dec 1, 2023

View reviewed changes

jaystarshot force-pushed the presto-cte branch from 9852616 to 15a1700 Compare December 8, 2023 19:38

jaystarshot force-pushed the presto-cte branch from 15a1700 to 90b46c7 Compare December 12, 2023 02:35

tdcmeehan requested changes Dec 12, 2023

View reviewed changes

Add CTE Materialization session property and ast changes

383a420

jaystarshot force-pushed the presto-cte branch 3 times, most recently from 4e97080 to a268f5e Compare December 12, 2023 18:48

jaystarshot requested a review from tdcmeehan December 12, 2023 19:56

tdcmeehan approved these changes Dec 12, 2023

View reviewed changes

presto-spi/src/main/java/com/facebook/presto/spi/plan/CteReferenceNode.java Outdated Show resolved Hide resolved

jaystarshot added 2 commits December 12, 2023 13:20

CTE Materialization Planner Changes

d444710

CTE Materialization scheduling changes

8993b02

jaystarshot force-pushed the presto-cte branch from ab89eb0 to 8993b02 Compare December 12, 2023 21:20

jaystarshot requested a review from tdcmeehan December 13, 2023 00:36

tdcmeehan approved these changes Dec 13, 2023

View reviewed changes

jaystarshot merged commit 77dfcc8 into prestodb:master Dec 13, 2023
56 checks passed

wanglinsong mentioned this pull request Feb 12, 2024

Add release notes for 0.286 #21906

Merged

64 tasks

rschlussel reviewed Mar 14, 2024

View reviewed changes

aditi-pandit mentioned this pull request Apr 30, 2024

[Native] CTE support in Prestissimo #22630

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support persistent CTE's #20887

Support persistent CTE's #20887

jaystarshot commented Sep 15, 2023 •

edited

github-actions bot commented Oct 24, 2023 •

edited

mlyublena commented Oct 24, 2023

jaystarshot commented Nov 27, 2023

mlyublena commented Nov 27, 2023

jaystarshot commented Nov 28, 2023

jaystarshot commented Nov 28, 2023 •

edited

jaystarshot commented Nov 29, 2023

tdcmeehan commented Nov 29, 2023

mlyublena Dec 1, 2023

jaystarshot Dec 1, 2023

mlyublena left a comment

jaystarshot commented Dec 12, 2023

tdcmeehan left a comment

tdcmeehan Dec 12, 2023

tdcmeehan Dec 12, 2023

jaystarshot Dec 12, 2023

tdcmeehan Dec 12, 2023

tdcmeehan Dec 12, 2023

tdcmeehan Dec 12, 2023

tdcmeehan Dec 12, 2023

tdcmeehan Dec 12, 2023

tdcmeehan Dec 12, 2023

tdcmeehan Dec 12, 2023

tdcmeehan Dec 12, 2023

tdcmeehan commented Dec 12, 2023

tdcmeehan left a comment

rschlussel left a comment

rschlussel Mar 14, 2024

rschlussel Mar 14, 2024

rschlussel Mar 14, 2024

rschlussel Mar 14, 2024

rschlussel Mar 14, 2024

rschlussel Mar 14, 2024

Support persistent CTE's #20887

Support persistent CTE's #20887

Conversation

jaystarshot commented Sep 15, 2023 • edited

Description

Motivation and Context

Planner changes

Scheduling changes

Tests

Impact

Test Plan

Contributor checklist

Release Notes

github-actions bot commented Oct 24, 2023 • edited

mlyublena commented Oct 24, 2023

jaystarshot commented Nov 27, 2023

mlyublena commented Nov 27, 2023

jaystarshot commented Nov 28, 2023

jaystarshot commented Nov 28, 2023 • edited

jaystarshot commented Nov 29, 2023

tdcmeehan commented Nov 29, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mlyublena left a comment

Choose a reason for hiding this comment

jaystarshot commented Dec 12, 2023

tdcmeehan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tdcmeehan commented Dec 12, 2023

tdcmeehan left a comment

Choose a reason for hiding this comment

rschlussel left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jaystarshot commented Sep 15, 2023 •

edited

github-actions bot commented Oct 24, 2023 •

edited

jaystarshot commented Nov 28, 2023 •

edited