Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Planning error for correlated queries over Hive connector #6215

Closed
martint opened this issue Sep 28, 2016 · 5 comments
Closed

Planning error for correlated queries over Hive connector #6215

martint opened this issue Sep 28, 2016 · 5 comments
Assignees
@kokosing
Labels
bug

Comments

@martint
Copy link
Contributor

martint commented Sep 28, 2016

Running the following query as part of TestHiveDistributedQueries fails with a planning error. It seems to happen during partition pruning.

SELECT count(*) 
FROM orders o 
WHERE EXISTS (
    SELECT avg(l.orderkey) 
    FROM lineitem l 
    WHERE o.orderkey = l.orderkey 
    GROUP BY l.linenumber)
java.lang.IllegalArgumentException: Missing column assignment for orderkey

    at com.google.common.base.Preconditions.checkArgument(Preconditions.java:145)
    at com.facebook.presto.sql.planner.LookupSymbolResolver.getValue(LookupSymbolResolver.java:44)
    at com.facebook.presto.sql.planner.ExpressionInterpreter$Visitor.visitSymbolReference(ExpressionInterpreter.java:430)
    at com.facebook.presto.sql.tree.SymbolReference.accept(SymbolReference.java:38)
    at com.facebook.presto.sql.tree.AstVisitor.process(AstVisitor.java:22)
    at com.facebook.presto.sql.planner.ExpressionInterpreter$Visitor.visitComparisonExpression(ExpressionInterpreter.java:744)
    at com.facebook.presto.sql.tree.ComparisonExpression.accept(ComparisonExpression.java:133)
    at com.facebook.presto.sql.tree.AstVisitor.process(AstVisitor.java:22)
    at com.facebook.presto.sql.planner.ExpressionInterpreter.optimize(ExpressionInterpreter.java:254)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.shouldPrune(AddExchanges.java:659)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.lambda$planTableScan$4(AddExchanges.java:572)
    at com.facebook.presto.hive.HiveMetadata.lambda$getTableLayouts$5(HiveMetadata.java:1034)
    at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174)
    at java.util.Iterator.forEachRemaining(Iterator.java:116)
    at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
    at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
    at com.facebook.presto.hive.HiveMetadata.getTableLayouts(HiveMetadata.java:1035)
    at com.facebook.presto.spi.connector.classloader.ClassLoaderSafeConnectorMetadata.getTableLayouts(ClassLoaderSafeConnectorMetadata.java:67)
    at com.facebook.presto.metadata.MetadataManager.getLayouts(MetadataManager.java:337)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.planTableScan(AddExchanges.java:570)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.visitFilter(AddExchanges.java:513)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.visitFilter(AddExchanges.java:182)
    at com.facebook.presto.sql.planner.plan.FilterNode.accept(FilterNode.java:71)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.planChild(AddExchanges.java:1152)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.visitProject(AddExchanges.java:223)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.visitProject(AddExchanges.java:182)
    at com.facebook.presto.sql.planner.plan.ProjectNode.accept(ProjectNode.java:81)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.planChild(AddExchanges.java:1152)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.visitAggregation(AddExchanges.java:268)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.visitAggregation(AddExchanges.java:182)
    at com.facebook.presto.sql.planner.plan.AggregationNode.accept(AggregationNode.java:204)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.planChild(AddExchanges.java:1152)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.visitLimit(AddExchanges.java:477)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.visitLimit(AddExchanges.java:182)
    at com.facebook.presto.sql.planner.plan.LimitNode.accept(LimitNode.java:86)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.planChild(AddExchanges.java:1152)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.visitProject(AddExchanges.java:223)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.visitProject(AddExchanges.java:182)
    at com.facebook.presto.sql.planner.plan.ProjectNode.accept(ProjectNode.java:81)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.planChild(AddExchanges.java:1152)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.visitAggregation(AddExchanges.java:268)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.visitAggregation(AddExchanges.java:182)
    at com.facebook.presto.sql.planner.plan.AggregationNode.accept(AggregationNode.java:204)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.planChild(AddExchanges.java:1152)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.visitProject(AddExchanges.java:223)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.visitProject(AddExchanges.java:182)
    at com.facebook.presto.sql.planner.plan.ProjectNode.accept(ProjectNode.java:81)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.planChild(AddExchanges.java:1152)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.visitEnforceSingleRow(AddExchanges.java:243)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.visitEnforceSingleRow(AddExchanges.java:182)
    at com.facebook.presto.sql.planner.plan.EnforceSingleRowNode.accept(EnforceSingleRowNode.java:64)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.visitApply(AddExchanges.java:1140)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.visitApply(AddExchanges.java:182)
    at com.facebook.presto.sql.planner.plan.ApplyNode.accept(ApplyNode.java:96)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.planChild(AddExchanges.java:1152)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.visitFilter(AddExchanges.java:516)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.visitFilter(AddExchanges.java:182)
    at com.facebook.presto.sql.planner.plan.FilterNode.accept(FilterNode.java:71)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.planChild(AddExchanges.java:1152)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.visitAggregation(AddExchanges.java:268)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.visitAggregation(AddExchanges.java:182)
    at com.facebook.presto.sql.planner.plan.AggregationNode.accept(AggregationNode.java:204)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.planChild(AddExchanges.java:1152)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.visitOutput(AddExchanges.java:229)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.visitOutput(AddExchanges.java:182)
    at com.facebook.presto.sql.planner.plan.OutputNode.accept(OutputNode.java:81)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges.optimize(AddExchanges.java:144)
    at com.facebook.presto.sql.planner.LogicalPlanner.plan(LogicalPlanner.java:116)
    at com.facebook.presto.sql.planner.LogicalPlanner.plan(LogicalPlanner.java:107)
    at com.facebook.presto.execution.SqlQueryExecution.doAnalyzeQuery(SqlQueryExecution.java:291)
    at com.facebook.presto.execution.SqlQueryExecution.analyzeQuery(SqlQueryExecution.java:270)
    at com.facebook.presto.execution.SqlQueryExecution.start(SqlQueryExecution.java:228)
    at com.facebook.presto.execution.QueuedExecution.lambda$start$1(QueuedExecution.java:62)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
@martint martint added the bug label Sep 28, 2016
@martint
Copy link
Contributor Author

martint commented Sep 28, 2016

@kokosing, can you take a look?

@kokosing
Copy link
Contributor

Sure thing ;)

The query is similar to the query we have already talked about: #6155 (comment)

It should raise an exception that it is not supported for now. I will fix that.

Anyway I think that using EXISTS on aggregation does not make much sense. Correct me if I am wrong, but such query can be rewritten to:

SELECT count(*) 
FROM orders o 
WHERE EXISTS (
    SELECT 1
    FROM lineitem l 
    WHERE o.orderkey = l.orderkey)

And as such it should be working now in Presto.

I will post a PR that does this rewrite automatically.

@kokosing
Copy link
Contributor

Never mind my previous comment. I didn't know that TestHiveDistributedQueries is not executed on travis.

kokosing added a commit to Teradata/presto that referenced this issue Sep 28, 2016
@kokosing
Do not optimize expression with correlated symbols
1841923 
Do not optimize expressions which contains symbols to the outer query.

Fixes prestodb#6215
@kokosing kokosing mentioned this issue Sep 28, 2016
Closed
@kokosing
Copy link
Contributor

Here is a fix: #6217

I just disabled optmizing expression which contains correlated symbols to outer query. The next step would be to make ExpressionOptimizer be able to handle correlated symbols so for example some_bigint_correlated_symbol * null could evaluate to null etc.

@kokosing
Copy link
Contributor

I wanted to add that the mentioned next step will be only needed when ApplyNode become eligible to be physically executed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kokosing kokosing

Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@martint @kokosing