New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Planning error for correlated queries over Hive connector #6215

Closed
martint opened this Issue Sep 28, 2016 · 5 comments

Comments

Projects
None yet
2 participants
@martint
Contributor

martint commented Sep 28, 2016

Running the following query as part of TestHiveDistributedQueries fails with a planning error. It seems to happen during partition pruning.

SELECT count(*) 
FROM orders o 
WHERE EXISTS (
    SELECT avg(l.orderkey) 
    FROM lineitem l 
    WHERE o.orderkey = l.orderkey 
    GROUP BY l.linenumber)
java.lang.IllegalArgumentException: Missing column assignment for orderkey

    at com.google.common.base.Preconditions.checkArgument(Preconditions.java:145)
    at com.facebook.presto.sql.planner.LookupSymbolResolver.getValue(LookupSymbolResolver.java:44)
    at com.facebook.presto.sql.planner.ExpressionInterpreter$Visitor.visitSymbolReference(ExpressionInterpreter.java:430)
    at com.facebook.presto.sql.tree.SymbolReference.accept(SymbolReference.java:38)
    at com.facebook.presto.sql.tree.AstVisitor.process(AstVisitor.java:22)
    at com.facebook.presto.sql.planner.ExpressionInterpreter$Visitor.visitComparisonExpression(ExpressionInterpreter.java:744)
    at com.facebook.presto.sql.tree.ComparisonExpression.accept(ComparisonExpression.java:133)
    at com.facebook.presto.sql.tree.AstVisitor.process(AstVisitor.java:22)
    at com.facebook.presto.sql.planner.ExpressionInterpreter.optimize(ExpressionInterpreter.java:254)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.shouldPrune(AddExchanges.java:659)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.lambda$planTableScan$4(AddExchanges.java:572)
    at com.facebook.presto.hive.HiveMetadata.lambda$getTableLayouts$5(HiveMetadata.java:1034)
    at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174)
    at java.util.Iterator.forEachRemaining(Iterator.java:116)
    at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
    at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
    at com.facebook.presto.hive.HiveMetadata.getTableLayouts(HiveMetadata.java:1035)
    at com.facebook.presto.spi.connector.classloader.ClassLoaderSafeConnectorMetadata.getTableLayouts(ClassLoaderSafeConnectorMetadata.java:67)
    at com.facebook.presto.metadata.MetadataManager.getLayouts(MetadataManager.java:337)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.planTableScan(AddExchanges.java:570)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.visitFilter(AddExchanges.java:513)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.visitFilter(AddExchanges.java:182)
    at com.facebook.presto.sql.planner.plan.FilterNode.accept(FilterNode.java:71)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.planChild(AddExchanges.java:1152)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.visitProject(AddExchanges.java:223)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.visitProject(AddExchanges.java:182)
    at com.facebook.presto.sql.planner.plan.ProjectNode.accept(ProjectNode.java:81)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.planChild(AddExchanges.java:1152)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.visitAggregation(AddExchanges.java:268)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.visitAggregation(AddExchanges.java:182)
    at com.facebook.presto.sql.planner.plan.AggregationNode.accept(AggregationNode.java:204)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.planChild(AddExchanges.java:1152)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.visitLimit(AddExchanges.java:477)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.visitLimit(AddExchanges.java:182)
    at com.facebook.presto.sql.planner.plan.LimitNode.accept(LimitNode.java:86)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.planChild(AddExchanges.java:1152)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.visitProject(AddExchanges.java:223)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.visitProject(AddExchanges.java:182)
    at com.facebook.presto.sql.planner.plan.ProjectNode.accept(ProjectNode.java:81)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.planChild(AddExchanges.java:1152)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.visitAggregation(AddExchanges.java:268)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.visitAggregation(AddExchanges.java:182)
    at com.facebook.presto.sql.planner.plan.AggregationNode.accept(AggregationNode.java:204)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.planChild(AddExchanges.java:1152)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.visitProject(AddExchanges.java:223)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.visitProject(AddExchanges.java:182)
    at com.facebook.presto.sql.planner.plan.ProjectNode.accept(ProjectNode.java:81)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.planChild(AddExchanges.java:1152)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.visitEnforceSingleRow(AddExchanges.java:243)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.visitEnforceSingleRow(AddExchanges.java:182)
    at com.facebook.presto.sql.planner.plan.EnforceSingleRowNode.accept(EnforceSingleRowNode.java:64)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.visitApply(AddExchanges.java:1140)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.visitApply(AddExchanges.java:182)
    at com.facebook.presto.sql.planner.plan.ApplyNode.accept(ApplyNode.java:96)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.planChild(AddExchanges.java:1152)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.visitFilter(AddExchanges.java:516)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.visitFilter(AddExchanges.java:182)
    at com.facebook.presto.sql.planner.plan.FilterNode.accept(FilterNode.java:71)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.planChild(AddExchanges.java:1152)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.visitAggregation(AddExchanges.java:268)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.visitAggregation(AddExchanges.java:182)
    at com.facebook.presto.sql.planner.plan.AggregationNode.accept(AggregationNode.java:204)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.planChild(AddExchanges.java:1152)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.visitOutput(AddExchanges.java:229)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter.visitOutput(AddExchanges.java:182)
    at com.facebook.presto.sql.planner.plan.OutputNode.accept(OutputNode.java:81)
    at com.facebook.presto.sql.planner.optimizations.AddExchanges.optimize(AddExchanges.java:144)
    at com.facebook.presto.sql.planner.LogicalPlanner.plan(LogicalPlanner.java:116)
    at com.facebook.presto.sql.planner.LogicalPlanner.plan(LogicalPlanner.java:107)
    at com.facebook.presto.execution.SqlQueryExecution.doAnalyzeQuery(SqlQueryExecution.java:291)
    at com.facebook.presto.execution.SqlQueryExecution.analyzeQuery(SqlQueryExecution.java:270)
    at com.facebook.presto.execution.SqlQueryExecution.start(SqlQueryExecution.java:228)
    at com.facebook.presto.execution.QueuedExecution.lambda$start$1(QueuedExecution.java:62)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
@martint

This comment has been minimized.

Show comment
Hide comment
@martint

martint Sep 28, 2016

Contributor

@kokosing, can you take a look?

Contributor

martint commented Sep 28, 2016

@kokosing, can you take a look?

@kokosing

This comment has been minimized.

Show comment
Hide comment
@kokosing

kokosing Sep 28, 2016

Member

Sure thing ;)

The query is similar to the query we have already talked about: #6155 (comment)

It should raise an exception that it is not supported for now. I will fix that.

Anyway I think that using EXISTS on aggregation does not make much sense. Correct me if I am wrong, but such query can be rewritten to:

SELECT count(*) 
FROM orders o 
WHERE EXISTS (
    SELECT 1
    FROM lineitem l 
    WHERE o.orderkey = l.orderkey)

And as such it should be working now in Presto.

I will post a PR that does this rewrite automatically.

Member

kokosing commented Sep 28, 2016

Sure thing ;)

The query is similar to the query we have already talked about: #6155 (comment)

It should raise an exception that it is not supported for now. I will fix that.

Anyway I think that using EXISTS on aggregation does not make much sense. Correct me if I am wrong, but such query can be rewritten to:

SELECT count(*) 
FROM orders o 
WHERE EXISTS (
    SELECT 1
    FROM lineitem l 
    WHERE o.orderkey = l.orderkey)

And as such it should be working now in Presto.

I will post a PR that does this rewrite automatically.

@kokosing

This comment has been minimized.

Show comment
Hide comment
@kokosing

kokosing Sep 28, 2016

Member

Never mind my previous comment. I didn't know that TestHiveDistributedQueries is not executed on travis.

Member

kokosing commented Sep 28, 2016

Never mind my previous comment. I didn't know that TestHiveDistributedQueries is not executed on travis.

kokosing added a commit to Teradata/presto that referenced this issue Sep 28, 2016

Do not optimize expression with correlated symbols
Do not optimize expressions which contains symbols to the outer query.

Fixes prestodb#6215
@kokosing

This comment has been minimized.

Show comment
Hide comment
@kokosing

kokosing Sep 28, 2016

Member

Here is a fix: #6217

I just disabled optmizing expression which contains correlated symbols to outer query. The next step would be to make ExpressionOptimizer be able to handle correlated symbols so for example some_bigint_correlated_symbol * null could evaluate to null etc.

Member

kokosing commented Sep 28, 2016

Here is a fix: #6217

I just disabled optmizing expression which contains correlated symbols to outer query. The next step would be to make ExpressionOptimizer be able to handle correlated symbols so for example some_bigint_correlated_symbol * null could evaluate to null etc.

@kokosing

This comment has been minimized.

Show comment
Hide comment
@kokosing

kokosing Sep 28, 2016

Member

I wanted to add that the mentioned next step will be only needed when ApplyNode become eligible to be physically executed

Member

kokosing commented Sep 28, 2016

I wanted to add that the mentioned next step will be only needed when ApplyNode become eligible to be physically executed

@martint martint closed this in 17bd33f Sep 28, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment