New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use spark sql optimizer to optimize two ranges join #11
Comments
Hi @bithw1 I'll try to take a look today or tomorrow. Best regards, |
The range method returns a DataFrame with a single one column called id. It's why the engines looks for it in your query declaration:
If you rewrite your query like this it should work: val spark = SparkSession.builder().master("local").appName("SparkTest").enableHiveSupport().getOrCreate()
spark.experimental.extraOptimizations = Seq(RangeIntersectRule)
spark.range(10, 40).createOrReplaceTempView("t1")
spark.range(20, 50).createOrReplaceTempView("t2")
val df = spark.sql("select t1.id from t1 join t2 on t1.id = t2.id")
df.explain(true)
df.show(truncate = false) Best regards, |
Thanks @bartosz25 , but when rewriting it using column When using
|
Hi @bartosz25 I wrap
|
Hi @bithw1 Sorry, I missed your message last week. I'll add the topic of extra optimizations to my backlog and try to answer your question here when I'll write about it. Best regards, |
Sure, thank you @bartosz25 ! |
Hi @bithw1 Today I started the topic of custom optimizations. Since the topic is quite new for me, I will go slowly from the basics and try to cover more advanced concepts at the end. The first post is there : https://www.waitingforcode.com/apache-spark-sql/introduction-custom-optimization-apache-spark-sql/read Best regards, |
That's great, thanks @bartosz25. Looking forward to your great posts and learn, :-) |
Hi @bartosz25 ,
I got a question that I would like you to help take a look, thank you.
I want to use spark sql optimizer to optimize two ranges join , just to calculate two range intersection,so that it can avoid join
The rule takes effect, but it throws exception as follows, looks I haven't implemented the apply method appropriately
The plan is
The exception is:
The text was updated successfully, but these errors were encountered: