[SPARK-26626][SQL] Maximum size for repeatedly substituted aliases in SQL expressions

[j-esse]

What changes were proposed in this pull request?

This adds a spark.sql.maxRepeatedAliasSize config option, which specifies the maximum size of an aliased expression to be substituted (in CollapseProject and PhysicalOperation). This prevents large aliased expressions from being substituted multiple times and exploding the size of the expression tree, eventually OOMing the driver.

The default config value of 100 was chosen through testing to find the optimally performant value:

How was this patch tested?

Added unit tests, and did manual testing

[mccheah]

ok to test

[vinooganesh]

@cloud-fan could we get you to take a look here? should hopefully be a quick review

[SparkQA]

Test build #101280 has finished for PR 23556 at commit eaaf8a3.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

I'm not sure about this approach. I think there is no problem if we repeat an attribute 1000 times, but it can be a problem if we repeat an expensive expression twice (like UDF).

How about we just blacklist UDF in CollapseProject?

[j-esse]

@cloud-fan Ah sorry I think I didn't explain the problem clearly enough - the OOMs happen inside CollapseProject while performing the optimisation (and inside PhysicalOperation). The recursive alias substitution grows the expression tree so large that it OOMs. We're not using UDFs or any expensive expressions.

To repro, try this in Spark Shell:

var df = Seq(1, 2, 3).toDF("a").withColumn("b", lit(10)).cache()
for( i <- 1 to 20 ) {
  df = df.select(('a + 'b).as('a), ('a - 'b).as('b))
 }
df.collect()

[maropu]

Not enough to just keep an Alias on the top only like CleanupAliases in the analyzer?

[cloud-fan]

I'm not sure about using treeSize as the cost of an expression. UDF can be very expensive even if its treeSize is 1.

How about we simplify it with a blacklist? e.g. UDF is expensive and we shouldn't collapse projects if udf is repeated.

[j-esse]

This isn't trying to determine the cost of the expression - the cost of the expression is irrelevant here, we're just trying to determine the size of the expression itself (using tree size as a proxy for memory size). That way, if the expression is too large (takes up too much memory) we can prevent OOMs by not de-aliasing it multiple times (and thus greatly increasing the amount of heap the expression tree takes up).

[cloud-fan]

so your fix only care about memory usage of the expressions, instead of execution time?

[j-esse]

@cloud-fan That's right, the primary concern is memory usage, since the exponential increase in memory usage currently causes crashes (due to OOMs), time outs, and performance issues.

[j-esse]

@maropu no I don't think this is related to CleanupAliases - we can't clean up the aliases, because they still need to be used in the expression; we just can't substitute them (if they're large) or we risk OOMing

[SparkQA]

Test build #101648 has finished for PR 23556 at commit 22f594a.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

Is the purpose to give up this rule basically? Why don't we consider using spark.sql.optimizer.excludedRules? I think it's more general way to resolve such issues.

[j-esse]

@HyukjinKwon The purpose is to improve the rule so that it only applies when it will yield a performance improvement (and not apply when it could cause memory issues). This was the preferred solution, since if we excluded the rule entirely we wouldn't benefit from it in the instances where it would be beneficial.

[HyukjinKwon]

So, basically what you want to do is to give up a rule given a condition because processing huge tree causes OOM issue only in the driver. Am I correct?

What's the diff if we set the threshold spark.sql.maxRepeatedAliasSize to set the specific number based upon the rough estimation vs explicitly excluding the rule by spark.sql.optimizer.excludedRules based on user's rough estimation?

[j-esse]

@HyukjinKwon it's not that processing a huge tree causes an OOM, it's that the user can write a small tree, that seems very reasonable to execute, but under the hood the optimiser turns it into a huge tree that OOMs. The user doesn't know beforehand that the optimiser issue is going to happen, in order to disable the rule. It takes a lot of debugging, looking through stack traces, etc, to identify that the OOM is caused by CollapseProject and that you can disable it. Also, we typically run many different queries within a spark session, and wouldn't want to disable CollapseProject for all of them.

This change means that we can still run CollapseProject, we just don't substitute overly large aliases. In the types of query we had problems with, this means that it will collapse the query until the aliases get too large, and then stop. So we still do apply CollapseProject to every query, we just stop substituting any alias the gets too large.

spark.sql.maxRepeatedAliasSize just determines the size of alias tree that is determined to be too large to efficiently substitute multiple times. The default value of 100 was determined by some basic testing to find the best perf balance (see charts at top), but happy to tweak this if you don't htink it's appropriate?

[SparkQA]

Test build #103512 has finished for PR 23556 at commit 11e2b8a.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #103516 has finished for PR 23556 at commit 2f5c55e.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #103630 has finished for PR 23556 at commit cfa09db.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

It does add something automatic stuff but I don't think this is so much worth since we already have a general mechanism. Note that you can also increases the driver side's memory. How does it relate with this configuration?

[j-esse]

@HyukjinKwon increasing the driver memory unfortunately isn't an option, because due to the exponential tree size explosion, the necessary memory would be much larger than that available on most servers. Also, users wouldn't know that they needed a very large driver memory size, because they can be running small queries over small data.

[HyukjinKwon]

If there is not anything I missed, -1 from me. Because:

1. We already have a general mechanism to disable a specific rule
2. It does some automatic stuff but still requires rough estimation to set the value - sounds virtually no diff if we disable the rule based upon a rough estimation for the job
3. If the issue is related with driver side's memory, the value has to have some logic related with it. Looks that's missing.

[j-esse]

@HyukjinKwon I think there are a few more things:
The issue doesn't just manifest in CollapseProject, it happens in collectProjectsAndFilters in PhysicalOperation as well (https://github.com/apache/spark/pull/23556/files#diff-820e654df2a5133c0f86c17e2fc5512e), even when CollapseProject is excluded. We're actually investigating another instance of this issue, which we think lies in PushDownPredicate, we might have to make another fix there.

In response to your concerns:

1. As I said above, we don't want to disable the rule - for most queries, the rule will be applied unchanged. For some queries, the rule will be partially applied (some aliases that get overly large will stop being substituted). But CollapseProject will never be fully disabled (except in the unlikely case that the original aliases have more than spark.sql.maxRepeatedAliasSize aliases)
2. spark.sql.maxRepeatedAliasSize really just needs to be a high threshold to catch exponential alias expansion, we'd generally never expect anyone to need to change the value from the default. We can think about other heuristics to detect exponential alias expansion, if you're concerned about having a fixed value?
3. Sorry, I'm not quite sure what you mean here? The issue isn't specifically around driver memory OOMs, that's just one resulting effect of the explosive alias expansion - other effects include slowness, hangs, etc.

[HyukjinKwon]

We still can disable the rule for job-based. This PR sounds it only adds some fine-grind partial automatic logic to disable the rule on the certain condition, which still needs a manual estimation from users to set the configuration value. I don't see so much value on it to be honest. How common the case is it, and how much does it save the memory before/after?

And, are there only applicable in CollapseProject and PhysicalOperation? I don't think we should add each configuration for each rule to set the threshold. Looks an overkill as well.

[HyukjinKwon]

BTW, can you describe the chart in the PR description about what's x and y?

[j-esse]

@HyukjinKwon just to reiterate:

• We don't want to disable the rule, and this PR doesn't disable the rule in any circumstances, it just improves it - it will still collapse projects, it will just not fully collapse some (when it doesn't make sense to). The whole point if the optimiser is to make queries better, and in some cases CollapseProject is failing at this, and definitively making queries worse.
• Currently, even if you do disable CollapseProject, you will still have problems (e.g. OOMs) since the issue also exists in PhysicalOperation (and in PushDownPredicate, which we're working on another fix for)
• Even if the solution was to disable CollapseProject (which it isn't), it would not be obvious to Spark users that this issue exists, or when they might encounter it, or even that they are hitting the issue - they will just encounter slowness and OOMs, and though extensive debugging, stack trace checks, etc may eventually discover that CollapseProject is causing the problem. (But to reiterate, currently even disabling CollapseProject doesn't fix the issue). The only other solution would be to disable CollapseProject in Spark by default, and have well-documented warnings about the issues it can cause, and that Spark users should only enable it if they're confident that thy won't hit these issues.
• There is no manual configuration or estimation required - we set spark.sql.maxRepeatedAliasSize to a sensible default that we expect to maintain or improve performance in all cases.
• We have been hitting this issue in a range of situations - any query that applies a series of changes to a column will hit it.

In the chart above, the x axis is the spark.sql.maxRepeatedAliasSize setting, and the y axis is the time taken for execution of our test query (when the setting is 1300 or higher it starts to OOM)

Here's a simple example to illustrate the problem:

Take this simple query, running in Spark shell, which simulates making multiple (10) changes to a column:

scala> var df = Seq(1, 2, 3).toDF("a").withColumn("b", lit(10)).cache()
df: org.apache.spark.sql.Dataset[org.apache.spark.sql.Row] = [a: int, b: int]

scala> for( i <- 1 to 10 ) {
  df = df.select(('a + 'b).as('a), ('a - 'b).as('b))
}

This is the original query plan:

Project [(a#51 + b#52) AS a#55, (a#51 - b#52) AS b#56]
+- Project [(a#47 + b#48) AS a#51, (a#47 - b#48) AS b#52]
   +- Project [(a#43 + b#44) AS a#47, (a#43 - b#44) AS b#48]
      +- Project [(a#39 + b#40) AS a#43, (a#39 - b#40) AS b#44]
         +- Project [(a#35 + b#36) AS a#39, (a#35 - b#36) AS b#40]
            +- Project [(a#31 + b#32) AS a#35, (a#31 - b#32) AS b#36]
               +- Project [(a#27 + b#28) AS a#31, (a#27 - b#28) AS b#32]
                  +- Project [(a#23 + b#24) AS a#27, (a#23 - b#24) AS b#28]
                     +- Project [(a#19 + b#20) AS a#23, (a#19 - b#20) AS b#24]
                        +- Project [(a#4 + b#6) AS a#19, (a#4 - b#6) AS b#20]
                           +- Project [a#4, 10 AS b#6]
                              +- Project [value#1 AS a#4]
                                 +- LocalRelation [value#1]

Here is the optimized query plan, with this fix applied (using the default spark.sql.maxRepeatedAliasSize of 100) - you can see that CollapseProject does still collapse, but not fully - it collapses it down to 2 projects, since it doesn't substitute aliases that get too large:

scala> print(df.queryExecution.optimizedPlan)
Project [((((a#39 + b#40) + (a#39 - b#40)) + ((a#39 + b#40) - (a#39 - b#40))) + (((a#39 + b#40) + (a#39 - b#40)) - ((a#39 + b#40) - (a#39 - b#40)))) AS a#55, ((((a#39 + b#40) + (a#39 - b#40)) + ((a#39 + b#40) - (a#39 - b#40))) - (((a#39 + b#40) + (a#39 - b#40)) - ((a#39 + b#40) - (a#39 - b#40)))) AS b#56]
+- Project [((((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) + ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6))))) + (((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) - ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))))) AS a#39, ((((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) + ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6))))) - (((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) - ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))))) AS b#40]
+- InMemoryRelation [a#4, b#6], StorageLevel(disk, memory, deserialized, 1 replicas)
+- LocalTableScan [a#4, b#6]

Now below is the optimized query plan without this fix applied, where CollapseProject collapses everything down to 1 project. You can see how the query tree explodes exponentially. This is only a very simple query, with only 10 changes - more complex queries with, with 100s of changes, will easily OOM. We've seen vast reductions in the memory required for queries - without the fix, some still OOM using all available server memory; with the fix those queries run fast with only 512MB.

scala> print(df.queryExecution.optimizedPlan)
Project [((((((((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) + ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6))))) + (((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) - ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))))) + ((((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) + ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6))))) - (((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) - ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6))))))) + (((((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) + ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6))))) + (((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) - ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))))) - ((((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) + ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6))))) - (((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) - ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))))))) + ((((((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) + ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6))))) + (((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) - ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))))) + ((((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) + ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6))))) - (((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) - ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6))))))) - (((((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) + ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6))))) + (((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) - ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))))) - ((((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) + ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6))))) - (((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) - ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6))))))))) + (((((((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) + ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6))))) + (((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) - ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))))) + ((((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) + ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6))))) - (((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) - ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6))))))) + (((((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) + ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6))))) + (((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) - ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))))) - ((((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) + ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6))))) - (((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) - ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))))))) - ((((((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) + ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6))))) + (((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) - ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))))) + ((((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) + ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6))))) - (((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) - ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6))))))) - (((((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) + ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6))))) + (((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) - ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))))) - ((((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) + ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6))))) - (((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) - ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))))))))) AS a#55, ((((((((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) + ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6))))) + (((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) - ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))))) + ((((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) + ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6))))) - (((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) - ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6))))))) + (((((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) + ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6))))) + (((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) - ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))))) - ((((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) + ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6))))) - (((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) - ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))))))) + ((((((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) + ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6))))) + (((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) - ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))))) + ((((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) + ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6))))) - (((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) - ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6))))))) - (((((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) + ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6))))) + (((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) - ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))))) - ((((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) + ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6))))) - (((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) - ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6))))))))) - (((((((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) + ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6))))) + (((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) - ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))))) + ((((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) + ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6))))) - (((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) - ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6))))))) + (((((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) + ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6))))) + (((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) - ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))))) - ((((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) + ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6))))) - (((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) - ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))))))) - ((((((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) + ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6))))) + (((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) - ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))))) + ((((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) + ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6))))) - (((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) - ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6))))))) - (((((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) + ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6))))) + (((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) - ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))))) - ((((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) + ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6))))) - (((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) + (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))) - ((((a#4 + b#6) + (a#4 - b#6)) + ((a#4 + b#6) - (a#4 - b#6))) - (((a#4 + b#6) + (a#4 - b#6)) - ((a#4 + b#6) - (a#4 - b#6)))))))))) AS b#56]
+- InMemoryRelation [a#4, b#6], StorageLevel(disk, memory, deserialized, 1 replicas)
+- LocalTableScan [a#4, b#6]

[HyukjinKwon]

@j-esse, to reiterate,

• It disables a specific rule on a certain condition - that's why you have multiple projects after this PR.

• I don't think we should add each configuration for each rule to avoid this. Since you already see similar instances, I think it's better to have a more general solution rather than to fix each rule.

• You're basically just working around the issue within Spark. If you have multiple plans, it will cause OOM anyway.

• The default is measured based on numbers but the problem is driver memory size. Why it's not taken into account?

[AmplabJenkins]

Can one of the admins verify this patch?

[HyukjinKwon]

Closing this due to author's inactivity.