Skip to content
Permalink
Browse files
[CARBONDATA-4306] Fix Query Performance issue for Spark 3.1
Why is this PR needed?
Some non-partition filters, which cannot be handled by carbon, is not pushed down to spark.

What changes were proposed in this PR?
If partition filters is non empty, then the filter column is not partition column, then push the filter to spark

This closes #4252
  • Loading branch information
Indhumathi27 authored and kunal642 committed Mar 7, 2022
1 parent e25d5b6 commit a838531f243f4b826778b51d1f238461d3414fe4
Showing 1 changed file with 3 additions and 3 deletions.
@@ -158,10 +158,10 @@ private[sql] object CarbonSourceStrategy extends SparkStrategy {
SparkSession.getActiveSession.get,
relation.catalogTable.get.identifier
)
// remove dynamic partition filter from predicates
filterPredicates = CarbonToSparkAdapter.getDataFilter(partitionSet,
allPredicates, partitionsFilter)
}
// remove dynamic partition filter from predicates
filterPredicates = CarbonToSparkAdapter.getDataFilter(partitionSet,
allPredicates, partitionsFilter)
val table = relation.relation.asInstanceOf[CarbonDatasourceHadoopRelation]
val projects = rawProjects.map {p =>
p.transform {

0 comments on commit a838531

Please sign in to comment.