[CARBONDATA-3217] Optimize implicit filter expression performance by removing extra serialization #3039
Conversation
1. Removed serialization all the implicit filter values in each task. Instead serialized values only for the blocks going to particular task 2. Removed 2 times deserialization of implicit filter values in executor for each task. 1 time is sufficient
manishgupta88
changed the title
|
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2094/ |
|
Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10348/ |
|
Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2299/ |
| blockletIds = new HashSet<>(); | ||
| blockIdToBlockletIdMapping.put(blockId, blockletIds); | ||
| } | ||
| blockletIds.add(Integer.parseInt(blockletPath.substring(blockId.length() + 1))); |
There was a problem hiding this comment.
Better to catch the NumberFormatException for Integer.parseInt
There was a problem hiding this comment.
Not required to catch the NumberFormatException. Blocklet Id is always expected to be an integer number and in the complete flow it is used as an integer. So if there is any exception in the conversion then the code problem is not here but from the other part of code. So if we handle the exception here the actual cause will get suppressed here
|
LGTM |
1 similar comment
|
LGTM |
…removing extra serialization Fixed performance issue for Implicit filter column 1. Removed serialization all the implicit filter values in each task. Instead serialized values only for the blocks going to particular task 2. Removed 2 times deserialization of implicit filter values in executor for each task. 1 time is sufficient This closes #3039
…removing extra serialization Fixed performance issue for Implicit filter column 1. Removed serialization all the implicit filter values in each task. Instead serialized values only for the blocks going to particular task 2. Removed 2 times deserialization of implicit filter values in executor for each task. 1 time is sufficient This closes apache#3039
Fixed performance issue for Implicit filter column
No
No
No
Added UT
NA