[CARBONDATA-2808] Insert into select is crashing as both are sharing the same task context #2591
Conversation
|
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7688/ |
|
Retest this please |
|
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6414/ |
|
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7691/ |
|
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6417/ |
|
retest sdv please |
| closeReader.apply() | ||
| close() | ||
| logStatistics(executionId, taskId, queryStartTime, model.getStatisticsRecorder, split) | ||
| context.addTaskCompletionListener { new QueryTaskCompletionListener(!isAdded, |
There was a problem hiding this comment.
Remove the duplicated code which is copied to QueryTaskCompletionListener from this class
|
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6087/ |
dhatchayani
force-pushed
the
CARBONDATA-2808
branch
2 times, most recently
from
43892f9
to
1d60c39
Compare
|
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6091/ |
dhatchayani
force-pushed
the
CARBONDATA-2808
branch
from
1d60c39
to
4f3e1a1
Compare
| @@ -390,4 +393,12 @@ public String toString() { | |||
| projection.getDimensions().size() + projection.getMeasures().size(), | |||
| filterExpressionResolverTree.getFilterExpression().toString()); | |||
| } | |||
|
|
|||
| public boolean isFreeUnsafeMemory() { | |||
There was a problem hiding this comment.
change the method name
| val listeners = CarbonReflectionUtils.getField("onCompleteCallbacks", context) | ||
| .asInstanceOf[ArrayBuffer[TaskCompletionListener]] | ||
| val isAdded = listeners.exists(p => p.isInstanceOf[InsertTaskCompletionListener]) | ||
| if (isAdded) { |
There was a problem hiding this comment.
If condition is not required
|
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7696/ |
|
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6422/ |
dhatchayani
force-pushed
the
CARBONDATA-2808
branch
from
4f3e1a1
to
f3fab33
Compare
|
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6093/ |
|
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7700/ |
|
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6426/ |
|
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7704/ |
|
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6430/ |
dhatchayani
changed the title
| @@ -390,4 +393,12 @@ public String toString() { | |||
| projection.getDimensions().size() + projection.getMeasures().size(), | |||
| filterExpressionResolverTree.getFilterExpression().toString()); | |||
| } | |||
|
|
|||
| public boolean getFreeUnsafeMemory() { | |||
There was a problem hiding this comment.
supposed to be istFreeUnsafeMemory
dhatchayani
force-pushed
the
CARBONDATA-2808
branch
from
f3fab33
to
1fd0854
Compare
…the same task context
|
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6096/ |
|
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7709/ |
|
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6435/ |
|
LGTM |
|
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6102/ |
Insert into select is failing as both are running as single task, both are sharing the same taskcontext and resources are cleared once if any one of the RDD(Select query's ScanRDD) is completed, so the other RDD(LoadRDD) running is crashing as it is trying to access the cleared memory. Solution: Check if any other RDD is sharing the same task context. If so, don't the clear the resource at that time, the other RDD which shared the context should clear the memory once after the task is finished. This closes apache#2591
Insert into select is failing as both are running as single task, both are sharing the same taskcontext and resources are cleared once if any one of the RDD(Select query's ScanRDD) is completed, so the other RDD(LoadRDD) running is crashing as it is trying to access the cleared memory. Solution: Check if any other RDD is sharing the same task context. If so, don't the clear the resource at that time, the other RDD which shared the context should clear the memory once after the task is finished. This closes #2591
Problem:
Insert into select is failing as both are running as single task, both are sharing the same taskcontext and resources are cleared once if any one of the RDD(Select query's ScanRDD) is completed, so the other RDD(LoadRDD) running is crashing as it is trying to access the cleared memory.
Solution:
Check if any other RDD is sharing the same task context. If so, don't the clear the resource at that time, the other RDD which shared the context should clear the memory once after the task is finished.
Any interfaces changed?
Any backward compatibility impacted?
Document update required?
Testing done
Manual Testing
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.