[CARBONDATA-3620][CARBONDATA-3622]: Update does not load cache in memory, behavior inconsistent with scenario when index server is not running #3511
Conversation
vikramahuja1001
changed the title
|
Build Success with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/1187/ |
vikramahuja1001
force-pushed
the
update_issue
branch
from
36af10e
to
2a27d06
Compare
|
Build Success with Spark 2.2.1, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.2/1197/ |
|
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1207/ |
|
Build Success with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/1189/ |
| @@ -473,6 +473,13 @@ object CarbonDataRDDFactory { | |||
| .STATUS_FILE_UPDATION_FAILURE | |||
| return null | |||
| } | |||
| // code to handle Pre-Priming cache for update command | |||
There was a problem hiding this comment.
may be you can remove the empty if block above this
| @@ -473,6 +473,13 @@ object CarbonDataRDDFactory { | |||
| .STATUS_FILE_UPDATION_FAILURE | |||
| return null | |||
| } | |||
| // code to handle Pre-Priming cache for update command | |||
| if (!segmentFiles.isEmpty) { | |||
| val segmentstoprime = segmentFiles.asScala.map(iterator => iterator.getSegmentNo).toSeq | |||
vikramahuja1001
force-pushed
the
update_issue
branch
from
d86c7b1
to
2a375d0
Compare
|
Build Success with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/1198/ |
|
Build Success with Spark 2.2.1, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.2/1206/ |
|
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1216/ |
vikramahuja1001
changed the title
vikramahuja1001
force-pushed
the
update_issue
branch
from
4c4b231
to
9b4ff28
Compare
|
Build Success with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/1209/ |
|
@vikramahuja1001 in description, in Problem section, please list problem of both jira's and then in solution section, write corresponding solution points. |
|
Build Success with Spark 2.2.1, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.2/1218/ |
|
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1228/ |
|
Build Success with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/1225/ |
|
Build Success with Spark 2.2.1, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.2/1235/ |
|
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1244/ |
|
LGTM |
|
retest this please |
|
Build Success with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/1282/ |
|
Build Success with Spark 2.2.1, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.2/1291/ |
|
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1302/ |
|
@vikramahuja1001 please resolve conflicts |
|
@vikramahuja1001 please rebase |
vikramahuja1001
force-pushed
the
update_issue
branch
from
b3efb7c
to
c4eae44
Compare
vikramahuja1001
force-pushed
the
update_issue
branch
from
c4eae44
to
46dc1bd
Compare
|
retest this please |
|
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1499/ |
|
@akashrn5 , done. Please check |
|
retest this please |
|
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1570/ |
|
LGTM |
1 similar comment
|
LGTM |
Problems
1: Segments for prepriming were never being sent to the index server after update success.
2 While dropping a table the table is dropped first and then clear datamap is called, If the table is being accessed somewhere while clearing datamaps in the index server then it will fail and throw exception, thus the cache value will never be cleared.
Solutions
1: Segments for prepriming to the index server added after Update success.
2: While dropping the table, to clear cache first and then drop the table.
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
Any interfaces changed?
Any backward compatibility impacted?
Document update required?
Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.