Skip to content

[CARBONDATA-4225] Fix Update performance issues when auto merge compaction is enabled#4156

Closed
Indhumathi27 wants to merge 1 commit intoapache:masterfrom
Indhumathi27:update_issue
Closed

[CARBONDATA-4225] Fix Update performance issues when auto merge compaction is enabled#4156
Indhumathi27 wants to merge 1 commit intoapache:masterfrom
Indhumathi27:update_issue

Conversation

@Indhumathi27
Copy link
Copy Markdown
Contributor

@Indhumathi27 Indhumathi27 commented Jun 21, 2021 •
edited
Loading

Why is this PR needed?

  1. When auto-compaction is enabled, during update, we are trying to do compaction after Insert. Auto-Compaction throws exception, after multiple retries. Carbon does not allow concurrent compaction and Update.
  2. dataframe.rdd.isEmpty will launch a Job. This code is called two times in code, which is not reused.

What changes were proposed in this PR?

  1. Avoid trying to do Auto-compaction during Update.
  2. Reuse dataframe.rdd.isEmpty and avoided launching a Job.

Does this PR introduce any user interface change?

  • No

Is any new testcase added?

  • Yes

@CarbonDataQA2
Copy link
Copy Markdown

CarbonDataQA2 commented Jun 21, 2021

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5595/

@CarbonDataQA2
Copy link
Copy Markdown

CarbonDataQA2 commented Jun 21, 2021

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/3851/

@Indhumathi27
Copy link
Copy Markdown
Contributor Author

Indhumathi27 commented Jun 21, 2021

retest this please

@CarbonDataQA2
Copy link
Copy Markdown

CarbonDataQA2 commented Jun 21, 2021

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5596/

@CarbonDataQA2
Copy link
Copy Markdown

CarbonDataQA2 commented Jun 21, 2021

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/3853/

@Indhumathi27 Indhumathi27 changed the title [WIP] Fix Update performance issues when auto merge compaction is enabled [CARBONDATA-4225] Fix Update performance issues when auto merge compaction is enabled Jun 22, 2021
akashrn5
akashrn5 reviewed Jun 22, 2021
View reviewed changes
...ark/src/test/scala/org/apache/carbondata/spark/testsuite/iud/UpdateCarbonTableTestCase.scala Outdated
sql(s"""LOAD DATA LOCAL INPATH '$resourcesPath/IUD/dest.csv' INTO table iud.autoMergeUpdate""")
sql("update iud.autoMergeUpdate up_TAble set(up_table.C1)=('abc')").show()
sql(s"""LOAD DATA LOCAL INPATH '$resourcesPath/IUD/dest.csv' INTO table iud.autoMergeUpdate""")
sql("update iud.autoMergeUpdate up_TAble set(up_table.C1)=('abcd')").show()
Copy link
Copy Markdown
Contributor

@akashrn5 akashrn5 Jun 22, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here in this test case may be can add assert for segment number by changing the compaction threshold

Copy link
Copy Markdown
Contributor Author

@Indhumathi27 Indhumathi27 Jun 22, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

auto compaction does not happen without this fix. Added a assert to check the time difference

@CarbonDataQA2
Copy link
Copy Markdown

CarbonDataQA2 commented Jun 22, 2021

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5609/

@CarbonDataQA2
Copy link
Copy Markdown

CarbonDataQA2 commented Jun 22, 2021

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/3864/

@brijoobopanna
Copy link
Copy Markdown
Contributor

brijoobopanna commented Jun 22, 2021

retest this please

@CarbonDataQA2
Copy link
Copy Markdown

CarbonDataQA2 commented Jun 22, 2021

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5612/

@CarbonDataQA2
Copy link
Copy Markdown

CarbonDataQA2 commented Jun 22, 2021

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/3867/

@akashrn5
Copy link
Copy Markdown
Contributor

akashrn5 commented Jun 23, 2021

retest this please

@CarbonDataQA2
Copy link
Copy Markdown

CarbonDataQA2 commented Jun 23, 2021

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/3870/

@CarbonDataQA2
Copy link
Copy Markdown

CarbonDataQA2 commented Jun 23, 2021

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5615/

@akashrn5
Copy link
Copy Markdown
Contributor

akashrn5 commented Jun 23, 2021

LGTM

@asfgit asfgit closed this in d4ddd07 Jun 23, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@akashrn5 akashrn5 akashrn5 left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@Indhumathi27 @CarbonDataQA2 @brijoobopanna @akashrn5