Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[TEMP] [WIP] Transaction manager support #4143

Closed
wants to merge 1 commit into from

Conversation

ajantha-bhat
Copy link
Member

@ajantha-bhat ajantha-bhat commented May 31, 2021

Why is this PR needed?

Based on the discussion and design document from https://issues.apache.org/jira/browse/CARBONDATA-4171

What changes were proposed in this PR?

  • Added a new ACID module.
  • Defined a transaction manager interface (can support db and table) and implemented a file-based transaction manager to write the transaction log file.
  • Defined a segment store interface (can support db and table) and implemented a file-based segment store to write the table status with the transaction id (multiple table status file)
  • Handled the basic transaction conflict scenario for the file-based segment store.
  • Lock support for transaction log file

TODO:

  1. Write wrappers for the segment store interface (segment status updater and reader) and replace them in the current code.
  2. Remove table status lock everywhere and changing segment id logic
  3. Adding segment id as the return value of each transaction operation to fill the affected segments
  4. similar to segment store interface (table status), need to handle for update status file and segment file also
  5. Try to decouple data and metadata operation for each transaction command (may be write new classes), so metadata can be supported for other open formats also.
  6. Test optimistic concurrency by running parallel transactions (update, compation) and cover all the scenarios.
    On top of these, we can provide time travel SQL support and multi-format support. Across table transaction support

Does this PR introduce any user interface change?

  • No

Is any new test case added?

  • No
  • Yes

@CarbonDataQA2
Copy link

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/3706/

@CarbonDataQA2
Copy link

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5450/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants