Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[refactor](load) Refactor the loader to expand the way data is written #187

Merged
merged 2 commits into from
Feb 1, 2024

Conversation

gnehil
Copy link
Contributor

@gnehil gnehil commented Jan 31, 2024

Proposed changes

Issue Number: close #xxx

Problem Summary:

In order to facilitate the expansion of more writing methods in the future, the writing part has been reconstructed. The main reconstruction contents are as follows:

  1. Loader API
    The Loader API defines three behaviors:

    • load: write data to doris
    • commit: commit transaction
    • abort: abort transaction
  2. TransactionHandler
    The TransactionHandler component is used to process all pre-committed transactions after the task or batch ends.
    When all tasks succeed, all pre-submitted tasks will be submitted; when some tasks fail, all pre-submitted tasks will be rolled back.
    The execution of transaction commit and rollback depends on the specific implementation of the commit and abort methods implemented by the loader.

Other changes:

  1. update revision to 1.4.0-SNAPSHOT
  2. add isSuccess method for RespContent to check whether the stream load is sucess internally

Checklist(Required)

  1. Does it affect the original behavior: (Yes/No/I Don't know)
  2. Has unit tests been added: (Yes/No/No Need)
  3. Has document been added or modified: (Yes/No/No Need)
  4. Does it need to update dependencies: (Yes/No)
  5. Are there any changes that cannot be rolled back: (Yes/No)

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

Copy link
Member

@JNSimba JNSimba left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JNSimba JNSimba merged commit d157a44 into apache:master Feb 1, 2024
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants