Skip to content

NIFI-12700: refactored PutKudu to optimize memory handling for AUTO_F…#8322

Closed
emiliosetiadarma wants to merge 2 commits intoapache:mainfrom
emiliosetiadarma:NIFI-12700
Closed

NIFI-12700: refactored PutKudu to optimize memory handling for AUTO_F…#8322
emiliosetiadarma wants to merge 2 commits intoapache:mainfrom
emiliosetiadarma:NIFI-12700

Conversation

@emiliosetiadarma
Copy link
Contributor

@emiliosetiadarma emiliosetiadarma commented Jan 30, 2024

…LUSH_SYNC flush mode (unbatched flush)

  • Introduced PutKuduResult to store results of processing FlowFiles in PutKudu.
  • Introduced two implementations for PutKuduResult, namely AutoFlushSyncPutKuduResult and StandardPutKuduResult. AutoFlushSyncPutKuduResult includes optimizations considering it's an un-batched flush. StandardPutKuduResult encapsulates all the logic that was used to store the results for processing FlowFiles in the existing implementation.
  • Refactored AbstractKuduProcessor to separate out the closeKuduSession and flushKuduSession functionality.
  • Refactored PutKudu to utilize the new PutKuduResult class.

Summary

NIFI-12700

Tracking

Please complete the following tracking steps prior to pull request creation.

Issue Tracking

Pull Request Tracking

  • Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
  • Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000

Pull Request Formatting

  • Pull Request based on current revision of the main branch
  • Pull Request refers to a feature branch with one commit containing changes

Verification

Please indicate the verification steps performed prior to pull request creation.

Build

  • Build completed using mvn clean install -P contrib-check
    • JDK 21

Licensing

  • New dependencies are compatible with the Apache License 2.0 according to the License Policy
  • New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

  • Documentation formatting appears as expected in rendered files

@turcsanyip turcsanyip self-requested a review January 31, 2024 14:34
@turcsanyip
Copy link
Contributor

Will review...

@emiliosetiadarma emiliosetiadarma marked this pull request as ready for review February 1, 2024 18:25
@mattyb149 mattyb149 self-requested a review March 13, 2024 20:13
@mattyb149
Copy link
Contributor

Reviewing...

…involving determination of whether or not there are flowfile failures/rowErrors. Separated out getting rowErrors from OperationResponses into its own function
@mattyb149
Copy link
Contributor

Verified the requested changes were made. +1 LGTM, thanks for the improvement! Merging to main

@mattyb149 mattyb149 closed this in 3719fdd Mar 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants