NIFI-6895: Fix PutKudu processor concurrency issues #3910
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Calls to `trigger()` may be called concurrently from different threads, however the PutKudu processor is storing the `kuduSession` in a class level field. This can result in the logging issue reported in NIFI-6895 and likely other unusual anomolies including performace issues depending on the processor configuration. Additionally the `operationType` was also stored in a class level field and could be set concurrently resulting in the incorrect operation type used. This patch fixes the issue by moving both kuduSession and operationType to be local. Additionaly some minor code cleanup was included. An integration test, ITPutKudu, was added and used to manual verify the logging issue existed and is fixed by this patch. I ran the test using `mvn -Pintegration-tests verify -Dtest=ITPutKudu`
|
Hey @granthenke - thanks for the PR, I'm going to manually fix the checkstyle issue and merge. Does it also solve the memory leak reported in NIFI-6908? |
pvillard31
approved these changes
Dec 3, 2019
|
@pvillard31 I think it does, though I haven't had a chance to validate. |
|
Is there a 1.10.x branch to backport into? This should definitely be included in the next maintenance release. |
|
Sounds good. We usually backport bug fixes to a maintenance branch only when we decide to do a minor release via a thread on the mailing lists. |
|
@pvillard31 also experiencing this issue and the memleak https://issues.apache.org/jira/browse/NIFI-6908. Any chance there will be a minor release shortly to address? Thanks |
|
@DaveCLowe - there is an ongoing discussion on the mailing list to release Apache NiFi 1.11.0 very soon, I don't think there will be a maintenance release for the 1.10 branch. |
patricker
pushed a commit
to patricker/nifi
that referenced
this pull request
Jan 22, 2020
Calls to `trigger()` may be called concurrently from different threads, however the PutKudu processor is storing the `kuduSession` in a class level field. This can result in the logging issue reported in NIFI-6895 and likely other unusual anomolies including performace issues depending on the processor configuration. Additionally the `operationType` was also stored in a class level field and could be set concurrently resulting in the incorrect operation type used. This patch fixes the issue by moving both kuduSession and operationType to be local. Additionaly some minor code cleanup was included. An integration test, ITPutKudu, was added and used to manual verify the logging issue existed and is fixed by this patch. I ran the test using `mvn -Pintegration-tests verify -Dtest=ITPutKudu` Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes apache#3910.
natural
pushed a commit
to natural/nifi
that referenced
this pull request
Feb 1, 2020
Calls to `trigger()` may be called concurrently from different threads, however the PutKudu processor is storing the `kuduSession` in a class level field. This can result in the logging issue reported in NIFI-6895 and likely other unusual anomolies including performace issues depending on the processor configuration. Additionally the `operationType` was also stored in a class level field and could be set concurrently resulting in the incorrect operation type used. This patch fixes the issue by moving both kuduSession and operationType to be local. Additionaly some minor code cleanup was included. An integration test, ITPutKudu, was added and used to manual verify the logging issue existed and is fixed by this patch. I ran the test using `mvn -Pintegration-tests verify -Dtest=ITPutKudu` Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes apache#3910.
ekovacs
pushed a commit
to ekovacs/nifi-apache
that referenced
this pull request
Feb 24, 2020
Calls to `trigger()` may be called concurrently from different threads, however the PutKudu processor is storing the `kuduSession` in a class level field. This can result in the logging issue reported in NIFI-6895 and likely other unusual anomolies including performace issues depending on the processor configuration. Additionally the `operationType` was also stored in a class level field and could be set concurrently resulting in the incorrect operation type used. This patch fixes the issue by moving both kuduSession and operationType to be local. Additionaly some minor code cleanup was included. An integration test, ITPutKudu, was added and used to manual verify the logging issue existed and is fixed by this patch. I ran the test using `mvn -Pintegration-tests verify -Dtest=ITPutKudu` Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes apache#3910.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Calls to
trigger()may be called concurrently from different threads,however the PutKudu processor is storing the
kuduSessionin a class level field. This can result in the logging issue reported in
NIFI-6895 and likely other unusual anomolies including performace
issues depending on the processor configuration.
Additionally the
operationTypewas also stored in a class level fieldand could be set concurrently resulting in the incorrect operation type
used.
This patch fixes the issue by moving both kuduSession and operationType
to be local. Additionaly some minor code cleanup was included.
An integration test, ITPutKudu, was added and used to manual verify the
logging issue existed and is fixed by this patch. I ran the test using
mvn -Pintegration-tests verify -Dtest=ITPutKudu