Issue HTTP Site-to-Site with lots of queued flow-file#656
Closed
ijokarumawak wants to merge 2 commits intoapache:masterfrom
Closed
Issue HTTP Site-to-Site with lots of queued flow-file#656ijokarumawak wants to merge 2 commits intoapache:masterfrom
ijokarumawak wants to merge 2 commits intoapache:masterfrom
Conversation
HTTP Site-to-Site can't handle TRANSACTION_FINISHED_BUT_DESTINATION_FULL scenario as expected. That happens if the remote NiFi's input port destination relationship becomes full during Site-to-Site client sends data. The data which has already sent to the remote NiFi has to be committed successfully, however, the remote NiFi returns 503 as a response of commit HTTP request. Because it does check port availability. The port availability check shouldn't be done with commit request, since the session at source NiFi has already been committed. The remote NiFi should commit its session as well, and return TRANSACTION_FINISHED_BUT_DESTINATION_FULL response. This fix make remote NiFi keep the handshaken properties when it holds transaction to be committed. Then if a transaction already has handshaken properties, then use it, instead of doing handshake process again.
Member
Author
|
Closing this PR to open a new one solely for NIFI-2259, as this one is getting stale and having conflicts, also, no updates about PipedInputStream related issue. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR is not an actual fix for NIFI-2259. I just want to share it with @markap14.
@markap14 I've been trying to reproduce the issue about PipedInputStream.awaitSpace stops working when there're lots of small flow-files to be sent, like 10,000 files, 1kb each. But couldn't reproduce it so far.
So, I would like to investigate the issue differently. While I was trying to reproduce it, I found another issue related to the amount of queued flow-files. That is NIFI-2259.
With this fix, I was able to queue up 60,000 1kb flow files at Site-to-Site client, then successfully transferred it to remote input-port using HTTP Site-to-Site. (I think it can hold more than that, but stopped increasing at that point)
I also tested by adding RateControl after remote input-port, simulate the slow data consumption. It worked, too, without stopping or losing data.
Would you pull this branch and test it again? I am not thinking this will fix your problem, but certainly make HTTP Site-to-Site works better in a case with lots of flow-files.
Also, if you can reproduce the issue:
Thanks!