-
Notifications
You must be signed in to change notification settings - Fork 3.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Functions] Process async results in the same Java runnable thread #10618
Conversation
*Motivation* After introducing the support for async functions, the java function processing semantic is not enforced. For example, if it fails to write a sink, it doesn't fail the java instance or fail the message. Hence it keeps receiving messages but never ack or nack. *Modification* Change the way how aysnc function requests are processed to fix the issues we have seen in Kinesis connector.
handleResult(currentRecord, result); | ||
} else { | ||
// process the asynchronous results | ||
processAsyncResults(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The null
result indicates the current record is placed into the queue for async processing.
The processAsyncResults()
method takes the first element from the queue and checks/proceeds with the result.
- It seems a newly added record triggers the check of whether the oldest added record is done. This behavior is undesirable since records are independent.
- I'm wondering there'll always be one element left in the queue regardless whether it's done or not.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@nlu90 Nice catch! I have addressed your comment in commit 326ae15
srcRecord, t); | ||
stats.incrUserExceptions(t); | ||
srcRecord.fail(); | ||
private void processAsyncResults() throws InterruptedException { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
no need for this empty method
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM except the above minor comment
…10618) * [Functions] Process async results in the same Java runnable thread *Motivation* After introducing the support for async functions, the java function processing semantic is not enforced. For example, if it fails to write a sink, it doesn't fail the java instance or fail the message. Hence it keeps receiving messages but never ack or nack. *Modification* Change the way how aysnc function requests are processed to fix the issues we have seen in Kinesis connector. * Process the async results in the different thread (cherry picked from commit 1cbd5b4)
…pache#10618) * [Functions] Process async results in the same Java runnable thread *Motivation* After introducing the support for async functions, the java function processing semantic is not enforced. For example, if it fails to write a sink, it doesn't fail the java instance or fail the message. Hence it keeps receiving messages but never ack or nack. *Modification* Change the way how aysnc function requests are processed to fix the issues we have seen in Kinesis connector. * Process the async results in the different thread
…pache#10618) * [Functions] Process async results in the same Java runnable thread *Motivation* After introducing the support for async functions, the java function processing semantic is not enforced. For example, if it fails to write a sink, it doesn't fail the java instance or fail the message. Hence it keeps receiving messages but never ack or nack. *Modification* Change the way how aysnc function requests are processed to fix the issues we have seen in Kinesis connector. * Process the async results in the different thread
Motivation
After introducing the support for async functions, the java function processing semantic is not enforced.
For example, if it fails to write a sink, it doesn't fail the java instance or fail the message. Hence it keeps
receiving messages but never ack or nack.
Modification
Change the way how aysnc function requests are processed to fix the issues we have seen in Kinesis connector.