Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixes a bug in the S3 sink where events without handles throw NPE #2814

Merged

Conversation

dlvenable
Copy link
Member

@dlvenable dlvenable commented Jun 2, 2023

Description

Fixes a bug in the s3 sink for null Event handles.

2023-06-02T22:22:12,052 [performance-alb-pipeline-sink-worker-2-thread-3] ERROR org.opensearch.dataprepper.plugins.sink.S3SinkService - Exception while write event into buffer :
java.lang.NullPointerException: Cannot invoke "org.opensearch.dataprepper.model.event.EventHandle.release(boolean)" because "eventHandle" is null
	at org.opensearch.dataprepper.plugins.sink.S3SinkService.releaseEventHandles(S3SinkService.java:139) ~[s3-sink-2.3.0-SNAPSHOT.jar:?]
	at org.opensearch.dataprepper.plugins.sink.S3SinkService.output(S3SinkService.java:120) ~[s3-sink-2.3.0-SNAPSHOT.jar:?]
	at org.opensearch.dataprepper.plugins.sink.S3Sink.doOutput(S3Sink.java:106) ~[s3-sink-2.3.0-SNAPSHOT.jar:?]
	at org.opensearch.dataprepper.model.sink.AbstractSink.lambda$output$0(AbstractSink.java:64) ~[data-prepper-api-2.3.0-SNAPSHOT.jar:?]
	at io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:141) ~[micrometer-core-1.10.5.jar:1.10.5]
	at org.opensearch.dataprepper.model.sink.AbstractSink.output(AbstractSink.java:64) ~[data-prepper-api-2.3.0-SNAPSHOT.jar:?]
	at org.opensearch.dataprepper.pipeline.Pipeline.lambda$publishToSinks$5(Pipeline.java:336) ~[data-prepper-core-2.3.0-SNAPSHOT.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]

Issues Resolved

N/A

Check List

  • New functionality includes testing.
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

…NPEs by skipping any such handles.

Signed-off-by: David Venable <dlv@amazon.com>
@@ -106,7 +106,9 @@ void output(Collection<Record<Event>> records) {
final byte[] encodedBytes = encodedEvent.getBytes();

currentBuffer.writeEvent(encodedBytes);
bufferedEventHandles.add(event.getEventHandle());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we add a unit test to confirm this is fixed? And help prevent future NPEs around this leak into this plugin?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I actually tried this, but I was not able to get it to crash such that the test would actually show any verification.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Null event handle would be the default case when not using end-to-end acknowledgements.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cmanning09 , I found that there was an unnecessary try-catch on NullPointerException. I removed that and was able to add a valid test.

…rupt the thread on exceptions.

Signed-off-by: David Venable <dlv@amazon.com>
@dlvenable dlvenable merged commit 3a70e73 into opensearch-project:main Jun 5, 2023
24 checks passed
@dlvenable dlvenable mentioned this pull request Jun 5, 2023
@dlvenable dlvenable deleted the fix-s3-sink-event-handle branch June 5, 2023 21:45
MaGonzalMayedo pushed a commit to MaGonzalMayedo/data-prepper that referenced this pull request Jun 21, 2023
…ensearch-project#2814)

Fixes a bug in the S3 sink where events without handles are throwing NPEs by skipping any such handles.

Signed-off-by: David Venable <dlv@amazon.com>
Signed-off-by: Marcos_Gonzalez_Mayedo <alemayed@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants