Skip to content

HDDS-9659. Intermittent failure in TestOzoneECClient#testPartialStripeWithPartialChunkRetry#5601

Merged
adoroszlai merged 1 commit intoapache:masterfrom
adoroszlai:HDDS-9659
Nov 15, 2023
Merged

HDDS-9659. Intermittent failure in TestOzoneECClient#testPartialStripeWithPartialChunkRetry#5601
adoroszlai merged 1 commit intoapache:masterfrom
adoroszlai:HDDS-9659

Conversation

@adoroszlai
Copy link
Copy Markdown
Contributor

@adoroszlai adoroszlai commented Nov 14, 2023

What changes were proposed in this pull request?

Fix intermittent failures in testPartialStripeWithPartialChunkRetry:

IndexOutOfBoundsException: Index: -1
	at java.util.Collections$EmptyList.get(Collections.java:4456)
	at java.util.Collections$UnmodifiableList.get(Collections.java:1311)
	at org.apache.hadoop.ozone.client.io.ECBlockOutputStreamEntry.calculateChecksum(ECBlockOutputStreamEntry.java:416)
	at org.apache.hadoop.ozone.client.io.ECKeyOutputStream.commitStripeWrite(ECKeyOutputStream.java:266)
	at org.apache.hadoop.ozone.client.io.ECKeyOutputStream.flushStripeToDatanodes(ECKeyOutputStream.java:585)
	at org.apache.hadoop.ozone.client.io.ECKeyOutputStream.flushStripeFromQueue(ECKeyOutputStream.java:567)

and

NullPointerException
	at org.apache.hadoop.ozone.client.io.ECBlockOutputStreamEntry.calculateChecksum(ECBlockOutputStreamEntry.java:412)
	at org.apache.hadoop.ozone.client.io.ECKeyOutputStream.commitStripeWrite(ECKeyOutputStream.java:266)
	at org.apache.hadoop.ozone.client.io.ECKeyOutputStream.flushStripeToDatanodes(ECKeyOutputStream.java:585)
	at org.apache.hadoop.ozone.client.io.ECKeyOutputStream.flushStripeFromQueue(ECKeyOutputStream.java:567)

and

ConcurrentModificationException
	at java.util.HashMap$HashIterator.remove(HashMap.java:1483)
	at org.apache.hadoop.ozone.client.MockXceiverClientFactory.mockStorageFailure(MockXceiverClientFactory.java:71)
	at org.apache.hadoop.ozone.client.MockXceiverClientFactory.mockStorageFailure(MockXceiverClientFactory.java:57)
	at org.apache.hadoop.ozone.client.MockXceiverClientFactory.setFailedStorages(MockXceiverClientFactory.java:47)
	at org.apache.hadoop.ozone.client.TestOzoneECClient.testPartialStripeWithPartialChunkRetry(TestOzoneECClient.java:1037)
  1. Completely initialize array of ECBlockOutputStream before exposing it via blockOutputStream member variable.
  2. Fix ConcurrentModificationException in MockXceiverClientFactory by changing to concurrent set/map.

https://issues.apache.org/jira/browse/HDDS-9659

How was this patch tested?

Passed 100x:
https://github.com/adoroszlai/ozone/actions/runs/6862777705

and 250x:
https://github.com/adoroszlai/ozone/actions/runs/6863403759

For reference, failed 2/100 without the fix:
https://github.com/adoroszlai/ozone/actions/runs/6873776049/job/18695127972

Copy link
Copy Markdown
Member

@aswinshakil aswinshakil left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the patch @adoroszlai. LGTM

@adoroszlai adoroszlai merged commit 450749a into apache:master Nov 15, 2023
@adoroszlai adoroszlai deleted the HDDS-9659 branch November 15, 2023 08:25
@adoroszlai
Copy link
Copy Markdown
Contributor Author

Thanks @aswinshakil for the review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants