Skip to content

HDDS-9012. Log more information about failed EC block allocation#5062

Merged
adoroszlai merged 1 commit intoapache:masterfrom
adoroszlai:HDDS-9012
Jul 14, 2023
Merged

HDDS-9012. Log more information about failed EC block allocation#5062
adoroszlai merged 1 commit intoapache:masterfrom
adoroszlai:HDDS-9012

Conversation

@adoroszlai
Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Log more information, including the exclude list, if EC block allocation fails due to max. open pipeline limit.

https://issues.apache.org/jira/browse/HDDS-9012

How was this patch tested?

CI:
https://github.com/adoroszlai/hadoop-ozone/actions/runs/5547145701

Example output from unit test:

WARN  pipeline.WritableECContainerProvider (WritableECContainerProvider.java:getContainer(170)) - Unable to allocate a container after trying 5 existing ones; requested size=1, replication=EC{rs-3-2-1024k}, owner=SCM, ExcludeList {datanodes = [], containerIds = [#2381628560543191, #2381628561553557, #2381628561265675, #2381628560940851, #2381628561836869], pipelineIds = []}
java.io.IOException: Pipeline limit (5) reached (5), none closed
	at org.apache.hadoop.hdds.scm.pipeline.WritableECContainerProvider.getContainer(WritableECContainerProvider.java:166)
	at org.apache.hadoop.hdds.scm.pipeline.WritableECContainerProvider.getContainer(WritableECContainerProvider.java:51)
WARN  pipeline.WritableECContainerProvider (WritableECContainerProvider.java:getContainer(105)) - Unable to allocate a container with 0 existing ones; requested size=1, replication=EC{rs-3-2-1024k}, owner=SCM, ExcludeList {datanodes = [], containerIds = [], pipelineIds = []}
java.io.IOException: Cannot create pipelines
	at org.apache.hadoop.hdds.scm.pipeline.TestWritableECContainerProvider$2.createPipeline(TestWritableECContainerProvider.java:315)
	at org.apache.hadoop.hdds.scm.pipeline.MockPipelineManager.createPipeline(MockPipelineManager.java:65)
	at org.apache.hadoop.hdds.scm.pipeline.TestWritableECContainerProvider$2.createPipeline(TestWritableECContainerProvider.java:318)
	at org.apache.hadoop.hdds.scm.pipeline.WritableECContainerProvider.allocateContainer(WritableECContainerProvider.java:196)
	at org.apache.hadoop.hdds.scm.pipeline.WritableECContainerProvider.getContainer(WritableECContainerProvider.java:103)
	at org.apache.hadoop.hdds.scm.pipeline.WritableECContainerProvider.getContainer(WritableECContainerProvider.java:51)

@adoroszlai adoroszlai self-assigned this Jul 13, 2023
@adoroszlai adoroszlai added the EC label Jul 13, 2023
@guohao-rosicky
Copy link
Copy Markdown
Contributor

Thanks @adoroszlai work on this. The change look good and it helps us to localize the problem.

Copy link
Copy Markdown
Contributor

@sadanand48 sadanand48 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @adoroszlai for the change, LGTM

@adoroszlai adoroszlai merged commit 0157da9 into apache:master Jul 14, 2023
@adoroszlai adoroszlai deleted the HDDS-9012 branch July 14, 2023 06:53
@adoroszlai
Copy link
Copy Markdown
Contributor Author

Thanks @guohao-rosicky, @sadanand48 for the review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants