HDDS-8855. Filter owner when getting ec container

[guohao-rosicky]

What changes were proposed in this pull request?

Filter owner when getting ec container

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-8855

How was this patch tested?

UT: TestWritableECContainerProvider.java

[guohao-rosicky]

I changed the code so that it doesn't close the pipeline. hi @sodonnel PTAL. Thanks.

[guohao-rosicky]

I accidentally closed this pr(#4905), let's continue the discussion with this pr

[sodonnel]

I am still not sure about this change. If the cluster is started, and owner1 starts writing EC data, then it will create pipelines up to the pipeline limit. Then if owner2 comes in and tries to get a pipeline, it will fail to get one and an error will be returned. Or you could end up with a case where ower1 has most of the pipelines and all writes for owner2 go to a very small number of pipelines.

We are enforcing open container limits at the pipeline level with EC, rather than the container level. You can think of a pipeline and container as the same thing with EC, as they are tightly coupled.

It is still not clear to me how you are using "owner". Where does it come from? How many owners can be seen on a cluster? Is it controlled by OM, or can clients pass different owners freely?

[ivandika3]

It is still not clear to me how you are using "owner". Where does it come from? How many owners can be seen on a cluster?

Maybe I can shed some light on this. Currently, SCM container owner is set as the OM HA leader's Node ID (i.e. OzoneManager#getOmNodeId) which is set by the Ozone configuration. Hence, within a cluster with N OM nodes, there might be up to N different container owners. I have raised MR regarding this (#4922), maybe we can discuss there as well.

Is it controlled by OM, or can clients pass different owners freely?

Currently only OM will set the container owner through the allocateBlock call.

[adoroszlai]

@guohao-rosicky @ivandika3

From HDDS-8874 it looks like owner metadata, designed before OM HA, is incorrectly set when using OM HA. Thus, the information currently stored is basically useless. Until that is fixed, this change would introduce further problems as @sodonnel described. This filtering issue should be fixed after or as part of the OM HA owner problem, certainly not sooner.

Therefore I propose closing this PR but keeping HDDS-8855 open.

[guohao-rosicky]

@adoroszlai @ivandika3
As @sodonnel said, we should follow up with a plan to support the ec container to the owner, just as the ratis container does now.