HDDS-10874. Freon tool DNRPCLoadGenerator should not cache client objects.

[jojochuang]

What changes were proposed in this pull request?

HDDS-10874. Freon tool DNRPCLoadGenerator should not cache client objects.

The original XceiverClientSpi API always caches client objects to a specific pipeline. But in this freon tool we wanted to have multiple clients to parallelize throughput.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-10874

How was this patch tested?

Ran on a real cluster. Other than that it leverages the existing tests.

After this change:

ozone freon dne --clients=32 --container-id=1 -t 32 -n 1000000 --ratis --read-only
rpc-payload
mean rate = 38794.07 calls/second

ozone freon dne --clients=1 --container-id=1 -t 32 -n 1000000 --ratis --read-only
count = 1000000
mean rate = 9987.42 calls/second

But number of clients has no impact for Ratis in write mode and GRPC.

[kerneltime]

@jojochuang why not make this the default behavior for reads on client side? Maybe we can change the logic on the client side to have a pool of clients per container, I guess the benefit for the client cache is to avoid the setup cost when there is sequential access to the same container but when there is parallel access to the same container we see a performance hit. Having a pool of clients per container and no upper bound on the number of clients across containers might be right way forward. We might still need a TTL or LRU for the clients.

[adoroszlai]

The original XceiverClientSpi API always caches client objects to a specific pipeline. But in this freon tool we wanted to have multiple clients to parallelize throughput.

Distinct clients can be acquired simply by providing pipeline with unique ID.

[smengcl]

@jojochuang why not make this the default behavior for reads on client side?

@kerneltime I want to take a different angle. -- I wonder where the overhead is coming from in the first place.

In the case of Ratis, XceiverClientRatis itself does not appear to have much synchronization overhead. So I did a little bit of digging. And I think the limiting factor can be this config in each RaftClient:

raft.client.async.outstanding-requests.max

by default each RaftClient allows 100 outstanding requests. As a result only 100 parallel RaftClient#send can be in-flight at one point:

https://github.com/apache/ratis/blob/release-3.0.1/ratis-client/src/main/java/org/apache/ratis/client/impl/OrderedAsync.java#L160-L183

We should try setting raft.client.async.outstanding-requests.max to 100000 and see what happens. cc @jojochuang @szetszwo

[smengcl]

And btw for grpc we also have a similar semaphore config in XceiverClientGrpc that limits parallelization for each XceiverClient instance:

ozone/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/XceiverClientGrpc.java

Lines 126 to 127 in 9b248a0

	this.semaphore =
	new Semaphore(HddsClientUtils.getMaxOutstandingRequests(config));

and it is set to a default of 32:

ozone/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/ratis/conf/RatisClientConfig.java

Lines 51 to 54 in 6839345

	description =
	"Controls the maximum number of outstanding async requests that can"
	+ " be handled by the Standalone as well as Ratis client.")
	private int maxOutstandingRequests = 32;

[jojochuang]

Worth point out that the Echo request is implemented as a blocking call and the number of threads limits number of concurrent requests. And that is consistent across all ContainerProtocolCalls APIs.

For GRPC client, It made no difference increasing hdds.ratis.raft.client.async.outstanding-requests.max. They were all blocked waiting for response from DN.

Hadn't compared RPC Client. But given that 32 thread implies 32 inflight requests, I don't expect tuning that raft.client.async.outstanding-requests.max would make a difference.

[adoroszlai]

Instead of adding a new method to the XceiverClientFactory interface, how about creating a new, non-caching implementation as parent class of XceiverClientManager?

[adoroszlai]

see ad95eb8