too many requests for /gateway.Gateway, exceeding concurrency limit (500)

[lgy1027]

fabric: 2.4.1
fabric-gateway: 1.0.1

Using fabric-gateway to insert data asynchronously reports too many requests for /gateway.Gateway, exceeding concurrency limit (500), it was normal before today, what is the reason?

at ai.cp.boot.web.exception.ServiceGlobalExceptionInitializer.printGlobalExceptionLog(ServiceGlobalExceptionInitializer.java:163)
- 
---------------------------------------------
the request: http://order-service:8080/v1/records/actions/deduction is failed, throw exception: 
ai.cp.framework.core.exception.ServiceRuntimeException: io.grpc.StatusRuntimeException: UNKNOWN: too many requests for /gateway.Gateway, exceeding concurrency limit (500)
 at ai.cp.framework.core.exception.ServiceRuntimeException.throwOut(ServiceRuntimeException.java:105) ~[fusionhub-core-1.0.0.jar!/:1.0.0]
 at ai.cp.order.service.impl.FabricServiceImpl.asynchronousInsertJobRecord(FabricServiceImpl.java:95) ~[classes!/:1.0.0]
 at ai.cp.order.service.impl.ComputingPowerRecordServiceImpl.deduction(ComputingPowerRecordServiceImpl.java:286) ~[classes!/:1.0.0]
 at ai.cp.order.service.impl.ComputingPowerRecordServiceImpl$$FastClassBySpringCGLIB$$b2e82f59.invoke(<generated>) ~[classes!/:1.0.0]
 at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.2.12.RELEASE.jar!/:5.2.12.RELEASE]
 at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771) ~[spring-aop-5.2.12.RELEASE.jar!/:5.2.12.RELEASE]
 at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.2.12.RELEASE.jar!/:5.2.12.RELEASE]
 at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) ~[spring-aop-5.2.12.RELEASE.jar!/:5.2.12.RELEASE]
 at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:367) ~[spring-tx-5.2.12.RELEASE.jar!/:5.2.12.RELEASE]
 at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:118) ~[spring-tx-5.2.12.RELEASE.jar!/:5.2.12.RELEASE]
 at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.2.

[bestbeforetoday]

This StackOverflow question addresses the same issue:

https://stackoverflow.com/questions/62401964

The peer configuration to modify if you want to change this limit is here:

https://github.com/hyperledger/fabric/blob/445c3a570b7ec94b9ce65a2ab6675633cefa1ada/sampleconfig/core.yaml#L489

It might be worth investigating why there are more than 500 in-flight service invocations, if this is an unexpectedly high number for your use-case. You might also want to confirm that the peer is servicing requests in a timely manner. If a peer is getting overloaded and requests are getting queued up, you might consider load-balancing client requests across multiple Gateway peers.

[lgy1027]

This StackOverflow question addresses the same issue:

https://stackoverflow.com/questions/62401964

The peer configuration to modify if you want to change this limit is here:

https://github.com/hyperledger/fabric/blob/445c3a570b7ec94b9ce65a2ab6675633cefa1ada/sampleconfig/core.yaml#L489

It might be worth investigating why there are more than 500 in-flight service invocations, if this is an unexpectedly high number for your use-case. You might also want to confirm that the peer is servicing requests in a timely manner. If a peer is getting overloaded and requests are getting queued up, you might consider load-balancing client requests across multiple Gateway peers.

The concurrency does not exceed 500. Is it necessary to create a gateway object every time the ledger is requested, and the fabric-gateway can only specify one peer node information. How to do load balancing and specify more than one?

[bestbeforetoday]

Creating or discarding client-side Gateway objects should have no impact. What is important is the gRPC channel (which you pass in as the connection when creating the Gateway instance) and the number of requests you are making using the Gateway. The gRPC channel is the network connection to the Gateway peer.

Bear in mind that the peer concurrency limit applies to all clients communicating with that peer, not any single client application.

Be aware that some requests can take longer to process than others, such as getting transaction commit status, which blocks until a commit status is available for a given transaction.

For load-balancing, I recommend using a load balancer or ingress controller to receive connections at a single organisation Gateway endpoint, and direct those connections to any available organisation Gateway peer. Issue #257 has several links to information on gRPC load balancing. This sample documentation might also be helpful:

https://github.com/hyperledger/fabric-samples/blob/main/full-stack-asset-transfer-guide/docs/ApplicationDev/01-FabricGateway.md#production-deployment-of-fabric-gateway

You really should not be seeing that peer error unless there are more than 500 requests (from all Gateway clients) being processed by the peer concurrently. If that is not the case, maybe there is a bug in the peer, or in gRPC itself. However, this isn't something that has turned up in load testing of Fabric v2.4.

[lgy1027]

Creating or discarding client-side Gateway objects should have no impact. What is important is the gRPC channel (which you pass in as the connection when creating the Gateway instance) and the number of requests you are making using the Gateway. The gRPC channel is the network connection to the Gateway peer.

Bear in mind that the peer concurrency limit applies to all clients communicating with that peer, not any single client application.

Be aware that some requests can take longer to process than others, such as getting transaction commit status, which blocks until a commit status is available for a given transaction.

For load-balancing, I recommend using a load balancer or ingress controller to receive connections at a single organisation Gateway endpoint, and direct those connections to any available organisation Gateway peer. Issue #257 has several links to information on gRPC load balancing. This sample documentation might also be helpful:

https://github.com/hyperledger/fabric-samples/blob/main/full-stack-asset-transfer-guide/docs/ApplicationDev/01-FabricGateway.md#production-deployment-of-fabric-gateway

You really should not be seeing that peer error unless there are more than 500 requests (from all Gateway clients) being processed by the peer concurrently. If that is not the case, maybe there is a bug in the peer, or in gRPC itself. However, this isn't something that has turned up in load testing of Fabric v2.4.

Thanks for the answer, if you use java to operate the fabric, do you need to recreate the contract object, network object, gateway, channel object when the contract calls the submitAsync() method, for example App.java

For the calls to the contract in the createAsset(), transferAssetAsync(), updateNonExistentAsset() methods, if it is necessary to recreate the contract object, network object, gateway, channel object or some of them in the production environment every time?

[bestbeforetoday]

The sample you've linked to creates the Network and Contract objects only once and then uses them for multiple invocations:

https://github.com/hyperledger/fabric-samples/blob/fd2fffbd435699f3e4a4b2a467b307895e266530/asset-transfer-basic/application-gateway-java/src/main/java/App.java#L106-L113

The Gateway, Network and Contract objects can all be used multiple times and concurrently. They are actually very light weight objects that just provide context for calls made using them so it shouldn't really make a significant difference if you do throw them away and recreate them, but it is unnecessary. The gRPC channel is much more expensive to discard and recreate since it represents a network connection to the Gateway peer, and re-establishing network connections is expensive. A single gRPC channel can (and generally should) be shared by multiple Gateway instances.

[lgy1027]

The sample you've linked to creates the Network and Contract objects only once and then uses them for multiple invocations:

https://github.com/hyperledger/fabric-samples/blob/fd2fffbd435699f3e4a4b2a467b307895e266530/asset-transfer-basic/application-gateway-java/src/main/java/App.java#L106-L113

The Gateway, Network and Contract objects can all be used multiple times and concurrently. They are actually very light weight objects that just provide context for calls made using them so it shouldn't really make a significant difference if you do throw them away and recreate them, but it is unnecessary. The gRPC channel is much more expensive to discard and recreate since it represents a network connection to the Gateway peer, and re-establishing network connections is expensive. A single gRPC channel can (and generally should) be shared by multiple Gateway instances.

Hello, I encountered the same problem again today, but I really can't locate the cause of the problem, I hope you can help me, all the transactions I initiated during this period have not exceeded 500, why do I still report this error , I am wondering, is it necessary to recreate the grpc link and gateway object every time I call SubmitAsync to write asynchronously, thanks