adaptivesvc/limiter_mapper.go may cause dead lock

[li-si]

What happened:
Someday I found there are 1165 goroutines blocked by m.rwMutex.RLock() at filter/adaptivesvc/limiter_mapper.go:78. Stack is blow:

1165 @ 0x438696 .....
#       0x4653e4        sync.runtime_SemacquireMutex+0x24                                                               /root/.gvm/gos/go1.17.3/src/runtime/sema.go:71
#       0x1b020da       sync.(*RWMutex).RLock+0xda                                                                      /root/.gvm/gos/go1.17.3/src/sync/rwmutex.go:63
#       0x1b020a7       [dubbo.apache.org/dubbo-go/v3/filter/adaptivesvc.(*limiterMapper).getMethodLimiter+0xa7](http://dubbo.apache.org/dubbo-go/v3/filter/adaptivesvc.(*limiterMapper).getMethodLimiter+0xa7)          /go/pkg/mod/[dubbo.apache.org/dubbo-go/v3@v3.0.0/filter/adaptivesvc/limiter_mapper.go:78](http://dubbo.apache.org/dubbo-go/v3@v3.0.0/filter/adaptivesvc/limiter_mapper.go:78)
#       0x1b0156c       [dubbo.apache.org/dubbo-go/v3/filter/adaptivesvc.(*adaptiveServiceProviderFilter).Invoke+0x8c](http://dubbo.apache.org/dubbo-go/v3/filter/adaptivesvc.(*adaptiveServiceProviderFilter).Invoke+0x8c)    /go/pkg/mod/[dubbo.apache.org/dubbo-go/v3@v3.0.0/filter/adaptivesvc/filter.go:66](http://dubbo.apache.org/dubbo-go/v3@v3.0.0/filter/adaptivesvc/filter.go:66)
#       0x1234f74       [dubbo.apache.org/dubbo-go/v3/protocol/protocolwrapper.(*FilterInvoker).Invoke+0x54](http://dubbo.apache.org/dubbo-go/v3/protocol/protocolwrapper.(*FilterInvoker).Invoke+0x54)              /go/pkg/mod/[dubbo.apache.org/dubbo-go/v3@v3.0.0/protocol/protocolwrapper/protocol_filter_wrapper.go:128](http://dubbo.apache.org/dubbo-go/v3@v3.0.0/protocol/protocolwrapper/protocol_filter_wrapper.go:128)
#       0x1b09e38       [dubbo.apache.org/dubbo-go/v3/filter/gshutdown.(*Filter).Invoke+0x118](http://dubbo.apache.org/dubbo-go/v3/filter/gshutdown.(*Filter).Invoke+0x118)                            /go/pkg/mod/[dubbo.apache.org/dubbo-go/v3@v3.0.0/filter/gshutdown/filter.go:58](http://dubbo.apache.org/dubbo-go/v3@v3.0.0/filter/gshutdown/filter.go:58)
#       0x1234f74       [dubbo.apache.org/dubbo-go/v3/protocol/protocolwrapper.(*FilterInvoker).Invoke+0x54](http://dubbo.apache.org/dubbo-go/v3/protocol/protocolwrapper.(*FilterInvoker).Invoke+0x54)              /go/pkg/mod/[dubbo.apache.org/dubbo-go/v3@v3.0.0/protocol/protocolwrapper/protocol_filter_wrapper.go:128](http://dubbo.apache.org/dubbo-go/v3@v3.0.0/protocol/protocolwrapper/protocol_filter_wrapper.go:128)
#       0x1b04acd       [dubbo.apache.org/dubbo-go/v3/filter/exec_limit.(*executeLimitFilter).Invoke+0x50d](http://dubbo.apache.org/dubbo-go/v3/filter/exec_limit.(*executeLimitFilter).Invoke+0x50d)               /go/pkg/mod/[dubbo.apache.org/dubbo-go/v3@v3.0.0/filter/exec_limit/filter.go:118](http://dubbo.apache.org/dubbo-go/v3@v3.0.0/filter/exec_limit/filter.go:118)
#       0x1234f74       [dubbo.apache.org/dubbo-go/v3/protocol/protocolwrapper.(*FilterInvoker).Invoke+0x54](http://dubbo.apache.org/dubbo-go/v3/protocol/protocolwrapper.(*FilterInvoker).Invoke+0x54)              /go/pkg/mod/[dubbo.apache.org/dubbo-go/v3@v3.0.0/protocol/protocolwrapper/protocol_filter_wrapper.go:128](http://dubbo.apache.org/dubbo-go/v3@v3.0.0/protocol/protocolwrapper/protocol_filter_wrapper.go:128)
#       0x1b091a8       [dubbo.apache.org/dubbo-go/v3/filter/generic.(*genericServiceFilter).Invoke+0x588](http://dubbo.apache.org/dubbo-go/v3/filter/generic.(*genericServiceFilter).Invoke+0x588)                /go/pkg/mod/[dubbo.apache.org/dubbo-go/v3@v3.0.0/filter/generic/service_filter.go:63](http://dubbo.apache.org/dubbo-go/v3@v3.0.0/filter/generic/service_filter.go:63)
#       0x1234f74       [dubbo.apache.org/dubbo-go/v3/protocol/protocolwrapper.(*FilterInvoker).Invoke+0x54](http://dubbo.apache.org/dubbo-go/v3/protocol/protocolwrapper.(*FilterInvoker).Invoke+0x54)              /go/pkg/mod/[dubbo.apache.org/dubbo-go/v3@v3.0.0/protocol/protocolwrapper/protocol_filter_wrapper.go:128](http://dubbo.apache.org/dubbo-go/v3@v3.0.0/protocol/protocolwrapper/protocol_filter_wrapper.go:128)
#       0x1b33d16       [dubbo.apache.org/dubbo-go/v3/filter/tps.(*tpsLimitFilter).Invoke+0x276](http://dubbo.apache.org/dubbo-go/v3/filter/tps.(*tpsLimitFilter).Invoke+0x276)                          /go/pkg/mod/[dubbo.apache.org/dubbo-go/v3@v3.0.0/filter/tps/filter.go:82](http://dubbo.apache.org/dubbo-go/v3@v3.0.0/filter/tps/filter.go:82)
#       0x1234f74       [dubbo.apache.org/dubbo-go/v3/protocol/protocolwrapper.(*FilterInvoker).Invoke+0x54](http://dubbo.apache.org/dubbo-go/v3/protocol/protocolwrapper.(*FilterInvoker).Invoke+0x54)              /go/pkg/mod/[dubbo.apache.org/dubbo-go/v3@v3.0.0/protocol/protocolwrapper/protocol_filter_wrapper.go:128](http://dubbo.apache.org/dubbo-go/v3@v3.0.0/protocol/protocolwrapper/protocol_filter_wrapper.go:128)
#       0x1afe30d       [dubbo.apache.org/dubbo-go/v3/filter/accesslog.(*Filter).Invoke+0xcd](http://dubbo.apache.org/dubbo-go/v3/filter/accesslog.(*Filter).Invoke+0xcd)                             /go/pkg/mod/[dubbo.apache.org/dubbo-go/v3@v3.0.0/filter/accesslog/filter.go:109](http://dubbo.apache.org/dubbo-go/v3@v3.0.0/filter/accesslog/filter.go:109)
#       0x1234f74       [dubbo.apache.org/dubbo-go/v3/protocol/protocolwrapper.(*FilterInvoker).Invoke+0x54](http://dubbo.apache.org/dubbo-go/v3/protocol/protocolwrapper.(*FilterInvoker).Invoke+0x54)              /go/pkg/mod/[dubbo.apache.org/dubbo-go/v3@v3.0.0/protocol/protocolwrapper/protocol_filter_wrapper.go:128](http://dubbo.apache.org/dubbo-go/v3@v3.0.0/protocol/protocolwrapper/protocol_filter_wrapper.go:128)
#       0x1b32f09       [dubbo.apache.org/dubbo-go/v3/filter/token.(*tokenFilter).Invoke+0x229](http://dubbo.apache.org/dubbo-go/v3/filter/token.(*tokenFilter).Invoke+0x229)                           /go/pkg/mod/[dubbo.apache.org/dubbo-go/v3@v3.0.0/filter/token/filter.go:71](http://dubbo.apache.org/dubbo-go/v3@v3.0.0/filter/token/filter.go:71)
#       0x1234f74       [dubbo.apache.org/dubbo-go/v3/protocol/protocolwrapper.(*FilterInvoker).Invoke+0x54](http://dubbo.apache.org/dubbo-go/v3/protocol/protocolwrapper.(*FilterInvoker).Invoke+0x54)              /go/pkg/mod/[dubbo.apache.org/dubbo-go/v3@v3.0.0/protocol/protocolwrapper/protocol_filter_wrapper.go:128](http://dubbo.apache.org/dubbo-go/v3@v3.0.0/protocol/protocolwrapper/protocol_filter_wrapper.go:128)
#       0x1b14ac2       [dubbo.apache.org/dubbo-go/v3/filter/metrics.(*Filter).Invoke+0xa2](http://dubbo.apache.org/dubbo-go/v3/filter/metrics.(*Filter).Invoke+0xa2)                               /go/pkg/mod/[dubbo.apache.org/dubbo-go/v3@v3.0.0/filter/metrics/filter.go:49](http://dubbo.apache.org/dubbo-go/v3@v3.0.0/filter/metrics/filter.go:49)
#       0x1234f74       [dubbo.apache.org/dubbo-go/v3/protocol/protocolwrapper.(*FilterInvoker).Invoke+0x54](http://dubbo.apache.org/dubbo-go/v3/protocol/protocolwrapper.(*FilterInvoker).Invoke+0x54)              /go/pkg/mod/[dubbo.apache.org/dubbo-go/v3@v3.0.0/protocol/protocolwrapper/protocol_filter_wrapper.go:128](http://dubbo.apache.org/dubbo-go/v3@v3.0.0/protocol/protocolwrapper/protocol_filter_wrapper.go:128)
#       0x1b0416c       [dubbo.apache.org/dubbo-go/v3/filter/echo.(*echoFilter).Invoke+0x14c](http://dubbo.apache.org/dubbo-go/v3/filter/echo.(*echoFilter).Invoke+0x14c)                             /go/pkg/mod/[dubbo.apache.org/dubbo-go/v3@v3.0.0/filter/echo/filter.go:65](http://dubbo.apache.org/dubbo-go/v3@v3.0.0/filter/echo/filter.go:65)
#       0x1234f74       [dubbo.apache.org/dubbo-go/v3/protocol/protocolwrapper.(*FilterInvoker).Invoke+0x54](http://dubbo.apache.org/dubbo-go/v3/protocol/protocolwrapper.(*FilterInvoker).Invoke+0x54)              /go/pkg/mod/[dubbo.apache.org/dubbo-go/v3@v3.0.0/protocol/protocolwrapper/protocol_filter_wrapper.go:128](http://dubbo.apache.org/dubbo-go/v3@v3.0.0/protocol/protocolwrapper/protocol_filter_wrapper.go:128)
#       0x1378a8a       [gitlab.com/******_triple.pb.go:360
#       0x1189b4d       [github.com/dubbogo/grpc-go.(*Server).processUnaryRPC+0xe2d](http://github.com/dubbogo/grpc-go.(*Server).processUnaryRPC+0xe2d)                                      /go/pkg/mod/[github.com/dubbogo/grpc-go@v1.42.6-triple/server.go:1285](http://github.com/dubbogo/grpc-go@v1.42.6-triple/server.go:1285)
#       0x118dcc4       [github.com/dubbogo/grpc-go.(*Server).handleStream+0xac4](http://github.com/dubbogo/grpc-go.(*Server).handleStream+0xac4)                                         /go/pkg/mod/[github.com/dubbogo/grpc-go@v1.42.6-triple/server.go:1657](http://github.com/dubbogo/grpc-go@v1.42.6-triple/server.go:1657)
#       0x1187537       [github.com/dubbogo/grpc-go.(*Server).serveStreams.func1.2+0x97](http://github.com/dubbogo/grpc-go.(*Server).serveStreams.func1.2+0x97)                                  /go/pkg/mod/[github.com/dubbogo/grpc-go@v1.42.6-triple/server.go:921](http://github.com/dubbogo/grpc-go@v1.42.6-triple/server.go:921)

When I step into the source code I found the code blow. Func newAndSetMethodLimiter may produce a dead lock if it returns at line 62 or line 68. Is this a bug?

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

[justxuewei]

Thanks for reporting this problem. I wondering are you used it into your production environment? For now, the adaptive service is an experimental feature for the dubbo-go, and we are actively developing next version of it. Are you interested in developing new features of it?

[AlexStocks]

Maybe you can join in the DingDing group 23331795 and have a talk with me. I am the group owner.