100% CPU usage and keep posting event 'reset_instances' to go plugin server

[mong0520]

Is there an existing issue for this?

• I have searched the existing issues

Kong version ($ kong version)

Kong 3.5 go-pdk 0.6.0

Current Behavior

There is a self-development plugin written by Go with go-pdk v0.6.0. It is running well and stable more than one year.
Recently we've upgraded kong from 3.4.2 to 3.5.0 and then found this strange issue rarely happens.

My key configuration to enable the Go plugin server:

KONG_PLUGINSERVER_NAMES=my_plugin
KONG_PLUGINSERVER_MY_PLUGIN_QUERY_CMD="/usr/local/bin/my_plugin -dump"

When this issue happens, I noticed that:

• 100% CPU usage (see the figure below)
• Cannot find any plugin crash logs, exception, nil pointer, errors during that period.
• Not reproducible on my local environment but it happens more than one times on production environment.
• Log shows:

2024/01/17 10:27:46 [warn][kong] mp_rpc.lua:340 [myPlugin] no plugin instance 31
(repeat)
// ....after few moments...
2024/01/17 10:28:10 [error][kong] mp_rpc.lua:336 [myPlugin] failed to post plugin_server reset_instances event: failed to 
publish event: queue overflow
(repeat)

After some quick tracing, it seems like this function https://github.com/Kong/kong/blob/3.5.0/kong/runloop/plugin_servers/mp_rpc.lua#L328-L345 is a recursive function and keep posting event reset_instance and call itself again if the error is still no plugin instance and then run out of event queue size.

Also I found some similar issues:
#9301
#8293

Expected Behavior

Not making CPU 100% and restart the plugin server.

Steps To Reproduce

Not reproducible

Anything else?

No response

[ms2008]

Duplicated of #12364