[BUG] Mysql Connection pool exhausted. Cannot establish new connection before wait_timeout

[ArcherZdip]

uname -a && php -v && php --ri swoole
Linux web3.kaoqin.mjq 3.10.0-1127.el7.x86_64 #1 SMP Tue Mar 31 23:36:51 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
PHP 7.2.32 (cli) (built: Jul 8 2020 07:33:50) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.2.0, Copyright (c) 1998-2018 Zend Technologies
with Zend OPcache v7.2.32, Copyright (c) 1999-2018, by Zend Technologies

swoole

Swoole => enabled
Author => Swoole Team team@swoole.com
Version => 4.5.2
Built => Jul 24 2020 20:37:48
coroutine => enabled
epoll => enabled
eventfd => enabled
signalfd => enabled
cpu_affinity => enabled
spinlock => enabled
rwlock => enabled
sockets => enabled
openssl => OpenSSL 1.0.2k-fips 26 Jan 2017
http2 => enabled
pcre => enabled
zlib => 1.2.7
mutex_timedlock => enabled
pthread_barrier => enabled
futex => enabled
mysqlnd => enabled
async_redis => enabled

Directive => Local Value => Master Value
swoole.enable_coroutine => On => On
swoole.enable_library => On => On
swoole.enable_preemptive_scheduler => Off => Off
swoole.display_errors => On => On
swoole.use_shortname => Off => Off
swoole.unixsock_buffer_size => 8388608 => 8388608

[WARNING] RuntimeException:Connection pool exhausted. Cannot establish new connection before wait_timeout.(0) in /home/www/sites/cicada-emde/releases/20200918055324/vendor/hyperf/pool/s
rc/Pool.php:144
Stack trace:
#0 /home/www/sites/cicada-emde/releases/20200918055324/vendor/hyperf/pool/src/Pool.php(60): Hyperf\Pool\Pool->getConnection()
#1 /home/www/sites/cicada-emde/releases/20200918055324/vendor/hyperf/db-connection/src/ConnectionResolver.php(67): Hyperf\Pool\Pool->get()
#2 /home/www/sites/cicada-emde/releases/20200918055324/vendor/hyperf/db-connection/src/Model/Model.php(37): Hyperf\DbConnection\ConnectionResolver->connection('default')
#3 /home/www/sites/cicada-emde/releases/20200918055324/vendor/hyperf/database/src/Model/Model.php(1425): Hyperf\DbConnection\Model\Model->getConnection()
#4 /home/www/sites/cicada-emde/releases/20200918055324/vendor/hyperf/database/src/Model/Model.php(669): Hyperf\Database\Model\Model->newBaseQueryBuilder()
#5 /home/www/sites/cicada-emde/releases/20200918055324/vendor/hyperf/database/src/Model/Model.php(704): Hyperf\Database\Model\Model->newModelQuery()
#6 /home/www/sites/cicada-emde/releases/20200918055324/vendor/hyperf/database/src/Model/Model.php(659): Hyperf\Database\Model\Model->newQueryWithoutScopes()
#7 /home/www/sites/cicada-emde/releases/20200918055324/vendor/hyperf/database/src/Model/Model.php(649): Hyperf\Database\Model\Model->newQuery()
#8 /home/www/sites/cicada-emde/releases/20200918055324/app/Services/AccountService.php(40): Hyperf\Database\Model\Model::query()
#9 /home/www/sites/cicada-emde/releases/20200918055324/vendor/hyperf/json-rpc/src/CoreMiddleware.php(47): App\Services\AccountService->findOne(Array, Array, false)
#10 /home/www/sites/cicada-emde/releases/20200918055324/vendor/hyperf/http-server/src/CoreMiddleware.php(101): Hyperf\JsonRpc\CoreMiddleware->handleFound(Object(Hyperf\HttpServer\Router
\Dispatched), Object(Hyperf\HttpMessage\Server\Request))
#11 /home/www/sites/cicada-emde/releases/20200918055324/vendor/hyperf/dispatcher/src/AbstractRequestHandler.php(66): Hyperf\HttpServer\CoreMiddleware->process(Object(Hyperf\HttpMessage\
Server\Request), Object(Hyperf\Dispatcher\HttpRequestHandler))
#12 /home/www/sites/cicada-emde/releases/20200918055324/vendor/hyperf/dispatcher/src/HttpRequestHandler.php(27): Hyperf\Dispatcher\AbstractRequestHandler->handleRequest(Object(Hyperf\Ht
tpMessage\Server\Request))
#13 /home/www/sites/cicada-emde/releases/20200918055324/vendor/hyperf/rpc-server/src/RequestDispatcher.php(42): Hyperf\Dispatcher\HttpRequestHandler->handle(Object(Hyperf\HttpMessage\Se
rver\Request))
#14 /home/www/sites/cicada-emde/releases/20200918055324/vendor/hyperf/http-server/src/Server.php(111): Hyperf\RpcServer\RequestDispatcher->dispatch(Object(Hyperf\HttpMessage\Server\Requ
est), Array, Object(Hyperf\JsonRpc\HttpCoreMiddleware))
#15 {main}

Description:

项目运行时间一长，就容易报出这个问题。
并发和用户量并不高。

Steps To Reproduce:

[limingxinleo]

composer info | grep hyperf

[ArcherZdip]

composer info | grep hyperf

hyperf/cache v1.1.22 A cache component for hyperf.
hyperf/command v1.1.21 Command for hyperf
hyperf/config v1.1.22 An independent component that provides configuration container.
hyperf/constants v1.1.22 A constants component for hyperf.
hyperf/contract v1.1.32 The contracts of Hyperf.
hyperf/crontab v1.1.26 A crontab component for Hyperf.
hyperf/database v1.1.22 A flexible database library.
hyperf/db-connection v1.1.22 A hyperf db connection handler for hyperf/database.
hyperf/devtool v1.1.22 A Devtool for Hyperf.
hyperf/di v1.1.22 A DI for Hyperf.
hyperf/dispatcher v1.1.22 A HTTP Server for Hyperf.
hyperf/elasticsearch v1.1.26 Elasticsearch client for hyperf
hyperf/event v1.1.22 an event manager that implements PSR-14.
hyperf/exception-handler v1.1.6 Exception handler for hyperf
hyperf/framework v1.1.22 A coroutine framework that focuses on hyperspeed and flexible, specifically use for build microservices and middlewares.
hyperf/guzzle v1.1.23 Swoole coroutine handler for guzzle
hyperf/http-message v1.1.22 microservice framework base on swoole
hyperf/http-server v1.1.22 A HTTP Server for Hyperf.
hyperf/json-rpc v1.1.30 A JSON RPC component for Hyperf RPC Server or Client.
hyperf/load-balancer v1.1.22 A load balancer library for Hyperf.
hyperf/logger v1.1.22 A logger component for hyperf.
hyperf/memory v1.1.22 An independent component that use to operate and manage memory.
hyperf/model-cache v1.1.22 A model cache component for hyperf.
hyperf/model-listener v1.1.22 A model listener for Hyperf.
hyperf/paginator v1.1.22 A paginator component for hyperf.
hyperf/pool v1.1.23 An independent universal connection pool component.
hyperf/process v1.1.22 A process component for hyperf.
hyperf/redis v1.1.22 A redis component for hyperf.
hyperf/retry v1.1.26 A retry component for hyperf.
hyperf/rpc v1.1.22 A rpc basic library for Hyperf.
hyperf/rpc-client v1.1.23 An abstract rpc server component for Hyperf.
hyperf/rpc-server v1.1.26 An abstract rpc server component for Hyperf.
hyperf/server v1.1.22 A base server library for Hyperf.
hyperf/snowflake v1.1.22 A snowflake library
hyperf/translation v1.1.17 An independent translation component, forked by illuminate/translation.
hyperf/utils v1.1.31 A tools package that could help developer solved the problem quickly.
hyperf/validation v1.1.22 hyperf validation

[limingxinleo]

检查你的代码是否存在某些代码段，协程无法销毁。

比如某个接口里出现死循环

[ArcherZdip]

检查你的代码是否存在某些代码段，协程无法销毁。

比如某个接口里出现死循环

这个是通过RPC调用的， 程序里只有一个查库操作，没有其他逻辑。

[limingxinleo]

发一下 databases.php 配置

[ArcherZdip]

发一下 databases.php 配置

'default' => [
'driver' => env('DB_DRIVER', 'mysql'),
'read' => [
'host' => [
env('DB_READ_HOST', 'localhost')
],
],
'write' => [
'host' => [
env('DB_WRITE_HOST', 'localhost')
],
],
'sticky' => true,
'port' => env('DB_PORT', 3306),
'database' => env('DB_DATABASE', 'hyperf'),
'username' => env('DB_USERNAME', 'root'),
'password' => env('DB_PASSWORD', ''),
'charset' => env('DB_CHARSET', 'utf8'),
'collation' => env('DB_COLLATION', 'utf8_unicode_ci'),
'prefix' => env('DB_PREFIX', ''),
'pool' => [
'min_connections' => 1,
'max_connections' => 10,
'connect_timeout' => 10.0,
'wait_timeout' => 3.0,
'heartbeat' => -1,
'max_idle_time' => (float)env('DB_MAX_IDLE_TIME', 60),
],
'cache' => [
'handler' => Hyperf\ModelCache\Handler\RedisHandler::class,
'cache_key' => 'mc:%s:m:%s:%s:%s',
'prefix' => 'default',
'ttl' => 3600 * 24,
'empty_model_ttl' => 600,
'load_script' => true,
],
'commands' => [
'gen:model' => [
'path' => 'app/Model',
'force_casts' => true,
'inheritance' => 'Model',
],
],
],

是max_connections 太小了吗？

[limingxinleo]

是的，加大点吧

加到 32

[ArcherZdip]

是的，加大点吧

加到 32

好的 我试试。

[hesunfly]

这个配置数的大小怎么设置合适呢，有没有什么依据，我们的产品也有这个报错了，设置到32解决了

是的，加大点吧

加到 32

[limingxinleo]

@hesunfly 我现在就是 30 - 50 ，感觉比较合理，你们先设置 32 就行

[EwingYangs]

@limingxinleo 我们设置了200，也会报这个错，并发大概在10-20左右

[limingxinleo]

@EwingYangs 你们可能触发到某个bug了

[EwingYangs]

@EwingYangs 你们可能触发到某个bug了

看源码，导致这个异常的条件是属性值currentConnections大于等于配置值，$num=0，请问一下有什么思路能定位一下