Druid pool CannotGetJdbcConnectionException

[yyqangular1]

生产环境上有10个微服务，运行了一段时间后，有一个微服务的Druid连接池报错如下，其余9个微服务均正常，DB Server是同一台：

### Cause: org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is com.alibaba.druid.pool.GetConnectionTimeoutException: wait millis 3000, active 0, maxActive 2000, creating 0, createErrorCount 3
        at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:30)
        at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:149)
        at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:140)
        at sun.reflect.GeneratedMethodAccessor292.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:426)
        ... 96 common frames omitted
Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is com.alibaba.druid.pool.GetConnectionTimeoutException: wait millis 3000, active 0, maxActive 2000, creating 0, createErrorCount 3
        at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:82)
        at org.mybatis.spring.transaction.SpringManagedTransaction.openConnection(SpringManagedTransaction.java:80)
        at org.mybatis.spring.transaction.SpringManagedTransaction.getConnection(SpringManagedTransaction.java:67)
        at org.apache.ibatis.executor.BaseExecutor.getConnection(BaseExecutor.java:337)
        at com.baomidou.mybatisplus.core.executor.MybatisReuseExecutor.prepareStatement(MybatisReuseExecutor.java:93)
        at com.baomidou.mybatisplus.core.executor.MybatisReuseExecutor.doQuery(MybatisReuseExecutor.java:64)
        at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:325)
        at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:156)
        at com.baomidou.mybatisplus.core.executor.MybatisCachingExecutor.query(MybatisCachingExecutor.java:165)
        at com.baomidou.mybatisplus.core.executor.MybatisCachingExecutor.query(MybatisCachingExecutor.java:92)
        at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:147)
        ... 101 common frames omitted
Caused by: com.alibaba.druid.pool.GetConnectionTimeoutException: wait millis 3000, active 0, maxActive 2000, creating 0, createErrorCount 3
        at com.alibaba.druid.pool.DruidDataSource.getConnectionInternal(DruidDataSource.java:1773)
        at com.alibaba.druid.pool.DruidDataSource.getConnectionDirect(DruidDataSource.java:1427)
        at com.alibaba.druid.filter.FilterChainImpl.dataSource_connect(FilterChainImpl.java:5059)
        at com.alibaba.druid.filter.stat.StatFilter.dataSource_getConnection(StatFilter.java:726)
        at com.alibaba.druid.filter.FilterChainImpl.dataSource_connect(FilterChainImpl.java:5055)
        at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:1405)
        at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:1397)
        at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:100)
        at com.baomidou.dynamic.datasource.ds.ItemDataSource.getConnection(ItemDataSource.java:52)
        at com.baomidou.dynamic.datasource.ds.AbstractRoutingDataSource.getConnection(AbstractRoutingDataSource.java:43)
        at org.springframework.jdbc.datasource.DataSourceUtils.fetchConnection(DataSourceUtils.java:158)
        at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:116)
        at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:79)
        ... 111 common frames omitted
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded

[yyqangular1]

@wenshao 帮忙看下这个异常，只有一个instance发生这个问题，其他冗余的副本并没有出现获取不到链接的错误，重启之后，该实例也恢复正常了。
如果是db或者网络有问题的话，我理解其他实例也会受影响，而且就算网络波动出现瞬间问题，它自身也可以retry的，所以不应该一直报这个错误。直到重启服务后才恢复。

[wenshao]

druid什么版本？

[wenshao]

看起来是建立连接报错了，查下日志信息

[yyqangular1]

@wenshao druid 1.2.8
我能查到的日志信息就是下面的exception，creating是0，但是我通过客户端和其他实例看db server，都是可以正常连接的。

Cause: org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is com.alibaba.druid.pool.GetConnectionTimeoutException: wait millis 3000, active 0, maxActive 2000, creating 0, createErrorCount 3

[kimmking]

Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded

从这句看是内存溢出，导致CPU都用来做GC，卡死了。
这台机器的内存配置太小？

[yyqangular1]

@kimmking 这台服务器配置还是很高的，16core、64G的机器；容器分配的cpu和内存在1core 4G，最大可弹3倍。
而且从grafana的监控来看JVM堆栈的使用没有发生特别大的波峰，GC回收count也是很平稳。
我之前一直认为是DB服务器的网络可能产生过波动导致，或者是connection数不够了，导致无法create conn。

[jialiu123]

请问一下，最终怎么解决的，是出门左拐的方式吗

[yyqangular1]

请问一下，最终怎么解决的，是出门左拐的方式吗

没有找到根本原因，调整了下连接池的参数，减少了active链接数量，重启了服务，目前稳定运行了几个月了。