Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

生产环境运行一段时间后连接池发生错误 get connection timeout retry : 1 #4326

Closed
antporter opened this issue Jun 13, 2021 · 6 comments

Comments

@antporter
Copy link

antporter commented Jun 13, 2021

1.2.6版本
生产环境运行一段时间后连接池发生错误 get connection timeout retry : 1
接着创建连接发生 IO Error: Connection reset
最后 failContinuous is true

2021-06-08 11:21:28.710 ERROR 57448 --- [ task23] c.alibaba.druid.filter.stat.StatFilter : slow sql 25902 millis. select a from table
2021-06-08 11:21:21.927 ERROR 57448 --- [ task10] c.alibaba.druid.filter.stat.StatFilter : slow sql 22402 millis. select a from table
2021-06-08 11:23:55.604 ERROR 57448 --- [ task10] c.alibaba.druid.filter.stat.StatFilter : slow sql 8216 millis. select a from table
2021-06-08 11:23:55.612 ERROR 57448 --- [io-8081-exec-61] c.alibaba.druid.filter.stat.StatFilter : slow sql 8181 millis. select a from table
2021-06-08 11:23:55.615 WARN 57448 --- [io-8081-exec-43] com.alibaba.druid.pool.DruidDataSource : get connection timeout retry : 1
2021-06-08 11:23:55.617 WARN 57448 --- [io-8081-exec-37] com.alibaba.druid.pool.DruidDataSource : get connection timeout retry : 1
2021-06-08 11:23:55.617 WARN 57448 --- [io-8081-exec-59] com.alibaba.druid.pool.DruidDataSource : get connection timeout retry : 1
2021-06-08 11:23:55.623 WARN 57448 --- [io-8081-exec-42] com.alibaba.druid.pool.DruidDataSource : get connection timeout retry : 1
2021-06-08 11:23:55.618 WARN 57448 --- [io-8081-exec-58] com.alibaba.druid.pool.DruidDataSource : get connection timeout retry : 1
2021-06-08 11:23:55.623 WARN 57448 --- [io-8081-exec-35] com.alibaba.druid.pool.DruidDataSource : get connection timeout retry : 1
2021-06-08 11:23:55.636 WARN 57448 --- [io-8081-exec-50] com.alibaba.druid.pool.DruidDataSource : get connection timeout retry : 1
2021-06-08 11:23:55.636 WARN 57448 --- [io-8081-exec-48] com.alibaba.druid.pool.DruidDataSource : get connection timeout retry : 1
2021-06-08 11:23:55.636 WARN 57448 --- [io-8081-exec-49] com.alibaba.druid.pool.DruidDataSource : get connection timeout retry : 1
2021-06-08 11:23:55.637 WARN 57448 --- [io-8081-exec-40] com.alibaba.druid.pool.DruidDataSource : get connection timeout retry : 1
2021-06-08 11:24:03.983 ERROR 57448 --- [io-8081-exec-35] c.alibaba.druid.filter.stat.StatFilter : slow sql 8327 millis. select a from table
2021-06-08 11:24:04.006 ERROR 57448 --- [eate-1384646526] com.alibaba.druid.pool.DruidDataSource : create connection SQLException, url: jdbc:oracle:thin:@x.x.x.x:1521:orcl, errorCode 17002, state 08006

java.sql.SQLRecoverableException: IO Error: Connection reset
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:533) ~[ojdbc6-11.2.0.3.jar:11.2.0.3.0]
at oracle.jdbc.driver.PhysicalConnection.(PhysicalConnection.java:557) ~[ojdbc6-11.2.0.3.jar:11.2.0.3.0]
at oracle.jdbc.driver.T4CConnection.(T4CConnection.java:233) ~[ojdbc6-11.2.0.3.jar:11.2.0.3.0]
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:29) ~[ojdbc6-11.2.0.3.jar:11.2.0.3.0]
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:556) ~[ojdbc6-11.2.0.3.jar:11.2.0.3.0]
at com.alibaba.druid.filter.FilterChainImpl.connection_connect(FilterChainImpl.java:156) ~[druid-1.2.6.jar:1.2.6]
at com.alibaba.druid.filter.stat.StatFilter.connection_connect(StatFilter.java:227) ~[druid-1.2.6.jar:1.2.6]
at com.alibaba.druid.filter.FilterChainImpl.connection_connect(FilterChainImpl.java:150) ~[druid-1.2.6.jar:1.2.6]
at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1659) ~[druid-1.2.6.jar:1.2.6]
at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.6.jar:1.2.6]
at com.alibaba.druid.pool.DruidDataSource$CreateConnectionThread.run(DruidDataSource.java:2801) ~[druid-1.2.6.jar:1.2.6]
Caused by: java.net.SocketException: Connection reset
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113) ~[na:1.8.0_65]
at java.net.SocketOutputStream.write(SocketOutputStream.java:153) ~[na:1.8.0_65]
at oracle.net.ns.DataPacket.send(DataPacket.java:248) ~[ojdbc6-11.2.0.3.jar:11.2.0.3.0]
at oracle.net.ns.NetOutputStream.flush(NetOutputStream.java:227) ~[ojdbc6-11.2.0.3.jar:11.2.0.3.0]
at oracle.net.ano.AnoComm.b(Unknown Source) ~[ojdbc6-11.2.0.3.jar:11.2.0.3.0]
at oracle.net.ano.Ano.negotiation(Unknown Source) ~[ojdbc6-11.2.0.3.jar:11.2.0.3.0]
at oracle.net.ns.NSProtocol.connect(NSProtocol.java:510) ~[ojdbc6-11.2.0.3.jar:11.2.0.3.0]
at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1625) ~[ojdbc6-11.2.0.3.jar:11.2.0.3.0]
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:365) ~[ojdbc6-11.2.0.3.jar:11.2.0.3.0]
... 10 common frames omitted

java.sql.SQLTimeoutException: ORA-01013: user requested cancel of current operation
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:457)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:400)

2021-06-08 11:25:16.825 INFO 57448 --- [eate-1384646526] c.a.druid.pool.DruidAbstractDataSource : {dataSource-1} failContinuous is true

java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3332)
at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)
at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:514)

这个什么原因造成的呢?

@yangpeng366
Copy link

查询都8-22s了,网络或者数据库状态不太对?
c.alibaba.druid.filter.stat.StatFilter : slow sql 22402 millis. select a from table

@antporter
Copy link
Author

查询都8-22s了,网络或者数据库状态不太对?
c.alibaba.druid.filter.stat.StatFilter : slow sql 22402 millis. select a from table

这个时间点并发确实很大,但是集群的其他节点没有报错,不确定是数据库导致的还是网络导致的。这个节点最后进程还在,就是最后堆溢出了,导致整个应用挂掉。

@antporter
Copy link
Author

antporter commented Jun 15, 2021

已定位到原因。并发获取连接的个数超过”最大并发连接数“导致。但此时连接数远远未到达最大连接数。
select count(*) from v$session where status='ACTIVE' --最大并发连接数

@fatfatcat
Copy link

已定位到原因。并发获取连接的个数超过”最大并发连接数“导致。但此时连接数远远未到达最大连接数。
select count(*) from v$session where status='ACTIVE' --最大并发连接数

我MySQL也遇到了这个问题,请问下你最后是怎么解决的呢?

@antporter
Copy link
Author

antporter commented Jun 22, 2021

已定位到原因。并发获取连接的个数超过”最大并发连接数“导致。但此时连接数远远未到达最大连接数。
select count(*) from v$session where status='ACTIVE' --最大并发连接数

我MySQL也遇到了这个问题,请问下你最后是怎么解决的呢?

1、延长获取连接超时时间。
spring.datasource.druid.max-wait=180000
但是此方法治标不治本。
2、既然是并发获取连接导致,那就不让其产生并发。让连接池一直保持比较大的连接数。
保持连接池最小连接数,避免突然大量并发创建连接,来最终解决问题。
spring.datasource.druid.min-idle=20

@antporter antporter changed the title 生成环境运行一段时间后连接池发生错误 get connection timeout retry : 1 生产环境运行一段时间后连接池发生错误 get connection timeout retry : 1 Nov 14, 2021
@Jackweijin
Copy link

如果程序在运行时内存达到了最大限制,也会导致无法获取连接

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants