Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

连续插入数据报Information schema is out of date #105

Closed
daleyjh opened this issue Feb 22, 2017 · 4 comments
Closed

连续插入数据报Information schema is out of date #105

daleyjh opened this issue Feb 22, 2017 · 4 comments

Comments

@daleyjh
Copy link

daleyjh commented Feb 22, 2017 •
edited
Loading

我在从Hbase把数据导入到tidb的时候，批量插入一阵子就会报这个错误，请问这个错误是因为我客户端申请的连接太多还是什么导致的，重新启动插入又会好一会儿再报这个错误。

2017/02/22 12:39:04 2pc.go:129: [info] [BIG_TXN] table id:144 size:4202137, keys:60000, puts:60000, dels:0, locks:0, startTS:390001680197877761
2017/02/22 12:39:04 2pc.go:129: [info] [BIG_TXN] table id:144 size:4487166, keys:60000, puts:60000, dels:0, locks:0, startTS:390001680184770563
2017/02/22 12:39:07 2pc.go:129: [info] [BIG_TXN] table id:144 size:4204384, keys:60000, puts:60000, dels:0, locks:0, startTS:390001681062952961
2017/02/22 12:39:08 2pc.go:129: [info] [BIG_TXN] table id:144 size:4191993, keys:60000, puts:60000, dels:0, locks:0, startTS:390001681233346562
2017/02/22 12:39:09 session.go:230: [warning] [0] finished txn:<nil>, [domain:1]Information schema is out of date.
2017/02/22 12:39:09 session.go:466: [warning] [0] session error:
[domain:1]Information schema is out of date.
/home/jenkins/workspace/TIDB_POST_COMMIT_FLOW/go/src/github.com/pingcap/tidb/store/tikv/2pc.go:532: 
/home/jenkins/workspace/TIDB_POST_COMMIT_FLOW/go/src/github.com/pingcap/tidb/store/tikv/2pc.go:504: 
/home/jenkins/workspace/TIDB_POST_COMMIT_FLOW/go/src/github.com/pingcap/tidb/store/tikv/txn.go:152: 
/home/jenkins/workspace/TIDB_POST_COMMIT_FLOW/go/src/github.com/pingcap/tidb/session.go:208: 
/home/jenkins/workspace/TIDB_POST_COMMIT_FLOW/go/src/github.com/pingcap/tidb/session.go:231: 
/home/jenkins/workspace/TIDB_POST_COMMIT_FLOW/go/src/github.com/pingcap/tidb/tidb.go:186: 
{
  "currDBName": "",
  "id": 0,
  "stauts": 2,
  "strictMode": true,
  "user": ""
}
2017/02/22 12:39:09 session.go:466: [warning] [0] session error:
[domain:1]Information schema is out of date.
/home/jenkins/workspace/TIDB_POST_COMMIT_FLOW/go/src/github.com/pingcap/tidb/store/tikv/2pc.go:532: 
/home/jenkins/workspace/TIDB_POST_COMMIT_FLOW/go/src/github.com/pingcap/tidb/store/tikv/2pc.go:504: 
/home/jenkins/workspace/TIDB_POST_COMMIT_FLOW/go/src/github.com/pingcap/tidb/store/tikv/txn.go:152: 
/home/jenkins/workspace/TIDB_POST_COMMIT_FLOW/go/src/github.com/pingcap/tidb/session.go:208: 
/home/jenkins/workspace/TIDB_POST_COMMIT_FLOW/go/src/github.com/pingcap/tidb/session.go:231: 
/home/jenkins/workspace/TIDB_POST_COMMIT_FLOW/go/src/github.com/pingcap/tidb/tidb.go:186: 
{
  "currDBName": "",
  "id": 0,
  "stauts": 2,
  "strictMode": true,
  "user": ""
}
2017/02/22 12:39:09 gc_worker.go:115: [warning] [gc worker] check leader err: [domain:1]Information schema is out of date.
2017/02/22 12:39:09 2pc.go:478: [info] 2PC clean up done, tid: 390001680669736962
2017/02/22 12:39:10 2pc.go:129: [info] [BIG_TXN] table id:144 size:4241279, keys:60000, puts:60000, dels:0, locks:0, startTS:390001681547919362
2017/02/22 12:39:10 2pc.go:129: [info] [BIG_TXN] table id:144 size:4272906, keys:60000, puts:60000, dels:0, locks:0, startTS:390001681665884161
2017/02/22 12:39:11 2pc.go:129: [info] [BIG_TXN] table id:144 size:4199064, keys:60000, puts:60000, dels:0, locks:0, startTS:390001681862492162
2017/02/22 12:39:13 2pc.go:129: [info] [BIG_TXN] table id:144 size:4199854, keys:60000, puts:60000, dels:0, locks:0, startTS:390001682517852162
2017/02/22 12:39:15 2pc.go:129: [info] [BIG_TXN] table id:144 size:4201273, keys:60000, puts:60000, dels:0, locks:0, startTS:390001683015925762
2017/02/22 12:39:15 2pc.go:129: [info] [BIG_TXN] table id:144 size:4204160, keys:60000, puts:60000, dels:0, locks:0, startTS:390001683199426561
2017/02/22 12:39:16 server.go:192: [info] [320] close connection
2017/02/22 12:39:16 server.go:192: [info] [374] close connection
2017/02/22 12:39:16 server.go:192: [info] [317] close connection
2017/02/22 12:39:16 server.go:192: [info] [321] close connection
2017/02/22 12:39:16 server.go:192: [info] [322] close connection
2017/02/22 12:39:16 server.go:192: [info] [322] close connection
2017/02/22 12:39:16 server.go:192: [info] [312] close connection
2017/02/22 12:39:16 server.go:192: [info] [309] close connection
2017/02/22 12:39:16 server.go:192: [info] [310] close connection
2017/02/22 12:39:16 server.go:192: [info] [314] close connection
2017/02/22 12:39:16 server.go:192: [info] [371] close connection
2017/02/22 12:39:16 server.go:192: [info] [369] close connection

客户端错误是这样的：

java.sql.SQLException: An attempt by a client to checkout a Connection has timed out.
	at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:106)
	at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:65)
	at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:527)
	at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)
	at com.fancy.data.mysql.MysqlConnection.getConnection(MysqlConnection.java:67)
	at com.fancy.data.mysql.MysqlClient.getConnection(MysqlClient.java:32)
	at com.fancy.data.main.InsertMysql.run(InsertMysql.java:23)
	at com.fancy.data.backup.mysql.DealOneDayOneTable.flush(DealOneDayOneTable.java:206)
	at com.fancy.data.hbase.QueryHBase.runScan(QueryHBase.java:123)
	at com.fancy.data.backup.mysql.DealOneDayOneTable.run(DealOneDayOneTable.java:259)
	at java.lang.Thread.run(Thread.java:745)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: com.mchange.v2.resourcepool.TimeoutException: A client timed out while waiting to acquire a resource from com.mchange.v2.resourcepool.BasicResourcePool@5a52059c -- timeout at awaitAvailable()
	at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1317)
	at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557)
	at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)

	at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)
	... 11 more
12:39:15,910 ERROR MysqlClient:35 - 获取数据库连接异常，程序退出
java.lang.NullPointerException
	at com.fancy.data.mysql.MysqlClient.getConnection(MysqlClient.java:33)
	at com.fancy.data.main.InsertMysql.run(InsertMysql.java:23)
	at com.fancy.data.backup.mysql.DealOneDayOneTable.flush(DealOneDayOneTable.java:206)
	at com.fancy.data.hbase.QueryHBase.runScan(QueryHBase.java:123)
	at com.fancy.data.backup.mysql.DealOneDayOneTable.run(DealOneDayOneTable.java:259)
	at java.lang.Thread.run(Thread.java:745)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
@c4pt0r
Copy link
Member

c4pt0r commented Feb 22, 2017

Please post the hardware specifications of your test machine, and your tikv configuration file (e.g. tikv.toml).
The log shows that tikv is too busy, often caused by conservative configuration or weak hardware performance.

@daleyjh
Copy link
Author

daleyjh commented Feb 22, 2017 •
edited
Loading

@c4pt0r My test machine is four 8-Core 16G-Memory machine.I'm doing a test. Maybe the hardware is not enough,the tikv-server cpu is up to 80% when normal running, I try to set the mysql timeout time or add new machine to cluster.

@shenli
Copy link
Member

shenli commented Feb 22, 2017

@daleyjh 请问你使用多少线程插入？另外磁盘是机械盘还是SSD？一般出现“Information schema is out of date“是系统比较慢

@ngaut ngaut closed this as completed Apr 29, 2017
@darren darren mentioned this issue Oct 15, 2018
Closed
@zhuboshuai
Copy link

I encouter once today. the machines's performance monitor indicate that the performance is not problem . one pd-server of three is down .when I restart the pd-server , the problem dissappear.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@c4pt0r @ngaut @shenli @zhuboshuai @daleyjh