Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

VTTablet errors on high load #1873

Closed
adkhare opened this issue Jul 15, 2016 · 4 comments
Closed

VTTablet errors on high load #1873

adkhare opened this issue Jul 15, 2016 · 4 comments

Comments

@adkhare
Copy link
Contributor

adkhare commented Jul 15, 2016

I am giving high load on single vtgate. 2 shards are loaded with total of ~1.5K qps and I am seeing following errors on tablet:
E0715 20:30:14.116411 17492 throttled.go:61] TxPoolFull: skipped 33 log messages
E0715 20:33:05.479648 17492 tablet_error.go:245] TxPoolFull:tx_pool_full: Transaction pool connection limit exceeded

not sure if they are relevant or might lead to data loss.

@enisoc
Copy link
Member

enisoc commented Jul 15, 2016

The tx_pool_full error means the tablet is limiting the number of concurrent transactions to protect mysqld. You can tune this setting to your needs by changing the -queryserver-config-transaction-cap flag on vttablet. The default (20) is rather conservative. We generally keep it at several hundred, but it depends on the resources you allocate to mysqld, and also on the particular blend of transactions you send (mysql lock contention, etc.).

@adkhare
Copy link
Contributor Author

adkhare commented Jul 16, 2016

Will these errors cause the data loss? Or they will just reduce the concurrency and slow down the writes

@sougou
Copy link
Contributor

sougou commented Jul 16, 2016

tx pool full errors are returned on a call to Begin. As long as the app handles the error correctly, there should be no corruption.
With queryserver-config-txpool-timeout, you can specify how long you want vttablet to wait before returning a tx pool error.
In general, MySQL performance degrades when there are too many concurrent transaction. The low 100s of connections we recommend is for handling temporary spikes. In other words, you should avoid going overboard on how big you want to set your tx pool.

@sougou
Copy link
Contributor

sougou commented Jul 16, 2016

Forgot to provide the customary launch doc link: https://github.com/youtube/vitess/blob/master/doc/LaunchingVitess.md#query-server-parameters

@enisoc enisoc closed this as completed Aug 24, 2016
frouioui pushed a commit to planetscale/vitess that referenced this issue Nov 21, 2023
…itessio#1873)

* cherry pick of 12871

Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>

* Run correct imports

Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>

* Only run on push

Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>

* Actually run config for token

Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>

---------

Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>
Co-authored-by: Dirkjan Bussink <d.bussink@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants