Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Track the memory usage of Runtime Info #44047

Closed
wshwsh12 opened this issue May 22, 2023 · 0 comments · Fixed by #44269
Closed

Track the memory usage of Runtime Info #44047

wshwsh12 opened this issue May 22, 2023 · 0 comments · Fixed by #44269

Comments

@wshwsh12
Copy link
Contributor

wshwsh12 commented May 22, 2023

Enhancement

Now tidb will save an ExecDetails for each request, and will not release the memory until the end of the SQL. In generally, the number of requests that need to be sent by a SQL is not very large, so this part of the memory will not be very large. But in some extreme scenarios, when the number of requests reaches 10 million, the memory usage of this part will reach more than GB. This may cause frequent GC, CPU usage and memory usage. Therefore, this part of memory also needs to be tracked in order to intervene in extreme scenarios.

For example,

Reproduce Steps
use test;

create table base(t int);
insert into base values(1);
insert into base select * from base;
insert into base select * from base;
insert into base select * from base;
insert into base select * from base;
insert into base select * from base;
insert into base select * from base;
insert into base select * from base;
insert into base select * from base;
insert into base select * from base;
insert into base select * from base;

create table base2(a int);
insert into base2 select row_number() over() from base;
insert into base2 select a from base2;
insert into base2 select a from base2;
insert into base2 select a from base2;
insert into base2 select a from base2;
insert into base2 select a from base2;
insert into base2 select a from base2;
insert into base2 select a from base2;
insert into base2 select a from base2;
insert into base2 select a from base2;
insert into base2 select a from base2;

create table t(pk int primary key auto_increment, a int, b int, index idx(a));
insert into t(a) select a from base2; 
insert into t(a) select a from base2; 
insert into t(a) select a from base2;
insert into t(a) select a from base2;
insert into t(a) select a from base2;
insert into t(a) select a from base2;
insert into t(a) select a from base2;
insert into t(a) select a from base2;
insert into t(a) select a from base2;
insert into t(a) select a from base2;
insert into t(a) select a from base2;
insert into t(a) select a from base2;

set tidb_wait_split_region_finish=on;
split table t between (0) and (1048576) regions 1000;
split table t between (1048576) and (1048576*2) regions 1000;
split table t between (1048576*2) and (1048576*3) regions 1000;
split table t between (1048576*3) and (1048576*4) regions 1000;
split table t between (1048576*4) and (1048576*5) regions 1000;
split table t between (1048576*5) and (1048576*6) regions 1000;
split table t between (1048576*6) and (1048576*7) regions 1000;
split table t between (1048576*7) and (1048576*8) regions 1000;
split table t between (1048576*8) and (1048576*9) regions 1000;
split table t between (1048576*9) and (1048576*10) regions 1000;
split table t between (1048576*10) and (1048576*11) regions 1000;
split table t between (1048576*11) and (1048576*12) regions 1000;

set tidb_index_lookup_size = 10000;
desc analyze select /*+ use_index(t,idx) */ count(b) from t;

One SQL will send 12 million request and the runtime info will occury 1GB memory.

image
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants