Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

wrong default max_query_time #986

Closed
githubmanticore opened this issue Jan 11, 2023 · 3 comments
Closed

wrong default max_query_time #986

githubmanticore opened this issue Jan 11, 2023 · 3 comments
Labels
bug

Comments

@githubmanticore
Copy link
Contributor

It seems that there's an implicit max_query_time equaling to 6 seconds effective when you send a query through a distributed table:

# cat mqt.conf 
searchd { 
    listen = 127.0.0.1:9316 
    listen = 127.0.0.1:9315:mysql 
    log = searchd.log 
    pid_file = searchd.pid 
    data_dir = data 
    agent_connect_timeout = 10000 
} 
 
# searchd -c mqt.conf 
Manticore 5.0.3 d8c16a090@221213 dev 
 
# rm /tmp/sql; echo "drop table if exists t; create table t(f text);" >> /tmp/sql; for n in `seq 1 1000000`; do echo "insert into t values(0,'a b c d e f g h i j k l m n o p q r s t u v w x y z');" >> /tmp/sql; done; mysql -P9315 -h0 < /tmp/sql; 
 
# mysql -P9315 -h0 -e "drop table if exists d; create table d type='distributed' agent='127.0.0.1:9316:t'" 
 
# time mysql -P9315 -h0 -e "select * from d where match('a|b|c|d|e|f|g|h|i|j|k|l|m|n|o|p|q|r|s|t|u|v|w|x|y|z|a|b|c|d|e|f|g|h|i|j|k|l|m|n|o|p|q|r|s|t|u|v|w|x|y|z|a|b|c|d|e|f|g|h|i|j|k|l|m|n|o|p|q|r|s|t|u|v|w|x|y|z|a|b|c|d|e|f|g|h|i|j|k|l|m|n|o|p|q|r|s|t|u|v|w|x|y|z|a|b|c|d|e|f|g|h|i|j|k|l|m|n|o|p|q|r|s|t|u|v|w|x|y|z|a|b|c|d|e|f|g|h|i|j|k|l|m|n|o|p|q|r|s|t|u|v|w|x|y|z'); show warnings;" 
+---------------------+-----------------------------------------------------+ 
| id                  | f                                                   | 
+---------------------+-----------------------------------------------------+ 
| 1515126714626310587 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515126714626310585 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515126714626310586 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515126714626310583 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515126714626310584 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515126714626310582 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515126714626310580 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515126714626310581 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515126714626310578 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515126714626310579 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515126714626310577 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515126714626310575 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515126714626310576 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515126714626310574 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515126714626310572 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515126714626310573 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515126714626310570 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515126714626310571 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515126714626310569 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515126714626310567 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
+---------------------+-----------------------------------------------------+ 
+---------+------+--------------------------------------------------------------------------------------------------+ 
| Level   | Code | Message                                                                                          | 
+---------+------+--------------------------------------------------------------------------------------------------+ 
| warning | 1000 | index d: agent 127.0.0.1:9316: remote query warning: index t: query time exceeded max_query_time | 
+---------+------+--------------------------------------------------------------------------------------------------+ 
mysql -P9315 -h0 -e   0.01s user 0.01s system 0% cpu 6.045 total

Documentation reads:

Default value is 0 which means "do not limit"
@githubmanticore
Copy link
Contributor Author

➤ Stan commented:

this was done at #533 with a following comment

distributed index option agent_query_timeout should be set as query max_query_time option for requests to all agents in case user has not set max_query_time option for query.

That unload agent from useless work when master breaks request on timeout and agent still searching.

@githubmanticore
Copy link
Contributor Author

➤ Stan commented:

I'm going to rollback commit that fixed #533

However I unable to reproduce the case as I always get correct query timeout error. Here are records from query log

/* Mon Dec 19 16:34:26.897 2022 conn 3 real 3.095 wall 3.095 found 0 */ SELECT * FROM d WHERE MATCH('a|b|c|d|e|f|g|h|i|j|k|l|m|n|o|p|q|r|s|t|u|v|w|x|y|z|a|b|c|d|e|f|g|h|i|j|k|l|m|n|o|p|q|r|s|t|u|v|w|x|y|z|a|b|c|d|e|f|g|h|i|j|k|l|m|n|o|p|q|r|s|t|u|v|w|x|y|z|a|b|c|d|e|f|g|h|i|j|k|l|m|n|o|p|q|r|s|t|u|v|w|x|y|z|a|b|c|d|e|f|g|h|i|j|k|l|m|n|o|p|q|r|s|t|u|v|w|x|y|z|a|b|c|d|e|f|g|h|i|j|k|l|m|n|o|p|q|r|s|t|u|v|w|x|y|z'); /*error=index d: agent 127.0.0.1:9312: query timed out */ 
/* Mon Dec 19 16:34:26.901 2022 conn 4 real 3.099 wall 3.099 found 117918 */ SELECT * FROM t WHERE MATCH('a|b|c|d|e|f|g|h|i|j|k|l|m|n|o|p|q|r|s|t|u|v|w|x|y|z|a|b|c|d|e|f|g|h|i|j|k|l|m|n|o|p|q|r|s|t|u|v|w|x|y|z|a|b|c|d|e|f|g|h|i|j|k|l|m|n|o|p|q|r|s|t|u|v|w|x|y|z|a|b|c|d|e|f|g|h|i|j|k|l|m|n|o|p|q|r|s|t|u|v|w|x|y|z|a|b|c|d|e|f|g|h|i|j|k|l|m|n|o|p|q|r|s|t|u|v|w|x|y|z|a|b|c|d|e|f|g|h|i|j|k|l|m|n|o|p|q|r|s|t|u|v|w|x|y|z') LIMIT 0,1000 OPTION max_query_time=3000, retry_count=0, retry_delay=0; /*warning=index t: query time exceeded max_query_time */

from that log its clear that con 3 reports to client query timed out error and later agent reports query time exceeded max_query_time for request it got from master.

@githubmanticore
Copy link
Contributor Author

➤ Sergey Nikolaev commented:

I can't reproduce the original issue on perf3:

root@perf3 ~ # ~/searchd -v 
Manticore 5.0.3 d8c16a090@221213 dev (columnar 1.16.1 0ad0e8c@221202) (secondary 1.16.1 0ad0e8c@221202) 

root@perf3 ~ # time mysql -P9315 -h0 -e "select * from d where match('a|b|c|d|e|f|g|h|i|j|k|l|m|n|o|p|q|r|s|t|u|v|w|x|y|z|a|b|c|d|e|f|g|h|i|j|k|l|m|n|o|p|q|r|s|t|u|v|w|x|y|z|a|b|c|d|e|f|g|h|i|j|k|l|m|n|o|p|q|r|s|t|u|v|w|x|y|z|a|b|c|d|e|f|g|h|i|j|k|l|m|n|o|p|q|r|s|t|u|v|w|x|y|z|a|b|c|d|e|f|g|h|i|j|k|l|m|n|o|p|q|r|s|t|u|v|w|x|y|z|a|b|c|d|e|f|g|h|i|j|k|l|m|n|o|p|q|r|s|t|u|v|w|x|y|z'); show warnings;" 
ERROR 1064 (42000) at line 1: index d: agent 127.0.0.1:9316: query timed out 
 
real	0m3.011s 
user	0m0.006s 
sys	0m0.001s

but can reproduce it in macos:

➜  bin time mysql -P9315 -h0 -e "select * from d where match('a|b|c|d|e|f|g|h|i|j|k|l|m|n|o|p|q|r|s|t|u|v|w|x|y|z|a|b|c|d|e|f|g|h|i|j|k|l|m|n|o|p|q|r|s|t|u|v|w|x|y|z|a|b|c|d|e|f|g|h|i|j|k|l|m|n|o|p|q|r|s|t|u|v|w|x|y|z|a|b|c|d|e|f|g|h|i|j|k|l|m|n|o|p|q|r|s|t|u|v|w|x|y|z|a|b|c|d|e|f|g|h|i|j|k|l|m|n|o|p|q|r|s|t|u|v|w|x|y|z|a|b|c|d|e|f|g|h|i|j|k|l|m|n|o|p|q|r|s|t|u|v|w|x|y|z'); show warnings;" 
 
+---------------------+-----------------------------------------------------+ 
| id                  | f                                                   | 
+---------------------+-----------------------------------------------------+ 
| 1515136344999952827 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515136344999952825 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515136344999952826 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515136344999952823 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515136344999952824 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515136344999952822 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515136344999952820 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515136344999952821 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515136344999952818 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515136344999952819 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515136344999952817 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515136344999952815 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515136344999952816 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515136344999952814 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515136344999952812 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515136344999952813 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515136344999952810 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515136344999952811 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515136344999952809 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515136344999952807 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
+---------------------+-----------------------------------------------------+ 
+---------+------+--------------------------------------------------------------------------------------------------+ 
| Level   | Code | Message                                                                                          | 
+---------+------+--------------------------------------------------------------------------------------------------+ 
| warning | 1000 | index d: agent 127.0.0.1:9316: remote query warning: index t: query time exceeded max_query_time | 
+---------+------+--------------------------------------------------------------------------------------------------+ 
mysql -P9315 -h0 -e   0.01s user 0.01s system 0% cpu 6.046 total

The new version works fine:

➜  bin time mysql -P9315 -h0 -e "select * from d where match('a|b|c|d|e|f|g|h|i|j|k|l|m|n|o|p|q|r|s|t|u|v|w|x|y|z|a|b|c|d|e|f|g|h|i|j|k|l|m|n|o|p|q|r|s|t|u|v|w|x|y|z|a|b|c|d|e|f|g|h|i|j|k|l|m|n|o|p|q|r|s|t|u|v|w|x|y|z|a|b|c|d|e|f|g|h|i|j|k|l|m|n|o|p|q|r|s|t|u|v|w|x|y|z|a|b|c|d|e|f|g|h|i|j|k|l|m|n|o|p|q|r|s|t|u|v|w|x|y|z|a|b|c|d|e|f|g|h|i|j|k|l|m|n|o|p|q|r|s|t|u|v|w|x|y|z'); show warnings;" 
 
+---------------------+-----------------------------------------------------+ 
| id                  | f                                                   | 
+---------------------+-----------------------------------------------------+ 
| 1515136386087354811 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515136386087354809 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515136386087354810 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515136386087354807 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515136386087354808 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515136386087354806 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515136386087354804 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515136386087354805 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515136386087354802 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515136386087354803 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515136386087354801 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515136386087354799 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515136386087354800 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515136386087354798 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515136386087354796 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515136386087354797 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515136386087354794 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515136386087354795 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515136386087354793 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515136386087354791 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
+---------------------+-----------------------------------------------------+ 
mysql -P9315 -h0 -e   0.01s user 0.00s system 0% cpu 7.008 total 
➜  bin time mysql -P9315 -h0 -e "select * from d where match('a|b|c|d|e|f|g|h|i|j|k|l|m|n|o|p|q|r|s|t|u|v|w|x|y|z|a|b|c|d|e|f|g|h|i|j|k|l|m|n|o|p|q|r|s|t|u|v|w|x|y|z|a|b|c|d|e|f|g|h|i|j|k|l|m|n|o|p|q|r|s|t|u|v|w|x|y|z|a|b|c|d|e|f|g|h|i|j|k|l|m|n|o|p|q|r|s|t|u|v|w|x|y|z|a|b|c|d|e|f|g|h|i|j|k|l|m|n|o|p|q|r|s|t|u|v|w|x|y|z|a|b|c|d|e|f|g|h|i|j|k|l|m|n|o|p|q|r|s|t|u|v|w|x|y|z'); show warnings;" 
 
+---------------------+-----------------------------------------------------+ 
| id                  | f                                                   | 
+---------------------+-----------------------------------------------------+ 
| 1515136386087354811 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515136386087354809 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515136386087354810 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515136386087354807 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515136386087354808 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515136386087354806 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515136386087354804 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515136386087354805 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515136386087354802 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515136386087354803 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515136386087354801 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515136386087354799 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515136386087354800 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515136386087354798 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515136386087354796 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515136386087354797 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515136386087354794 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515136386087354795 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515136386087354793 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
| 1515136386087354791 | a b c d e f g h i j k l m n o p q r s t u v w x y z | 
+---------------------+-----------------------------------------------------+ 
mysql -P9315 -h0 -e   0.01s user 0.00s system 0% cpu 6.929 total

but sometimes I'm getting connect timed out which looks wrong provided the both tables are in the same instance:

➜  bin time mysql -P9315 -h0 -e "select * from d where match('a|b|c|d|e|f|g|h|i|j|k|l|m|n|o|p|q|r|s|t|u|v|w|x|y|z|a|b|c|d|e|f|g|h|i|j|k|l|m|n|o|p|q|r|s|t|u|v|w|x|y|z|a|b|c|d|e|f|g|h|i|j|k|l|m|n|o|p|q|r|s|t|u|v|w|x|y|z|a|b|c|d|e|f|g|h|i|j|k|l|m|n|o|p|q|r|s|t|u|v|w|x|y|z|a|b|c|d|e|f|g|h|i|j|k|l|m|n|o|p|q|r|s|t|u|v|w|x|y|z|a|b|c|d|e|f|g|h|i|j|k|l|m|n|o|p|q|r|s|t|u|v|w|x|y|z'); show warnings;" 
 
ERROR 1064 (42000) at line 1: index d: agent 127.0.0.1:9316: connect timed out 
mysql -P9315 -h0 -e   0.01s user 0.00s system 0% cpu 10.014 total

but it's another issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@githubmanticore