New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Test performance for queries with multiple disk chunks #2318

Merged

PavelShilin89 merged 11 commits into master from test-performance-for-queries-with-multiple-disk-chunks

Jun 20, 2024

Contributor

PavelShilin89 commented Jun 18, 2024

Created a test to check performance for wildcard queries with many matches when disk_chunks > 1 and disk_chunks = 1

Pavel_Shilin added 5 commits

June 18, 2024 11:46


          Create test-performance-for-queries-with-multiple-disk-chunks.rec

c6370d8


          Update test-performance-for-queries-with-multiple-disk-chunks.rec

1a08829


          Update test-performance-for-queries-with-multiple-disk-chunks.rec

15a14c8


          Upgrade test to eliminate bugs and change logic

1e754d4


          Created a new task in clt_nightly.yml

7cca450

github-actions bot commented Jun 18, 2024

clt-amd64

👎 CLT tests in test/clt-tests/core/test-performance-for-queries-with-multiple-disk-chunks
✅ OK: 0
❌ Failed: 1
⏳ Duration: 276s
👉 Check Action Results for commit 768a64e
Failed tests:

test/clt-tests/core/test-performance-for-queries-with-multiple-disk-chunks.rec

––– input –––
rm -f /var/log/manticore/searchd.log; searchd --stopwait > /dev/null; searchd; if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore/searchd.log;fi
––– output –––
Manticore %{SEMVER} %{COMMITDATE}#!/(\sdev)?\s/!#(columnar %{SEMVER} %{COMMITDATE}) (secondary %{SEMVER} %{COMMITDATE}) (knn %{SEMVER} %{COMMITDATE})
Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
Copyright (c) 2017-%{YEAR}, Manticore Software LTD (https://manticoresearch.com)
[#!/[0-9]{2}:[0-9]{2}.[0-9]{3}/!#] [#!/[0-9]+/!#] using config file '/etc/manticoresearch/manticore.conf' (%{NUMBER} chars)...
starting daemon version '%{SEMVER} %{COMMITDATE}#!/(\sdev)?\s/!#(columnar %{SEMVER} %{COMMITDATE}) (secondary %{SEMVER} %{COMMITDATE}) (knn %{SEMVER} %{COMMITDATE})' ...
listening on %{IPADDR}:9312 for sphinx and http(s)
listening on %{IPADDR}:9306 for mysql
listening on %{IPADDR}:9308 for sphinx and http(s)
Buddy started!
––– input –––
php -d memory_limit=-1 ./test/clt-tests/comparison-overhead-json-sql/load_us_names.php 10000 10 10000000 1 > /dev/null
––– output –––
––– input –––
mysql -h0 -P9306 -e "select count(*) from name;"
––– output –––
+----------+
| count(*) |
+----------+
| 10000000 |
+----------+
––– input –––
export start=$(date +%s%N)
––– output –––
––– input –––
mysql -h0 -P9306 -e "select * from name where match('*on*');" > /dev/null; echo $?
––– output –––
0
––– input –––
export end=$(date +%s%N)
––– output –––
––– input –––
export multiple_disk_chunks=$(( (end - start) / 1000000 ))
––– output –––
––– input –––
mysql -h0 -P9306 -e "show table name status" | grep "disk_chunks"
––– output –––
- | disk_chunks                 | 7                                                                                                      |
+ | disk_chunks                 | 6                                                                                                      |
––– input –––
mysql -h0 -P9306 -e "OPTIMIZE TABLE name OPTION cutoff=1, sync=1;"
––– output –––
––– input –––
mysql -h0 -P9306 -e "show table name status" | grep "disk_chunks"
––– output –––
| disk_chunks                 | 1                                                                                                      |
––– input –––
export start=$(date +%s%N)
––– output –––
––– input –––
mysql -h0 -P9306 -e "select * from name where match('*on*');" > /dev/null; echo $?
––– output –––
0
––– input –––
export end=$(date +%s%N)
––– output –––
––– input –––
export optimize=$(( (end - start) / 1000000 ))
––– output –––
––– input –––
echo "Duration: ${optimize} ${multiple_disk_chunks} ms"
––– output –––
Duration: %{NUMBER} %{NUMBER} ms
––– input –––
php -r "exit(round(abs($multiple_disk_chunks - $optimize) / ($multiple_disk_chunks + $optimize)/2 * 100, 0) >= 15);" 2>\dev\null && echo "----$?----"
––– output –––
----0----


          Added regex to test-performance-for-queries-with-multiple-disk-chunks…

e924f31

….rec

github-actions bot commented Jun 18, 2024

clt

👎 CLT tests in test/clt-tests/core/ test/clt-tests/expected-errors/
✅ OK: 15
❌ Failed: 1
⏳ Duration: 531s
👉 Check Action Results for commit 768a64e
Failed tests:

test/clt-tests/core/test-performance-for-queries-with-multiple-disk-chunks.rec

––– input –––
rm -f /var/log/manticore/searchd.log; searchd --stopwait > /dev/null; searchd; if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore/searchd.log;fi
––– output –––
Manticore %{SEMVER} %{COMMITDATE}#!/(\sdev)?\s/!#(columnar %{SEMVER} %{COMMITDATE}) (secondary %{SEMVER} %{COMMITDATE}) (knn %{SEMVER} %{COMMITDATE})
Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
Copyright (c) 2017-%{YEAR}, Manticore Software LTD (https://manticoresearch.com)
[#!/[0-9]{2}:[0-9]{2}.[0-9]{3}/!#] [#!/[0-9]+/!#] using config file '/etc/manticoresearch/manticore.conf' (%{NUMBER} chars)...
starting daemon version '%{SEMVER} %{COMMITDATE}#!/(\sdev)?\s/!#(columnar %{SEMVER} %{COMMITDATE}) (secondary %{SEMVER} %{COMMITDATE}) (knn %{SEMVER} %{COMMITDATE})' ...
listening on %{IPADDR}:9312 for sphinx and http(s)
listening on %{IPADDR}:9306 for mysql
listening on %{IPADDR}:9308 for sphinx and http(s)
Buddy started!
––– input –––
php -d memory_limit=-1 ./test/clt-tests/comparison-overhead-json-sql/load_us_names.php 10000 10 10000000 1 > /dev/null
––– output –––
––– input –––
mysql -h0 -P9306 -e "select count(*) from name;"
––– output –––
+----------+
| count(*) |
+----------+
| 10000000 |
+----------+
––– input –––
export start=$(date +%s%N)
––– output –––
––– input –––
mysql -h0 -P9306 -e "select * from name where match('*on*');" > /dev/null; echo $?
––– output –––
0
––– input –––
export end=$(date +%s%N)
––– output –––
––– input –––
export multiple_disk_chunks=$(( (end - start) / 1000000 ))
––– output –––
––– input –––
mysql -h0 -P9306 -e "show table name status" | grep "disk_chunks"
––– output –––
- | disk_chunks                 | 7                                                                                                      |
+ | disk_chunks                 | 6                                                                                                      |
––– input –––
mysql -h0 -P9306 -e "OPTIMIZE TABLE name OPTION cutoff=1, sync=1;"
––– output –––
––– input –––
mysql -h0 -P9306 -e "show table name status" | grep "disk_chunks"
––– output –––
| disk_chunks                 | 1                                                                                                      |
––– input –––
export start=$(date +%s%N)
––– output –––
––– input –––
mysql -h0 -P9306 -e "select * from name where match('*on*');" > /dev/null; echo $?
––– output –––
0
––– input –––
export end=$(date +%s%N)
––– output –––
––– input –––
export optimize=$(( (end - start) / 1000000 ))
––– output –––
––– input –––
echo "Duration: ${optimize} ${multiple_disk_chunks} ms"
––– output –––
Duration: %{NUMBER} %{NUMBER} ms
––– input –––
php -r "exit(round(abs($multiple_disk_chunks - $optimize) / ($multiple_disk_chunks + $optimize)/2 * 100, 0) >= 15);" 2>\dev\null && echo "----$?----"
––– output –––
----0----

github-actions bot commented Jun 18, 2024

clt-amd64

👎 CLT tests in test/clt-tests/core/test-performance-for-queries-with-multiple-disk-chunks
✅ OK: 0
❌ Failed: 1
⏳ Duration: 315s
👉 Check Action Results for commit 7fcbfeb
Failed tests:

test/clt-tests/core/test-performance-for-queries-with-multiple-disk-chunks.rec

––– input –––
rm -f /var/log/manticore/searchd.log; searchd --stopwait > /dev/null; searchd; if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore/searchd.log;fi
––– output –––
Manticore %{SEMVER} %{COMMITDATE}#!/(\sdev)?\s/!#(columnar %{SEMVER} %{COMMITDATE}) (secondary %{SEMVER} %{COMMITDATE}) (knn %{SEMVER} %{COMMITDATE})
Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
Copyright (c) 2017-%{YEAR}, Manticore Software LTD (https://manticoresearch.com)
[#!/[0-9]{2}:[0-9]{2}.[0-9]{3}/!#] [#!/[0-9]+/!#] using config file '/etc/manticoresearch/manticore.conf' (%{NUMBER} chars)...
starting daemon version '%{SEMVER} %{COMMITDATE}#!/(\sdev)?\s/!#(columnar %{SEMVER} %{COMMITDATE}) (secondary %{SEMVER} %{COMMITDATE}) (knn %{SEMVER} %{COMMITDATE})' ...
listening on %{IPADDR}:9312 for sphinx and http(s)
listening on %{IPADDR}:9306 for mysql
listening on %{IPADDR}:9308 for sphinx and http(s)
Buddy started!
––– input –––
php -d memory_limit=-1 ./test/clt-tests/comparison-overhead-json-sql/load_us_names.php 10000 10 10000000 1 > /dev/null
––– output –––
––– input –––
mysql -h0 -P9306 -e "select count(*) from name;"
––– output –––
+----------+
| count(*) |
+----------+
| 10000000 |
+----------+
––– input –––
export start=$(date +%s%N)
––– output –––
––– input –––
mysql -h0 -P9306 -e "select * from name where match('*on*');" > /dev/null; echo $?
––– output –––
0
––– input –––
export end=$(date +%s%N)
––– output –––
––– input –––
export multiple_disk_chunks=$(( (end - start) / 1000000 ))
––– output –––
––– input –––
mysql -h0 -P9306 -e "show table name status" | grep "disk_chunks"
––– output –––
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Syntax(
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
regex parse error:
    {7,6}
    ^
error: repetition operator missing expression
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
)', src/main.rs:210:62
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

github-actions bot commented Jun 18, 2024

clt

👎 CLT tests in test/clt-tests/core/ test/clt-tests/expected-errors/
✅ OK: 15
❌ Failed: 1
⏳ Duration: 564s
👉 Check Action Results for commit 7fcbfeb
Failed tests:

test/clt-tests/core/test-performance-for-queries-with-multiple-disk-chunks.rec

––– input –––
rm -f /var/log/manticore/searchd.log; searchd --stopwait > /dev/null; searchd; if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore/searchd.log;fi
––– output –––
Manticore %{SEMVER} %{COMMITDATE}#!/(\sdev)?\s/!#(columnar %{SEMVER} %{COMMITDATE}) (secondary %{SEMVER} %{COMMITDATE}) (knn %{SEMVER} %{COMMITDATE})
Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
Copyright (c) 2017-%{YEAR}, Manticore Software LTD (https://manticoresearch.com)
[#!/[0-9]{2}:[0-9]{2}.[0-9]{3}/!#] [#!/[0-9]+/!#] using config file '/etc/manticoresearch/manticore.conf' (%{NUMBER} chars)...
starting daemon version '%{SEMVER} %{COMMITDATE}#!/(\sdev)?\s/!#(columnar %{SEMVER} %{COMMITDATE}) (secondary %{SEMVER} %{COMMITDATE}) (knn %{SEMVER} %{COMMITDATE})' ...
listening on %{IPADDR}:9312 for sphinx and http(s)
listening on %{IPADDR}:9306 for mysql
listening on %{IPADDR}:9308 for sphinx and http(s)
Buddy started!
––– input –––
php -d memory_limit=-1 ./test/clt-tests/comparison-overhead-json-sql/load_us_names.php 10000 10 10000000 1 > /dev/null
––– output –––
––– input –––
mysql -h0 -P9306 -e "select count(*) from name;"
––– output –––
+----------+
| count(*) |
+----------+
| 10000000 |
+----------+
––– input –––
export start=$(date +%s%N)
––– output –––
––– input –––
mysql -h0 -P9306 -e "select * from name where match('*on*');" > /dev/null; echo $?
––– output –––
0
––– input –––
export end=$(date +%s%N)
––– output –––
––– input –––
export multiple_disk_chunks=$(( (end - start) / 1000000 ))
––– output –––
––– input –––
mysql -h0 -P9306 -e "show table name status" | grep "disk_chunks"
––– output –––
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Syntax(
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
regex parse error:
    {7,6}
    ^
error: repetition operator missing expression
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
)', src/main.rs:210:62
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


          Update test-performance-for-queries-with-multiple-disk-chunks.rec

bb95370

Update regex to test-performance-for-queries-with-multiple-disk-chunks.rec

github-actions bot commented Jun 19, 2024

clt

👎 CLT tests in test/clt-tests/core/ test/clt-tests/expected-errors/
✅ OK: 15
❌ Failed: 1
⏳ Duration: 520s
👉 Check Action Results for commit 384f68e
Failed tests:

test/clt-tests/core/test-performance-for-queries-with-multiple-disk-chunks.rec

––– input –––
rm -f /var/log/manticore/searchd.log; searchd --stopwait > /dev/null; searchd; if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore/searchd.log;fi
––– output –––
Manticore %{SEMVER} %{COMMITDATE}#!/(\sdev)?\s/!#(columnar %{SEMVER} %{COMMITDATE}) (secondary %{SEMVER} %{COMMITDATE}) (knn %{SEMVER} %{COMMITDATE})
Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
Copyright (c) 2017-%{YEAR}, Manticore Software LTD (https://manticoresearch.com)
[#!/[0-9]{2}:[0-9]{2}.[0-9]{3}/!#] [#!/[0-9]+/!#] using config file '/etc/manticoresearch/manticore.conf' (%{NUMBER} chars)...
starting daemon version '%{SEMVER} %{COMMITDATE}#!/(\sdev)?\s/!#(columnar %{SEMVER} %{COMMITDATE}) (secondary %{SEMVER} %{COMMITDATE}) (knn %{SEMVER} %{COMMITDATE})' ...
listening on %{IPADDR}:9312 for sphinx and http(s)
listening on %{IPADDR}:9306 for mysql
listening on %{IPADDR}:9308 for sphinx and http(s)
Buddy started!
––– input –––
php -d memory_limit=-1 ./test/clt-tests/comparison-overhead-json-sql/load_us_names.php 10000 10 10000000 1 > /dev/null
––– output –––
––– input –––
mysql -h0 -P9306 -e "select count(*) from name;"
––– output –––
+----------+
| count(*) |
+----------+
| 10000000 |
+----------+
––– input –––
export start=$(date +%s%N)
––– output –––
––– input –––
mysql -h0 -P9306 -e "select * from name where match('*on*');" > /dev/null; echo $?
––– output –––
0
––– input –––
export end=$(date +%s%N)
––– output –––
––– input –––
export multiple_disk_chunks=$(( (end - start) / 1000000 ))
––– output –––
––– input –––
mysql -h0 -P9306 -e "show table name status" | grep "disk_chunks"
––– output –––
- | disk_chunks                 | %{NUMBER}                                                                                                   |
+ | disk_chunks                 | 6                                                                                                      |
––– input –––
mysql -h0 -P9306 -e "OPTIMIZE TABLE name OPTION cutoff=1, sync=1;"
––– output –––
––– input –––
mysql -h0 -P9306 -e "show table name status" | grep "disk_chunks"
––– output –––
| disk_chunks                 | 1                                                                                                      |
––– input –––
export start=$(date +%s%N)
––– output –––
––– input –––
mysql -h0 -P9306 -e "select * from name where match('*on*');" > /dev/null; echo $?
––– output –––
0
––– input –––
export end=$(date +%s%N)
––– output –––
––– input –––
export optimize=$(( (end - start) / 1000000 ))
––– output –––
––– input –––
echo "Duration: ${optimize} ${multiple_disk_chunks} ms"
––– output –––
Duration: %{NUMBER} %{NUMBER} ms
––– input –––
php -r "exit(round(abs($multiple_disk_chunks - $optimize) / ($multiple_disk_chunks + $optimize)/2 * 100, 0) >= 15);" 2>\dev\null && echo "----$?----"
––– output –––
----0----


          Update test-performance-for-queries-with-multiple-disk-chunks.rec

07b6457

github-actions bot commented Jun 19, 2024

clt-amd64

👎 CLT tests in test/clt-tests/core/test-performance-for-queries-with-multiple-disk-chunks
✅ OK: 0
❌ Failed: 1
⏳ Duration: 314s
👉 Check Action Results for commit 00253fe
Failed tests:

test/clt-tests/core/test-performance-for-queries-with-multiple-disk-chunks.rec

––– input –––
rm -f /var/log/manticore/searchd.log; searchd --stopwait > /dev/null; searchd; if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore/searchd.log;fi
––– output –––
Manticore %{SEMVER} %{COMMITDATE}#!/(\sdev)?\s/!#(columnar %{SEMVER} %{COMMITDATE}) (secondary %{SEMVER} %{COMMITDATE}) (knn %{SEMVER} %{COMMITDATE})
Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
Copyright (c) 2017-%{YEAR}, Manticore Software LTD (https://manticoresearch.com)
[#!/[0-9]{2}:[0-9]{2}.[0-9]{3}/!#] [#!/[0-9]+/!#] using config file '/etc/manticoresearch/manticore.conf' (%{NUMBER} chars)...
starting daemon version '%{SEMVER} %{COMMITDATE}#!/(\sdev)?\s/!#(columnar %{SEMVER} %{COMMITDATE}) (secondary %{SEMVER} %{COMMITDATE}) (knn %{SEMVER} %{COMMITDATE})' ...
listening on %{IPADDR}:9312 for sphinx and http(s)
listening on %{IPADDR}:9306 for mysql
listening on %{IPADDR}:9308 for sphinx and http(s)
Buddy started!
––– input –––
mysql -h0 -P9306 -e "set global auto_optimize=0;"
––– output –––
––– input –––
php -d memory_limit=-1 ./test/clt-tests/comparison-overhead-json-sql/load_us_names.php 10000 10 10000000 1 > /dev/null
––– output –––
––– input –––
mysql -h0 -P9306 -e "select count(*) from name;"
––– output –––
+----------+
| count(*) |
+----------+
| 10000000 |
+----------+
––– input –––
export start=$(date +%s%N)
––– output –––
––– input –––
mysql -h0 -P9306 -e "select * from name where match('*on*');" > /dev/null; echo $?
––– output –––
0
––– input –––
export end=$(date +%s%N)
––– output –––
––– input –––
export multiple_disk_chunks=$(( (end - start) / 1000000 ))
––– output –––
––– input –––
mysql -P9306 -h0 -e "select count(*) from name.@status;"
––– output –––
+----------+
| count(*) |
+----------+
- |        8 |
+ |        7 |
+----------+
––– input –––
mysql -h0 -P9306 -e "OPTIMIZE TABLE name OPTION cutoff=1, sync=1;"
––– output –––
––– input –––
mysql -h0 -P9306 -e "show table name status" | grep "disk_chunks"
––– output –––
| disk_chunks                 | 1                                                                                                      |
––– input –––
export start=$(date +%s%N)
––– output –––
––– input –––
mysql -h0 -P9306 -e "select * from name where match('*on*');" > /dev/null; echo $?
––– output –––
0
––– input –––
export end=$(date +%s%N)
––– output –––
––– input –––
export optimize=$(( (end - start) / 1000000 ))
––– output –––
––– input –––
echo "Duration: ${optimize} ${multiple_disk_chunks} ms"
––– output –––
Duration: %{NUMBER} %{NUMBER} ms
––– input –––
php -r "exit(round(abs($multiple_disk_chunks - $optimize) / ($multiple_disk_chunks + $optimize)/2 * 100, 0) >= 20);" 2>\dev\null && echo "----$?----"
––– output –––
----0----

github-actions bot commented Jun 19, 2024

clt

👎 CLT tests in test/clt-tests/core/ test/clt-tests/expected-errors/
✅ OK: 15
❌ Failed: 1
⏳ Duration: 556s
👉 Check Action Results for commit 00253fe
Failed tests:

test/clt-tests/core/test-performance-for-queries-with-multiple-disk-chunks.rec

––– input –––
rm -f /var/log/manticore/searchd.log; searchd --stopwait > /dev/null; searchd; if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore/searchd.log;fi
––– output –––
Manticore %{SEMVER} %{COMMITDATE}#!/(\sdev)?\s/!#(columnar %{SEMVER} %{COMMITDATE}) (secondary %{SEMVER} %{COMMITDATE}) (knn %{SEMVER} %{COMMITDATE})
Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
Copyright (c) 2017-%{YEAR}, Manticore Software LTD (https://manticoresearch.com)
[#!/[0-9]{2}:[0-9]{2}.[0-9]{3}/!#] [#!/[0-9]+/!#] using config file '/etc/manticoresearch/manticore.conf' (%{NUMBER} chars)...
starting daemon version '%{SEMVER} %{COMMITDATE}#!/(\sdev)?\s/!#(columnar %{SEMVER} %{COMMITDATE}) (secondary %{SEMVER} %{COMMITDATE}) (knn %{SEMVER} %{COMMITDATE})' ...
listening on %{IPADDR}:9312 for sphinx and http(s)
listening on %{IPADDR}:9306 for mysql
listening on %{IPADDR}:9308 for sphinx and http(s)
Buddy started!
––– input –––
mysql -h0 -P9306 -e "set global auto_optimize=0;"
––– output –––
––– input –––
php -d memory_limit=-1 ./test/clt-tests/comparison-overhead-json-sql/load_us_names.php 10000 10 10000000 1 > /dev/null
––– output –––
––– input –––
mysql -h0 -P9306 -e "select count(*) from name;"
––– output –––
+----------+
| count(*) |
+----------+
| 10000000 |
+----------+
––– input –––
export start=$(date +%s%N)
––– output –––
––– input –––
mysql -h0 -P9306 -e "select * from name where match('*on*');" > /dev/null; echo $?
––– output –––
0
––– input –––
export end=$(date +%s%N)
––– output –––
––– input –––
export multiple_disk_chunks=$(( (end - start) / 1000000 ))
––– output –––
––– input –––
mysql -P9306 -h0 -e "select count(*) from name.@status;"
––– output –––
+----------+
| count(*) |
+----------+
- |        8 |
+ |        7 |
+----------+
––– input –––
mysql -h0 -P9306 -e "OPTIMIZE TABLE name OPTION cutoff=1, sync=1;"
––– output –––
––– input –––
mysql -h0 -P9306 -e "show table name status" | grep "disk_chunks"
––– output –––
| disk_chunks                 | 1                                                                                                      |
––– input –––
export start=$(date +%s%N)
––– output –––
––– input –––
mysql -h0 -P9306 -e "select * from name where match('*on*');" > /dev/null; echo $?
––– output –––
0
––– input –––
export end=$(date +%s%N)
––– output –––
––– input –––
export optimize=$(( (end - start) / 1000000 ))
––– output –––
––– input –––
echo "Duration: ${optimize} ${multiple_disk_chunks} ms"
––– output –––
Duration: %{NUMBER} %{NUMBER} ms
––– input –––
php -r "exit(round(abs($multiple_disk_chunks - $optimize) / ($multiple_disk_chunks + $optimize)/2 * 100, 0) >= 20);" 2>\dev\null && echo "----$?----"
––– output –––
----0----

Pavel_Shilin added 2 commits

June 19, 2024 14:45


          Added regex to test-performance-for-queries-with-multiple-disk-chunks…

a519866

….rec


          Change in the percentage of comparison

56d0d42

PavelShilin89 requested a review from sanikolaev

June 20, 2024 09:32

PavelShilin89 mentioned this pull request

performance degradation with wildcard queries with many matches when disk_chunks > 1 #2280

Closed

5 tasks

manticoresearch requested changes

View reviewed changes

devnull Outdated Show resolved Hide resolved


          Deleted devnull

de4aab0

manticoresearch approved these changes

View reviewed changes

PavelShilin89 requested a review from manticoresearch

June 20, 2024 10:42

manticoresearch approved these changes

View reviewed changes

PavelShilin89 merged commit 8c11cf3 into master

117 checks passed

PavelShilin89 linked an issue

that may be closed by this pull request

performance degradation with wildcard queries with many matches when disk_chunks > 1 #2280

Closed

5 tasks

glookka pushed a commit that referenced this pull request


          Test performance for queries with multiple disk chunks (#2318)

c782725

* Create test-performance-for-queries-with-multiple-disk-chunks.rec
---------

Co-authored-by: Pavel_Shilin <work@pavelshilin.com>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet