How to test database performance?

[michaelopdenacker]

Any ideas to test the performance of Elixir queries through various filesystems (ext4, btrfs, f2fs...) and through various database layouts (Btree, Hash, etc)?

I've got a laptop setup for such testing, and I wonder how we can use the existing test infrastructure for such performance testing, like running 1,000 queries in a row.

I'll test indexing performance too, but that's quite trivial ("time ./update.py").

The end goal is to reduce indexing time (about 4 days for Linux on our build server) and of course improve query performance.

@cxw42 , that's a question for you... I kinda recall that you already talked about testing, but that would be useful to document such testing somewhere. I'll update README.md accordingly.

Cheers,
Michael.

[cxw42]

use the existing test infrastructure

Copy t/200-api.t to a file that doesn't end in .t, e.g., t/speedtest.pm. Then replace the run_produces_ok statement with whatever queries you want, and run it with perl t/speedtest.pm. prove will ignore it since it doesn't end in .t.

I am happy to add functions to make particular types of tests easier to write if you would be interested. Those would go in t/TestHelpers.pm.

Will you still have (do you already have) the intern working on speedups?

[michaelopdenacker]

Hi Chris,
Thanks for the hints.
I'll try this. Some functions helper functions for identifier lookups and file contents would help indeed, factorizing some code with what's in 100-basic.t.
Our intern is joining next week :-)
Cheers,
Michael.

[michaelopdenacker]

Indeed, Maxime implemented a testing solution in Python. That's easier to call Python code (query.py) instead of using Perl to call Python.
Thanks anyway!
Michael.