Add benchmark data generator, basic ingestion/persist/merge/query benchmarks #2875
Conversation
jon-wei
force-pushed
the
datagen
branch
2 times, most recently
from
01ec95a
to
5954bb9
Compare
|
@jon-wei any way we can make this run everytime we run integration tests? |
|
@fjy IMO that should be a different PR, it's non-trivial to decide what "pass" and "fail" mean for performance tests and that will require a bit more work. |
|
see also #2823 |
jon-wei
force-pushed
the
datagen
branch
2 times, most recently
from
0f7cc41
to
f14857e
Compare
|
@jon-wei would be interesting to see how this benchmark perform across 0.8.3, 0.9.0, and current master, as well as differences from building v9 directly or not. Would it be a lot of work to test that? |
|
Here are numbers for 0.8.3 and 0.9.0, you can find the branches I used to run these here: 0.8.3 (no IndexMergerV9 here) 0.9.0 |
|
On master, 4/25/2016 |
|
benchmarks look pretty good |
|
@jon-wei can you benchmark master + dimension typing? |
|
👍 |
|
@jon-wei interesting to see that topN got about 5-10% slower in master compared to 0.9.0, we might want to investigate that. Can you try to see if you can get the error bands narrower for topN? |
| private final List<Double> enumeratedProbabilities; | ||
| private final Double nullProbability; | ||
|
|
||
| public BenchmarkColumnSchema( |
There was a problem hiding this comment.
some java docs would be nice to explain what all those inputs mean
There was a problem hiding this comment.
@xvrl Added documentation comments for the distribution types and the properties
jon-wei
force-pushed
the
datagen
branch
3 times, most recently
from
5a68e00
to
b8be128
Compare
|
@xvrl Sure, I can run this again on master for TopN |
|
Here are some comparison results for master and 0.9.0 on TopN, with 10 warmup iterations and 25 benchmark iterations. I ran the benchmark twice for each. I wasn't able to see the error band reduced but the numbers are pretty close. master 4-28 0.9.0 |
| @Benchmark | ||
| @BenchmarkMode(Mode.AverageTime) | ||
| @OutputTimeUnit(TimeUnit.MICROSECONDS) | ||
| public void queryQueryableIndex(Blackhole blackhole) throws Exception |
There was a problem hiding this comment.
hmm, I think ideally we should have benchmarks for a few levels of things. this will help us track down performance improvements and regressions more easily, as well as optimize specific things more easily.
- query a single incremental index (with
queryRunnerFactory.createRunner(...).run) and materialize the results - query a single queryable index and materialize the results
- query a bunch of queryable indexes with
mergeRunners+mergeResultsapplied (like a historical would do), with a variable number of threads in themergeRunnersthread pool
1 and 2 help us test single-segment performance. 3 helps us test whole-query performance, including the merge step.
There was a problem hiding this comment.
^ this comment applies to all query types
There was a problem hiding this comment.
each query type supports those 3 benchmark types now
jon-wei
force-pushed
the
datagen
branch
3 times, most recently
from
05f6338
to
985ffa4
Compare
jon-wei
changed the title
jon-wei
force-pushed
the
datagen
branch
2 times, most recently
from
959dda5
to
765e9d5
Compare
jon-wei
changed the title
jon-wei
force-pushed
the
datagen
branch
2 times, most recently
from
6562dc1
to
078ba7b
Compare
| import java.util.concurrent.TimeUnit; | ||
|
|
||
| @State(Scope.Benchmark) | ||
| @Fork(value = 1) |
There was a problem hiding this comment.
It's good practice to always use the -server vm, i.e. add @Fork(jvmArgsPrepend = "-server", value = 1)
There was a problem hiding this comment.
Added -server option to all of the benchmarks
|
@jon-wei do you think we could give the schema a bit more explicit column names? For instance, I was trying to benchmark multi-value dimensions and it wasn't clear upfront which dimensions may be multi-value or not. Doesn't have to hold up this PR, but it would be nice to have. |
|
👍 otherwise for me |
|
@xvrl I've renamed the columns to note the type of distribution and whether they're multivalued or not |
This PR adds a utility class called BenchmarkDataGenerator for randomly generating event rows for benchmarking purposes.
Each column is independently generated (no cross-column relationships are supported), column configuration is handled by the BenchmarkColumnSchema class which defines properties like type, cardinality and distribution of values.
This PR also adds benchmarks for:
Basic benchmarks for each query type, running against a single segment with no filters or extraction functions, are added as well.
Each query benchmark has the following 3 benchmarks:
The number of rows per segment, number of total segments, and other relevant configuration are specified as parameters to the benchmarks.
Each benchmark references a schema defined in BenchmarkSchemas, with the schema to be used defined by a benchmark parameter. Currently there is one schema, "basic".
Each query benchmark can have a set of supported queries per schema, currently each benchmark has one query defined for the "basic" schema. The choice of schema and query is specified by the "schemaAndQuery" parameter, using a schemaName.queryName format.