-
Notifications
You must be signed in to change notification settings - Fork 3.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
minor optimize IndexMerger's MMappedIndexRowIterable #2084
Conversation
The performance numbers is: 2015-12-11T15:01:42,938 INFO [main] io.druid.segment.IndexMerger - outDir[/var/folders/z7/g5zy3kfj7t54y1f074hsqw7h0000gn/T/base1258220153873171419flush/merged/v8-tmp] walked 500,000/500,000 rows in 15,017 millis. After: 2015-12-11T14:50:33,574 INFO [main] io.druid.segment.IndexMerger - outDir[/var/folders/z7/g5zy3kfj7t54y1f074hsqw7h0000gn/T/base2937734964603563157flush/merged/v8-tmp] walked 500,000/500,000 rows in 14,069 millis. |
+1 |
@binlijin what did you run the benchmarks on, ec2 or your laptop? (or some other dedicated machine) |
@drcrallen China doesn't have EC2, it has AliCloud edit: my bad, it does have EC2 :P |
allright I'll be more explicit: Given the short timerange of the tests presented, and the small improvement (small improvements are good if they are real!) are variations such as frequency governor fluctuations taken into account? |
@drcrallen I run it on my laptop. |
@binlijin Could you include the test you've used in druid? We might need standard test for the performance. |
@navis, i test it with our one million real data. |
@binlijin can you try running on a dedicated machine, the difference you are seeing could just be due to clock rate / temperature fluctuations on your laptop is what @drcrallen is saying |
I test it on our test machine, the performance number is: 2015-12-15 06:35:17,978 INFO [main] segment.IndexMerger (Logger.java:info(70)) - outDir[/tmp/base8564891586222722664flush/merged/v8-tmp] walked 500,000/500,000 rows in 22,278 millis. 2015-12-15 06:37:38,199 INFO [main] segment.IndexMerger (Logger.java:info(70)) - outDir[/tmp/base7286842343919247533flush/merged/v8-tmp] walked 500,000/500,000 rows in 21,865 millis. After: 2015-12-15 06:10:55,844 INFO [main] segment.IndexMerger (Logger.java:info(70)) - outDir[/tmp/base3754891995514154122flush/merged/v8-tmp] walked 500,000/500,000 rows in 19,548 millis. 2015-12-15 06:13:21,332 INFO [main] segment.IndexMerger (Logger.java:info(70)) - outDir[/tmp/base2471600163749106438flush/merged/v8-tmp] walked 500,000/500,000 rows in 21,338 millis. It is slow because my laptop is SSD and this machine is not. processor : 23 |
👍 |
minor optimize IndexMerger's MMappedIndexRowIterable
Copy from IndexMaker.