Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Port ScanFilterAndProjectOperator to WorkProcessorSourceOperator #909

Merged
merged 2 commits into from Jun 13, 2019

Conversation

4 participants
@sopel39
Copy link
Member

commented Jun 4, 2019

No description provided.

@sopel39 sopel39 added the WIP label Jun 4, 2019

@cla-bot cla-bot bot added the cla-signed label Jun 4, 2019

@martint

This comment has been minimized.

Copy link
Member

commented Jun 5, 2019

Build failed with the following. It may be related:

[ERROR] testScanFilterAndProjectOperator(io.prestosql.plugin.hive.TestOrcPageSourceMemoryTracking)  Time elapsed: 0.106 s  <<< FAILURE!
java.lang.AssertionError: expected:<0> to be between <90000> and <499999> inclusive
	at io.airlift.testing.Assertions.fail(Assertions.java:323)
	at io.airlift.testing.Assertions.assertBetweenInclusive(Assertions.java:252)
	at io.airlift.testing.Assertions.assertBetweenInclusive(Assertions.java:230)
	at io.prestosql.plugin.hive.TestOrcPageSourceMemoryTracking.testScanFilterAndProjectOperator(TestOrcPageSourceMemoryTracking.java:391)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
	at org.testng.internal.Invoker.invokeMethod(Invoker.java:645)
	at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:851)
	at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1177)
	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:129)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:112)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

@sopel39 sopel39 force-pushed the sopel39:ks/work_processor_scan branch 3 times, most recently from 740ee57 to d7be7f1 Jun 5, 2019

@sopel39 sopel39 removed the WIP label Jun 6, 2019

@sopel39

This comment has been minimized.

Copy link
Member Author

commented Jun 6, 2019

Some benchmarks (no difference nor should there be one)

AFTER
Benchmark                                                      (columnCount)  (dictionaryBlocks)  (positionsPerPage)   (type)  Mode  Cnt     Score    Error  Units
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented              2               false                  32  varchar  avgt   50   197,671 ±  1,605  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented              2               false                  32   bigint  avgt   50    52,653 ±  0,749  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented              2               false                1024  varchar  avgt   50   127,518 ±  0,721  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented              2               false                1024   bigint  avgt   50    12,883 ±  0,294  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented              2                true                  32  varchar  avgt   50   114,364 ±  0,806  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented              2                true                  32   bigint  avgt   50    58,120 ±  0,936  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented              2                true                1024  varchar  avgt   50    44,039 ±  0,206  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented              2                true                1024   bigint  avgt   50    11,288 ±  0,148  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented              4               false                  32  varchar  avgt   50   307,072 ±  2,100  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented              4               false                  32   bigint  avgt   50    90,145 ±  1,251  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented              4               false                1024  varchar  avgt   50   190,552 ±  0,644  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented              4               false                1024   bigint  avgt   50    19,464 ±  0,257  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented              4                true                  32  varchar  avgt   50   196,599 ±  1,365  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented              4                true                  32   bigint  avgt   50   100,113 ±  1,170  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented              4                true                1024  varchar  avgt   50    73,355 ±  0,416  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented              4                true                1024   bigint  avgt   50    17,414 ±  0,302  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented              8               false                  32   bigint  avgt   50   166,200 ±  1,238  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented              8               false                1024  varchar  avgt   50   326,349 ±  1,065  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented              8               false                1024   bigint  avgt   50    33,508 ±  0,380  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented              8                true                  32  varchar  avgt   50   369,083 ±  2,759  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented              8                true                  32   bigint  avgt   50   183,561 ±  2,248  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented              8                true                1024  varchar  avgt   50   134,422 ±  0,865  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented              8                true                1024   bigint  avgt   50    31,375 ±  0,567  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented             16               false                  32   bigint  avgt   50   302,419 ±  3,271  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented             16               false                1024  varchar  avgt   50   604,495 ±  2,642  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented             16               false                1024   bigint  avgt   50    67,055 ±  3,298  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented             16                true                  32  varchar  avgt   50   752,313 ±  5,945  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented             16                true                  32   bigint  avgt   50   365,388 ±  5,397  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented             16                true                1024  varchar  avgt   50   256,667 ±  1,466  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented             16                true                1024   bigint  avgt   50    59,493 ±  1,590  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented             32               false                  32   bigint  avgt   50   647,923 ±  5,126  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented             32               false                1024  varchar  avgt   50  1193,047 ± 17,834  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented             32               false                1024   bigint  avgt   50   149,988 ±  2,021  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented             32                true                  32   bigint  avgt   50   792,429 ± 12,177  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented             32                true                1024  varchar  avgt   50   532,090 ±  6,597  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented             32                true                1024   bigint  avgt   50   118,821 ±  2,943  ms/op


BEFORE:
Benchmark                                                      (columnCount)  (dictionaryBlocks)  (positionsPerPage)   (type)  Mode  Cnt     Score    Error  Units
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented              2               false                  32  varchar  avgt   50   198,075 ±  1,912  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented              2               false                  32   bigint  avgt   50    52,468 ±  0,831  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented              2               false                1024  varchar  avgt   50   127,872 ±  0,728  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented              2               false                1024   bigint  avgt   50    13,284 ±  0,340  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented              2                true                  32  varchar  avgt   50   114,822 ±  1,016  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented              2                true                  32   bigint  avgt   50    57,703 ±  0,683  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented              2                true                1024  varchar  avgt   50    43,479 ±  0,133  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented              2                true                1024   bigint  avgt   50    11,251 ±  0,118  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented              4               false                  32  varchar  avgt   50   309,169 ±  1,956  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented              4               false                  32   bigint  avgt   50    90,934 ±  1,482  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented              4               false                1024  varchar  avgt   50   190,672 ±  1,179  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented              4               false                1024   bigint  avgt   50    19,388 ±  0,648  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented              4                true                  32  varchar  avgt   50   195,510 ±  1,441  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented              4                true                  32   bigint  avgt   50   102,869 ±  1,491  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented              4                true                1024  varchar  avgt   50    73,339 ±  0,241  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented              4                true                1024   bigint  avgt   50    17,728 ±  0,365  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented              8               false                  32   bigint  avgt   50   164,731 ±  1,897  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented              8               false                1024  varchar  avgt   50   326,028 ±  1,114  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented              8               false                1024   bigint  avgt   50    36,608 ±  1,884  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented              8                true                  32  varchar  avgt   50   367,688 ±  2,467  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented              8                true                  32   bigint  avgt   50   186,429 ±  2,546  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented              8                true                1024  varchar  avgt   50   134,358 ±  0,607  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented              8                true                1024   bigint  avgt   50    31,749 ±  0,654  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented             16               false                  32   bigint  avgt   50   299,397 ±  2,471  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented             16               false                1024  varchar  avgt   50   593,794 ±  1,679  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented             16               false                1024   bigint  avgt   50    66,637 ±  3,281  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented             16                true                  32  varchar  avgt   50   737,361 ±  4,126  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented             16                true                  32   bigint  avgt   50   360,040 ±  4,444  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented             16                true                1024  varchar  avgt   50   253,364 ±  1,360  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented             16                true                1024   bigint  avgt   50    59,091 ±  1,935  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented             32               false                  32   bigint  avgt   50   646,715 ±  5,267  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented             32               false                1024  varchar  avgt   50  1186,717 ± 21,780  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented             32               false                1024   bigint  avgt   50   146,103 ±  6,443  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented             32                true                  32   bigint  avgt   50   773,952 ±  4,360  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented             32                true                1024  varchar  avgt   50   504,391 ±  3,496  ms/op
BenchmarkScanFilterAndProjectOperator.benchmarkColumnOriented             32                true                1024   bigint  avgt   50   115,704 ±  2,939  ms/op

@sopel39 sopel39 requested a review from dain Jun 6, 2019

@dain
Copy link
Member

left a comment

Looks good. There are a couple or readability suggestions, and one comment on blockedOnSplits that I think is bug.

Show resolved Hide resolved .../main/java/io/prestosql/operator/WorkProcessorSourceOperatorAdapter.java Outdated

Object splitInfo = split.getInfo();
if (splitInfo != null) {
operatorContext.setInfoSupplier(() -> new SplitOperatorInfo(splitInfo));

This comment has been minimized.

Copy link
@dain

dain Jun 11, 2019

Member

This is not an issue to solve today, but there is only one slot in operator context for an info supplier, so if you get multiple, only the last one will be respected. For ExchangeOperator we get around this by putting the exchange client status in this slot: https://github.com/prestosql/presto/blob/master/presto-main/src/main/java/io/prestosql/operator/ExchangeOperator.java#L108

This comment has been minimized.

Copy link
@sopel39

sopel39 Jun 12, 2019

Author Member

I will address that by exposing Optional<OperatorInfo> WorkProcessorOperator#getOperatorInfo method.

Show resolved Hide resolved .../main/java/io/prestosql/operator/WorkProcessorSourceOperatorAdapter.java Outdated
Show resolved Hide resolved .../main/java/io/prestosql/operator/WorkProcessorSourceOperatorAdapter.java
Show resolved Hide resolved .../main/java/io/prestosql/operator/WorkProcessorSourceOperatorAdapter.java Outdated
@sopel39

This comment has been minimized.

Copy link
Member Author

commented Jun 12, 2019

ac

@dain

dain approved these changes Jun 12, 2019

Copy link
Member

left a comment

One minor comment, but otherwise looks good.

private class SplitBuffer
implements WorkProcessor.Process<Split>
{
final List<Split> pendingSplits = new ArrayList<>();

This comment has been minimized.

Copy link
@dain

dain Jun 12, 2019

Member

I think all three of these can be private

@sopel39 sopel39 force-pushed the sopel39:ks/work_processor_scan branch from d7c6f96 to ae7b971 Jun 13, 2019

@sopel39 sopel39 merged commit aa33591 into prestosql:master Jun 13, 2019

1 of 2 checks passed

Travis CI - Pull Request Build Errored
Details
verification/cla-signed
Details

@sopel39 sopel39 deleted the sopel39:ks/work_processor_scan branch Jun 13, 2019

@electrum electrum added this to the 315 milestone Jun 13, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.