Approx Percentile #3301

andygrove · 2021-08-25T15:43:16Z

Signed-off-by: Andy Grove andygrove@nvidia.com

Depends on rapidsai/cudf#9094

Status:

Get one test working end to end (even if results are not 100% correct)
Add casts so final percentiles are in same data type as input
GpuOverride type checking
Add config to enable feature
GpuOverride fall back to CPU if used as a reduction
PySpark integration tests
Decide on translation (if any) from Spark delta to cuDF delta
Documentation

Signed-off-by: Andy Grove <andygrove@nvidia.com>

sql-plugin/src/main/scala/org/apache/spark/sql/rapids/AggregateFunctions.scala

… is at least as accurate as CPU

sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuApproximatePercentile.scala

revans2 · 2021-09-01T14:40:26Z

sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuApproximatePercentile.scala

+  override def aggBufferAttributes: Seq[AttributeReference] = outputBuf :: Nil
+
+  // Mark as lazy to avoid being initialized when creating a GpuApproximatePercentile.
+  override lazy val initialValues: Seq[GpuExpression] = throw new UnsupportedOperationException


I think this is needed in cases where we get no data to a given task, or in some cases it might be a reduction where no data showed up at all. Then the initialValues ends up being what is returned.

Okay. I will work on writing a test case to trigger this exception and then implement something here.

Any update on this?

It looks like this is only invoked in reduction use cases when there are empty batches. We don't support reduction yet for this operator so I have updated the exception to indicate that.

sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuApproximatePercentile.scala

sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOverrides.scala

sql-plugin/src/main/scala/org/apache/spark/sql/rapids/AggregateFunctions.scala

sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuApproximatePercentile.scala

…tePercentile.scala Co-authored-by: Alessandro Bellina <abellina@gmail.com>

… into approx-percentile

integration_tests/src/main/python/hash_aggregate_test.py

abellina · 2021-09-24T20:57:30Z

sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuApproximatePercentile.scala

+
+  // The result type is the same as the input type.
+  private lazy val internalDataType: DataType = {
+    if (returnPercentileArray) ArrayType(child.dataType,


nit, indentation seems a little odd in this if statement.

abellina · 2021-09-24T20:58:46Z

sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuApproximatePercentile.scala

+  override def columnarEval(batch: ColumnarBatch): Any = {
+    val expr = child.asInstanceOf[GpuExpression]
+    withResource(expr.columnarEval(batch).asInstanceOf[GpuColumnVector]) { cv =>
+


nit, extra line.

sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuApproximatePercentile.scala

andygrove · 2021-09-27T16:29:57Z

build

sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuApproximatePercentile.scala

andygrove · 2021-09-27T20:36:49Z

build

andygrove · 2021-09-27T22:24:49Z

build failed with connection error, will try again

andygrove · 2021-09-27T22:25:11Z

build

sameerz added the feature request New feature or request label Aug 25, 2021

Rough out approx percentile aggregation

8527f94

Signed-off-by: Andy Grove <andygrove@nvidia.com>

andygrove force-pushed the approx-percentile branch from 6d42899 to 8527f94 Compare August 26, 2021 17:46

andygrove added 3 commits August 26, 2021 15:02

Cast percentiles to array of input data type

ae1f093

Signed-off-by: Andy Grove <andygrove@nvidia.com>

expand tests

06ea6be

wip save current work

481ea42

ttnghia reviewed Aug 31, 2021

View reviewed changes

sql-plugin/src/main/scala/org/apache/spark/sql/rapids/AggregateFunctions.scala Outdated Show resolved Hide resolved

Fix bug in reported data type and implement tests that check that GPU…

edc7cda

… is at least as accurate as CPU

ttnghia reviewed Aug 31, 2021

View reviewed changes

sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuApproximatePercentile.scala Outdated Show resolved Hide resolved

andygrove added 6 commits August 31, 2021 17:10

test passes from command line and IDE now

a051425

add comment

a5a069a

docs and mapping of spark delta to gpu delta

5f6b4ac

code cleanup

208f7c6

code cleanup

02f83b7

add pyspark tests

8404cb7

revans2 reviewed Sep 1, 2021

View reviewed changes

andygrove added 5 commits September 1, 2021 09:57

remove window function, add high level docs

808a3c6

remove special handling for optional accuracy

eb8bcf8

address PR feedback

e1fbd61

address more feedback

c85cd3a

simply type checks

9cd15f7

andygrove added the cudf_dependency An issue or PR with this label depends on a new feature in cudf label Sep 2, 2021

andygrove self-assigned this Sep 9, 2021

andygrove added this to the Aug 30 - Sept 10 milestone Sep 9, 2021

handle nulls in tests

031d359

sameerz modified the milestones: Aug 30 - Sept 10, Sep 13 - Sep 24 Sep 11, 2021

andygrove added 2 commits September 13, 2021 16:53

Update t-digest data type and fix scala style issues

dc2f74f

add config to enable approx_percentile

5e76dba

andygrove and others added 6 commits September 22, 2021 16:11

add error handling for unexpected percentile type

8e7c986

remove custom config and use disableByDefault instead

4eea440

remove custom config and use disableByDefault instead

d6a73f7

cherry pick groupByOnly

aa05cc4

use ExprChecks.groupByOnly

a653ef8

generated docs

e49576e

abellina reviewed Sep 23, 2021

View reviewed changes

sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuApproximatePercentile.scala Outdated Show resolved Hide resolved

andygrove and others added 3 commits September 23, 2021 15:30

Update sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuApproxima…

5d42e2c

…tePercentile.scala Co-authored-by: Alessandro Bellina <abellina@gmail.com>

use extractListElement instead of getChildColumnView

3514d4e

Merge branch 'approx-percentile' of github.com:andygrove/spark-rapids…

a774f2c

… into approx-percentile

abellina reviewed Sep 24, 2021

View reviewed changes

sameerz modified the milestones: Sep 13 - Sep 24, Sep 27 - Oct 1 Sep 25, 2021

andygrove marked this pull request as ready for review September 27, 2021 15:15

andygrove changed the title ~~WIP: Approx Percentile~~ Approx Percentile Sep 27, 2021

fix regression and address nits

967db59

revans2 reviewed Sep 27, 2021

View reviewed changes

andygrove added 2 commits September 27, 2021 14:02

fix build error with 320

75efd33

Use columnarEvalToColumn and ignore ANSI mode

31759a7

revans2 approved these changes Sep 27, 2021

View reviewed changes

abellina approved these changes Sep 27, 2021

View reviewed changes

andygrove merged commit e149f4c into NVIDIA:branch-21.10 Sep 28, 2021

andygrove deleted the approx-percentile branch September 28, 2021 03:19

pxLi mentioned this pull request Sep 29, 2021

[BUG] test_hash_groupby_approx_percentile_long_repeated_keys could hang in integration tests intermittently #3692

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Approx Percentile #3301

Approx Percentile #3301

andygrove commented Aug 25, 2021 •

edited

Loading

revans2 Sep 1, 2021

andygrove Sep 1, 2021

revans2 Sep 22, 2021

andygrove Sep 22, 2021

abellina Sep 24, 2021

abellina Sep 24, 2021

andygrove commented Sep 27, 2021

andygrove commented Sep 27, 2021

andygrove commented Sep 27, 2021

andygrove commented Sep 27, 2021

Approx Percentile #3301

Approx Percentile #3301

Conversation

andygrove commented Aug 25, 2021 • edited Loading

revans2 Sep 1, 2021

Choose a reason for hiding this comment

andygrove Sep 1, 2021

Choose a reason for hiding this comment

revans2 Sep 22, 2021

Choose a reason for hiding this comment

andygrove Sep 22, 2021

Choose a reason for hiding this comment

abellina Sep 24, 2021

Choose a reason for hiding this comment

abellina Sep 24, 2021

Choose a reason for hiding this comment

andygrove commented Sep 27, 2021

andygrove commented Sep 27, 2021

andygrove commented Sep 27, 2021

andygrove commented Sep 27, 2021

andygrove commented Aug 25, 2021 •

edited

Loading