Skip to content

Commit

Permalink
[SPARK-35361][SQL][FOLLOWUP] Switch to use while loop
Browse files Browse the repository at this point in the history 
### What changes were proposed in this pull request?

Switch to plain `while` loop following Spark [style guide](https://github.com/databricks/scala-style-guide#traversal-and-zipwithindex).

### Why are the changes needed?

`while` loop may yield better performance comparing to `foreach`.

### Does this PR introduce _any_ user-facing change?

No

### How was this patch tested?

N/A

Closes #32522 from sunchao/SPARK-35361-follow-up.

Authored-by: Chao Sun <sunchao@apple.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
  • Loading branch information
@sunchao @dongjoon-hyun
sunchao authored and dongjoon-hyun committed May 12, 2021
1 parent 77b7fe1 commit bc95c3a
Show file tree
Hide file tree
Showing 3 changed files with 57 additions and 56 deletions.
9 changes: 5 additions & 4 deletions ...st/src/main/scala/org/apache/spark/sql/catalyst/expressions/ApplyFunctionExpression.scala
View file
Original file line number Diff line number Diff line change
Expand Up @@ -30,13 +30,14 @@ case class ApplyFunctionExpression(
override def dataType: DataType = function.resultType()

private lazy val reusedRow = new GenericInternalRow(children.size)
private lazy val childrenWithIndex = children.zipWithIndex

/** Returns the result of evaluating this expression on a given input Row */
override def eval(input: InternalRow): Any = {
childrenWithIndex.foreach {
case (expr, pos) =>
reusedRow.update(pos, expr.eval(input))
var i = 0
while (i < children.length) {
val expr = children(i)
reusedRow.update(i, expr.eval(input))
i += 1
}

function.produceResult(reusedRow)
Expand Down
48 changes: 24 additions & 24 deletions sql/core/benchmarks/V2FunctionBenchmark-jdk11-results.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,43 +2,43 @@ OpenJDK 64-Bit Server VM 11.0.11+9-LTS on Linux 5.4.0-1046-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
scalar function (long + long) -> long, result_nullable = true codegen = true: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------------------------------------------
native_long_add 16079 16684 619 31.1 32.2 1.0X
java_long_add_default 45512 48772 NaN 11.0 91.0 0.4X
java_long_add_magic 19506 19672 262 25.6 39.0 0.8X
java_long_add_static_magic 18770 18901 164 26.6 37.5 0.9X
scala_long_add_default 46895 47662 1136 10.7 93.8 0.3X
scala_long_add_magic 19520 19667 188 25.6 39.0 0.8X
native_long_add 16015 16309 407 31.2 32.0 1.0X
java_long_add_default 48899 49122 352 10.2 97.8 0.3X
java_long_add_magic 19169 19302 117 26.1 38.3 0.8X
java_long_add_static_magic 18308 18373 57 27.3 36.6 0.9X
scala_long_add_default 48773 48922 136 10.3 97.5 0.3X
scala_long_add_magic 18372 18422 44 27.2 36.7 0.9X

OpenJDK 64-Bit Server VM 11.0.11+9-LTS on Linux 5.4.0-1046-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
scalar function (long + long) -> long, result_nullable = false codegen = true: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------------------------------------------------------------------
native_long_add 17363 17424 67 28.8 34.7 1.0X
java_long_add_default 43884 44592 615 11.4 87.8 0.4X
java_long_add_magic 18927 19100 206 26.4 37.9 0.9X
java_long_add_static_magic 16854 16918 76 29.7 33.7 1.0X
scala_long_add_default 43741 44016 288 11.4 87.5 0.4X
scala_long_add_magic 18770 19022 317 26.6 37.5 0.9X
native_long_add 16414 16452 41 30.5 32.8 1.0X
java_long_add_default 47640 47767 134 10.5 95.3 0.3X
java_long_add_magic 18413 18554 139 27.2 36.8 0.9X
java_long_add_static_magic 16659 16707 43 30.0 33.3 1.0X
scala_long_add_default 47821 47857 48 10.5 95.6 0.3X
scala_long_add_magic 18406 18502 99 27.2 36.8 0.9X

OpenJDK 64-Bit Server VM 11.0.11+9-LTS on Linux 5.4.0-1046-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
scalar function (long + long) -> long, result_nullable = true codegen = false: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------------------------------------------------------------------
native_long_add 41825 42237 668 12.0 83.6 1.0X
java_long_add_default 53779 53969 175 9.3 107.6 0.8X
java_long_add_magic 131478 133225 NaN 3.8 263.0 0.3X
java_long_add_static_magic 129304 129754 398 3.9 258.6 0.3X
scala_long_add_default 54602 54986 344 9.2 109.2 0.8X
scala_long_add_magic 132066 132243 159 3.8 264.1 0.3X
native_long_add 36335 36366 27 13.8 72.7 1.0X
java_long_add_default 53930 54056 155 9.3 107.9 0.7X
java_long_add_magic 126621 127109 471 3.9 253.2 0.3X
java_long_add_static_magic 126914 127193 251 3.9 253.8 0.3X
scala_long_add_default 55812 55949 141 9.0 111.6 0.7X
scala_long_add_magic 127629 127900 420 3.9 255.3 0.3X

OpenJDK 64-Bit Server VM 11.0.11+9-LTS on Linux 5.4.0-1046-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
scalar function (long + long) -> long, result_nullable = false codegen = false: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
--------------------------------------------------------------------------------------------------------------------------------------------------------------
native_long_add 40817 41093 388 12.2 81.6 1.0X
java_long_add_default 54090 54563 425 9.2 108.2 0.8X
java_long_add_magic 129341 129867 613 3.9 258.7 0.3X
java_long_add_static_magic 127292 127432 218 3.9 254.6 0.3X
scala_long_add_default 53397 53670 328 9.4 106.8 0.8X
scala_long_add_magic 128455 128541 138 3.9 256.9 0.3X
native_long_add 37433 37794 312 13.4 74.9 1.0X
java_long_add_default 53629 53946 416 9.3 107.3 0.7X
java_long_add_magic 160091 160605 549 3.1 320.2 0.2X
java_long_add_static_magic 157228 158430 1372 3.2 314.5 0.2X
scala_long_add_default 54026 54197 187 9.3 108.1 0.7X
scala_long_add_magic 160926 161351 526 3.1 321.9 0.2X

56 changes: 28 additions & 28 deletions sql/core/benchmarks/V2FunctionBenchmark-results.txt
View file
Original file line number Diff line number Diff line change
@@ -1,44 +1,44 @@
OpenJDK 64-Bit Server VM 1.8.0_292-b10 on Linux 5.4.0-1046-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
scalar function (long + long) -> long, result_nullable = true codegen = true: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------------------------------------------
native_long_add 9723 11619 1643 51.4 19.4 1.0X
java_long_add_default 38003 38591 513 13.2 76.0 0.3X
java_long_add_magic 12398 13007 792 40.3 24.8 0.8X
java_long_add_static_magic 11551 11711 138 43.3 23.1 0.8X
scala_long_add_default 39482 39762 275 12.7 79.0 0.2X
scala_long_add_magic 12794 12830 33 39.1 25.6 0.8X
native_long_add 11275 12171 998 44.3 22.5 1.0X
java_long_add_default 37040 37728 755 13.5 74.1 0.3X
java_long_add_magic 13974 14218 276 35.8 27.9 0.8X
java_long_add_static_magic 12019 12260 248 41.6 24.0 0.9X
scala_long_add_default 38262 38520 228 13.1 76.5 0.3X
scala_long_add_magic 13940 14062 202 35.9 27.9 0.8X

OpenJDK 64-Bit Server VM 1.8.0_292-b10 on Linux 5.4.0-1046-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
scalar function (long + long) -> long, result_nullable = false codegen = true: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------------------------------------------------------------------
native_long_add 9984 10285 303 50.1 20.0 1.0X
java_long_add_default 36510 36989 570 13.7 73.0 0.3X
java_long_add_magic 13391 13764 332 37.3 26.8 0.7X
java_long_add_static_magic 10033 10462 388 49.8 20.1 1.0X
scala_long_add_default 35104 35480 375 14.2 70.2 0.3X
scala_long_add_magic 13587 13899 366 36.8 27.2 0.7X
native_long_add 11041 11194 133 45.3 22.1 1.0X
java_long_add_default 35213 35716 454 14.2 70.4 0.3X
java_long_add_magic 13857 13945 78 36.1 27.7 0.8X
java_long_add_static_magic 11272 11382 97 44.4 22.5 1.0X
scala_long_add_default 35157 36078 1103 14.2 70.3 0.3X
scala_long_add_magic 14005 14194 223 35.7 28.0 0.8X

OpenJDK 64-Bit Server VM 1.8.0_292-b10 on Linux 5.4.0-1046-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
scalar function (long + long) -> long, result_nullable = true codegen = false: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------------------------------------------------------------------
native_long_add 32473 32622 247 15.4 64.9 1.0X
java_long_add_default 44108 44120 11 11.3 88.2 0.7X
java_long_add_magic 166139 167629 1828 3.0 332.3 0.2X
java_long_add_static_magic 181452 183355 1668 2.8 362.9 0.2X
scala_long_add_default 42405 42652 330 11.8 84.8 0.8X
scala_long_add_magic 196868 198003 1033 2.5 393.7 0.2X
native_long_add 33550 33648 144 14.9 67.1 1.0X
java_long_add_default 44094 44832 690 11.3 88.2 0.8X
java_long_add_magic 173631 175419 1611 2.9 347.3 0.2X
java_long_add_static_magic 173095 174496 1530 2.9 346.2 0.2X
scala_long_add_default 46445 46774 430 10.8 92.9 0.7X
scala_long_add_magic 174399 175016 550 2.9 348.8 0.2X

OpenJDK 64-Bit Server VM 1.8.0_292-b10 on Linux 5.4.0-1046-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
scalar function (long + long) -> long, result_nullable = false codegen = false: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
--------------------------------------------------------------------------------------------------------------------------------------------------------------
native_long_add 30182 30387 279 16.6 60.4 1.0X
java_long_add_default 42862 43009 209 11.7 85.7 0.7X
java_long_add_magic 218295 219387 1078 2.3 436.6 0.1X
java_long_add_static_magic 211812 213150 1898 2.4 423.6 0.1X
scala_long_add_default 42401 42642 234 11.8 84.8 0.7X
scala_long_add_magic 214497 214760 307 2.3 429.0 0.1X
native_long_add 36506 36656 251 13.7 73.0 1.0X
java_long_add_default 47151 47540 370 10.6 94.3 0.8X
java_long_add_magic 178691 182457 1327 2.8 357.4 0.2X
java_long_add_static_magic 177151 178258 1151 2.8 354.3 0.2X
scala_long_add_default 47127 47760 666 10.6 94.3 0.8X
scala_long_add_magic 179941 181417 1545 2.8 359.9 0.2X

0 comments on commit bc95c3a

Please sign in to comment.