[SPARK-32901][CORE] Do not allocate memory while spilling UnsafeExternalSorter #29785

tomvanbussel · 2020-09-17T11:39:49Z

What changes were proposed in this pull request?

This PR changes UnsafeExternalSorter to no longer allocate any memory while spilling. In particular it removes the allocation of a new pointer array in UnsafeInMemorySorter. Instead the new pointer array is allocated whenever the next record is inserted into the sorter.

Why are the changes needed?

Without this change the UnsafeExternalSorter could throw an OOM while spilling. The following sequence of events would have triggered an OOM:

UnsafeExternalSorter runs out of space in its pointer array and attempts to allocate a new large array to replace the old one.
TaskMemoryManager tries to allocate the memory backing the new large array using MemoryManager, but MemoryManager is only willing to return most but not all of the memory requested.
TaskMemoryManager asks UnsafeExternalSorter to spill, which causes UnsafeExternalSorter to spill the current run to disk, to free its record pages and to reset its UnsafeInMemorySorter.
UnsafeInMemorySorter frees the old pointer array, and tries to allocate a new small pointer array.
TaskMemoryManager tries to allocate the memory backing the small array using MemoryManager, but MemoryManager is unwilling to give it any memory, as the TaskMemoryManager is still holding on to the memory it got for the new large array.
TaskMemoryManager again asks UnsafeExternalSorter to spill, but this time there is nothing to spill.
UnsafeInMemorySorter receives less memory than it requested, and causes a SparkOutOfMemoryError to be thrown, which causes the current task to fail.

With the changes in the PR the following will happen instead:

UnsafeExternalSorter runs out of space in its pointer array and attempts to allocate a new large array to replace the old one.
TaskMemoryManager tries to allocate the memory backing the new large array using MemoryManager, but MemoryManager is only willing to return most but not all of the memory requested.
TaskMemoryManager asks UnsafeExternalSorter to spill, which causes UnsafeExternalSorter to spill the current run to disk, to free its record pages and to reset its UnsafeInMemorySorter.
UnsafeInMemorySorter frees the old pointer array.
TaskMemoryManager returns control to UnsafeExternalSorter.growPointerArrayIfNecessary (either by returning the the new large array or by throwing a SparkOutOfMemoryError).
UnsafeExternalSorter either frees the new large array or it ignores the SparkOutOfMemoryError depending on what happened in the previous step.
UnsafeExternalSorter successfully allocates a new small pointer array and operation continues as normal.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Tests were added in UnsafeExternalSorterSuite and UnsafeInMemorySorterSuite.

…nalSorter

tomvanbussel · 2020-09-17T11:40:06Z

cc @hvanhovell @cloud-fan

cloud-fan · 2020-09-17T12:05:16Z

OK to test

cloud-fan · 2020-09-17T12:06:23Z

I vaguely remember that, the memory manager won't ask a memory consumer to spill when the memory consumer is requesting memory right now. @tomvanbussel can you double-check the related code?

tomvanbussel · 2020-09-17T12:11:03Z

@cloud-fan I checked the code (TaskMemoryManager.acquireExecutionMemory()) and it seems that the memory will first ask the other memory consumers to spill before it asks the memory consumer that requested the memory to spill. But it does ask the requesting memory consumer to spill if necessary.

core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java

hvanhovell · 2020-09-17T14:33:51Z

core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeInMemorySorter.java

-      // checkstyle.on: RegexpSinglelineJava
+    if (array != null) {
+      if (newArray.size() < array.size()) {
+        // checkstyle.off: RegexpSinglelineJava


Any idea why this is needed?

No clue. The lines with the comments are newer than the line between them, so I'm sure they're good for something...

MaxGekk

In PR's description no no longer allocate. Double no compensate each other? ;-)

tomvanbussel · 2020-09-17T14:50:05Z

Fixed, thanks Max ;)

dongjoon-hyun · 2020-09-20T01:14:04Z

Retest this please.

SparkQA · 2020-09-20T04:21:20Z

Test build #128899 has finished for PR 29785 at commit a74aa0a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2020-09-21T08:58:39Z

@tomvanbussel thanks for putting the details steps about how to trigger the bug in the PR description! Can you also put the detailed steps after the fix?

tomvanbussel · 2020-09-21T09:09:49Z

@cloud-fan Done :)

core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java

Ngone51 · 2020-09-22T14:07:26Z

TaskMemoryManager tries to allocate the memory backing the small array using MemoryManager, but MemoryManager is unwilling to give it any memory, as the TaskMemoryManager is still holding on to the memory it got for the large array.

Doesn't the UnsafeInMemorySorter release the old large point array before requiring the new small point array? Why would TaskMemoryManager still hold the memory of the large array(I assume you mean the large point array)?

tomvanbussel · 2020-09-22T14:47:43Z

@Ngone51 There are three arrays in the example:

The old pointer array.
A new large pointer array that's allocated in growPointerArrayIfNecessary() to replace the old pointer array. The allocation of this array triggered the spill.
A new small pointer array that's allocated while spilling to replace the old pointer array. The allocation of this array causes the OOM exception.

The old pointer array has been released, but TaskMemoryManager is still trying to allocate the new large pointer array when spilling is triggered, and still holds onto the memory it got from the MemoryManager.

Ngone51 · 2020-09-23T06:57:38Z

The old pointer array has been released, but TaskMemoryManager is still trying to allocate the new large pointer array when spilling is triggered, and still holds onto the memory it got from the MemoryManager.

But from the description of step 3,4,5, the TaskMemoryManager is trying to allocate the new small pointer array rather than the large one:

TaskMemoryManager asks UnsafeExternalSorter to spill, which causes UnsafeExternalSorter to spill the current run to disk, to free its record pages and to reset its UnsafeInMemorySorter.

UnsafeInMemorySorter frees the old pointer array, and tries to allocate a new small pointer array.

TaskMemoryManager tries to allocate the memory backing the small array using MemoryManager, but MemoryManager is unwilling to give it any memory, as the TaskMemoryManager is still holding on to the memory it got for the new large array.

IIUC, it should be the small pointer array...

tomvanbussel · 2020-09-24T08:44:40Z

@Ngone51 Thanks for clarifying! It's both trying to allocate a new large pointer array and a new small pointer array. The allocation of the new large pointer array was what triggered the initial spill, and the new small pointer is allocated while spilling. While it's spilling (and the new small pointer array is allocated), the TaskMemoryManager is still holding to the memory for the partially allocated new large pointer array. Any ideas on how I can clarify this in the description?

hvanhovell

LGTM

core/src/test/scala/org/apache/spark/memory/TestMemoryManager.scala

cloud-fan · 2020-09-25T08:54:18Z

Is it possible to refactor it a little bit to make the logic simpler? It's really tricky that we must be careful at each step, in case something was spilled right before the step.

I'm fine if refactoring is too hard and merge the PR as it is.

tomvanbussel · 2020-09-25T09:48:34Z

@cloud-fan I don't really see a way to simply this without making invasive changes to memory management in Spark. For instance we could simplify this if we had some way to reserve memory up front. That seems like a lot of work though, so for now I have tried to keep it understand by adding lots of comments.

hvanhovell · 2020-09-29T11:04:45Z

Alright merging to master/3.0.

…nalSorter ### What changes were proposed in this pull request? This PR changes `UnsafeExternalSorter` to no longer allocate any memory while spilling. In particular it removes the allocation of a new pointer array in `UnsafeInMemorySorter`. Instead the new pointer array is allocated whenever the next record is inserted into the sorter. ### Why are the changes needed? Without this change the `UnsafeExternalSorter` could throw an OOM while spilling. The following sequence of events would have triggered an OOM: 1. `UnsafeExternalSorter` runs out of space in its pointer array and attempts to allocate a new large array to replace the old one. 2. `TaskMemoryManager` tries to allocate the memory backing the new large array using `MemoryManager`, but `MemoryManager` is only willing to return most but not all of the memory requested. 3. `TaskMemoryManager` asks `UnsafeExternalSorter` to spill, which causes `UnsafeExternalSorter` to spill the current run to disk, to free its record pages and to reset its `UnsafeInMemorySorter`. 4. `UnsafeInMemorySorter` frees the old pointer array, and tries to allocate a new small pointer array. 5. `TaskMemoryManager` tries to allocate the memory backing the small array using `MemoryManager`, but `MemoryManager` is unwilling to give it any memory, as the `TaskMemoryManager` is still holding on to the memory it got for the new large array. 6. `TaskMemoryManager` again asks `UnsafeExternalSorter` to spill, but this time there is nothing to spill. 7. `UnsafeInMemorySorter` receives less memory than it requested, and causes a `SparkOutOfMemoryError` to be thrown, which causes the current task to fail. With the changes in the PR the following will happen instead: 1. `UnsafeExternalSorter` runs out of space in its pointer array and attempts to allocate a new large array to replace the old one. 2. `TaskMemoryManager` tries to allocate the memory backing the new large array using `MemoryManager`, but `MemoryManager` is only willing to return most but not all of the memory requested. 3. `TaskMemoryManager` asks `UnsafeExternalSorter` to spill, which causes `UnsafeExternalSorter` to spill the current run to disk, to free its record pages and to reset its `UnsafeInMemorySorter`. 4. `UnsafeInMemorySorter` frees the old pointer array. 5. `TaskMemoryManager` returns control to `UnsafeExternalSorter.growPointerArrayIfNecessary` (either by returning the the new large array or by throwing a `SparkOutOfMemoryError`). 6. `UnsafeExternalSorter` either frees the new large array or it ignores the `SparkOutOfMemoryError` depending on what happened in the previous step. 7. `UnsafeExternalSorter` successfully allocates a new small pointer array and operation continues as normal. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Tests were added in `UnsafeExternalSorterSuite` and `UnsafeInMemorySorterSuite`. Closes #29785 from tomvanbussel/SPARK-32901. Authored-by: Tom van Bussel <tom.vanbussel@databricks.com> Signed-off-by: herman <herman@databricks.com> (cherry picked from commit f167002) Signed-off-by: herman <herman@databricks.com>

…nalSorter This PR changes `UnsafeExternalSorter` to no longer allocate any memory while spilling. In particular it removes the allocation of a new pointer array in `UnsafeInMemorySorter`. Instead the new pointer array is allocated whenever the next record is inserted into the sorter. Without this change the `UnsafeExternalSorter` could throw an OOM while spilling. The following sequence of events would have triggered an OOM: 1. `UnsafeExternalSorter` runs out of space in its pointer array and attempts to allocate a new large array to replace the old one. 2. `TaskMemoryManager` tries to allocate the memory backing the new large array using `MemoryManager`, but `MemoryManager` is only willing to return most but not all of the memory requested. 3. `TaskMemoryManager` asks `UnsafeExternalSorter` to spill, which causes `UnsafeExternalSorter` to spill the current run to disk, to free its record pages and to reset its `UnsafeInMemorySorter`. 4. `UnsafeInMemorySorter` frees the old pointer array, and tries to allocate a new small pointer array. 5. `TaskMemoryManager` tries to allocate the memory backing the small array using `MemoryManager`, but `MemoryManager` is unwilling to give it any memory, as the `TaskMemoryManager` is still holding on to the memory it got for the new large array. 6. `TaskMemoryManager` again asks `UnsafeExternalSorter` to spill, but this time there is nothing to spill. 7. `UnsafeInMemorySorter` receives less memory than it requested, and causes a `SparkOutOfMemoryError` to be thrown, which causes the current task to fail. With the changes in the PR the following will happen instead: 1. `UnsafeExternalSorter` runs out of space in its pointer array and attempts to allocate a new large array to replace the old one. 2. `TaskMemoryManager` tries to allocate the memory backing the new large array using `MemoryManager`, but `MemoryManager` is only willing to return most but not all of the memory requested. 3. `TaskMemoryManager` asks `UnsafeExternalSorter` to spill, which causes `UnsafeExternalSorter` to spill the current run to disk, to free its record pages and to reset its `UnsafeInMemorySorter`. 4. `UnsafeInMemorySorter` frees the old pointer array. 5. `TaskMemoryManager` returns control to `UnsafeExternalSorter.growPointerArrayIfNecessary` (either by returning the the new large array or by throwing a `SparkOutOfMemoryError`). 6. `UnsafeExternalSorter` either frees the new large array or it ignores the `SparkOutOfMemoryError` depending on what happened in the previous step. 7. `UnsafeExternalSorter` successfully allocates a new small pointer array and operation continues as normal. No Tests were added in `UnsafeExternalSorterSuite` and `UnsafeInMemorySorterSuite`. Closes apache#29785 from tomvanbussel/SPARK-32901. Authored-by: Tom van Bussel <tom.vanbussel@databricks.com> Signed-off-by: herman <herman@databricks.com>

…ExternalSorter Backport of #29785 to Spark 2.4 ### What changes were proposed in this pull request? This PR changes `UnsafeExternalSorter` to no longer allocate any memory while spilling. In particular it removes the allocation of a new pointer array in `UnsafeInMemorySorter`. Instead the new pointer array is allocated whenever the next record is inserted into the sorter. ### Why are the changes needed? Without this change the `UnsafeExternalSorter` could throw an OOM while spilling. The following sequence of events would have triggered an OOM: 1. `UnsafeExternalSorter` runs out of space in its pointer array and attempts to allocate a new large array to replace the old one. 2. `TaskMemoryManager` tries to allocate the memory backing the new large array using `MemoryManager`, but `MemoryManager` is only willing to return most but not all of the memory requested. 3. `TaskMemoryManager` asks `UnsafeExternalSorter` to spill, which causes `UnsafeExternalSorter` to spill the current run to disk, to free its record pages and to reset its `UnsafeInMemorySorter`. 4. `UnsafeInMemorySorter` frees the old pointer array, and tries to allocate a new small pointer array. 5. `TaskMemoryManager` tries to allocate the memory backing the small array using `MemoryManager`, but `MemoryManager` is unwilling to give it any memory, as the `TaskMemoryManager` is still holding on to the memory it got for the new large array. 6. `TaskMemoryManager` again asks `UnsafeExternalSorter` to spill, but this time there is nothing to spill. 7. `UnsafeInMemorySorter` receives less memory than it requested, and causes a `SparkOutOfMemoryError` to be thrown, which causes the current task to fail. With the changes in the PR the following will happen instead: 1. `UnsafeExternalSorter` runs out of space in its pointer array and attempts to allocate a new large array to replace the old one. 2. `TaskMemoryManager` tries to allocate the memory backing the new large array using `MemoryManager`, but `MemoryManager` is only willing to return most but not all of the memory requested. 3. `TaskMemoryManager` asks `UnsafeExternalSorter` to spill, which causes `UnsafeExternalSorter` to spill the current run to disk, to free its record pages and to reset its `UnsafeInMemorySorter`. 4. `UnsafeInMemorySorter` frees the old pointer array. 5. `TaskMemoryManager` returns control to `UnsafeExternalSorter.growPointerArrayIfNecessary` (either by returning the the new large array or by throwing a `SparkOutOfMemoryError`). 6. `UnsafeExternalSorter` either frees the new large array or it ignores the `SparkOutOfMemoryError` depending on what happened in the previous step. 7. `UnsafeExternalSorter` successfully allocates a new small pointer array and operation continues as normal. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Tests were added in `UnsafeExternalSorterSuite` and `UnsafeInMemorySorterSuite`. Closes #29910 from tomvanbussel/backport-SPARK-32901. Authored-by: Tom van Bussel <tom.vanbussel@databricks.com> Signed-off-by: herman <herman@databricks.com>

…nalSorter ### What changes were proposed in this pull request? This PR changes `UnsafeExternalSorter` to no longer allocate any memory while spilling. In particular it removes the allocation of a new pointer array in `UnsafeInMemorySorter`. Instead the new pointer array is allocated whenever the next record is inserted into the sorter. ### Why are the changes needed? Without this change the `UnsafeExternalSorter` could throw an OOM while spilling. The following sequence of events would have triggered an OOM: 1. `UnsafeExternalSorter` runs out of space in its pointer array and attempts to allocate a new large array to replace the old one. 2. `TaskMemoryManager` tries to allocate the memory backing the new large array using `MemoryManager`, but `MemoryManager` is only willing to return most but not all of the memory requested. 3. `TaskMemoryManager` asks `UnsafeExternalSorter` to spill, which causes `UnsafeExternalSorter` to spill the current run to disk, to free its record pages and to reset its `UnsafeInMemorySorter`. 4. `UnsafeInMemorySorter` frees the old pointer array, and tries to allocate a new small pointer array. 5. `TaskMemoryManager` tries to allocate the memory backing the small array using `MemoryManager`, but `MemoryManager` is unwilling to give it any memory, as the `TaskMemoryManager` is still holding on to the memory it got for the new large array. 6. `TaskMemoryManager` again asks `UnsafeExternalSorter` to spill, but this time there is nothing to spill. 7. `UnsafeInMemorySorter` receives less memory than it requested, and causes a `SparkOutOfMemoryError` to be thrown, which causes the current task to fail. With the changes in the PR the following will happen instead: 1. `UnsafeExternalSorter` runs out of space in its pointer array and attempts to allocate a new large array to replace the old one. 2. `TaskMemoryManager` tries to allocate the memory backing the new large array using `MemoryManager`, but `MemoryManager` is only willing to return most but not all of the memory requested. 3. `TaskMemoryManager` asks `UnsafeExternalSorter` to spill, which causes `UnsafeExternalSorter` to spill the current run to disk, to free its record pages and to reset its `UnsafeInMemorySorter`. 4. `UnsafeInMemorySorter` frees the old pointer array. 5. `TaskMemoryManager` returns control to `UnsafeExternalSorter.growPointerArrayIfNecessary` (either by returning the the new large array or by throwing a `SparkOutOfMemoryError`). 6. `UnsafeExternalSorter` either frees the new large array or it ignores the `SparkOutOfMemoryError` depending on what happened in the previous step. 7. `UnsafeExternalSorter` successfully allocates a new small pointer array and operation continues as normal. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Tests were added in `UnsafeExternalSorterSuite` and `UnsafeInMemorySorterSuite`. Closes apache#29785 from tomvanbussel/SPARK-32901. Authored-by: Tom van Bussel <tom.vanbussel@databricks.com> Signed-off-by: herman <herman@databricks.com> (cherry picked from commit f167002) Signed-off-by: herman <herman@databricks.com>

…ExternalSorter Backport of apache#29785 to Spark 2.4 This PR changes `UnsafeExternalSorter` to no longer allocate any memory while spilling. In particular it removes the allocation of a new pointer array in `UnsafeInMemorySorter`. Instead the new pointer array is allocated whenever the next record is inserted into the sorter. Without this change the `UnsafeExternalSorter` could throw an OOM while spilling. The following sequence of events would have triggered an OOM: 1. `UnsafeExternalSorter` runs out of space in its pointer array and attempts to allocate a new large array to replace the old one. 2. `TaskMemoryManager` tries to allocate the memory backing the new large array using `MemoryManager`, but `MemoryManager` is only willing to return most but not all of the memory requested. 3. `TaskMemoryManager` asks `UnsafeExternalSorter` to spill, which causes `UnsafeExternalSorter` to spill the current run to disk, to free its record pages and to reset its `UnsafeInMemorySorter`. 4. `UnsafeInMemorySorter` frees the old pointer array, and tries to allocate a new small pointer array. 5. `TaskMemoryManager` tries to allocate the memory backing the small array using `MemoryManager`, but `MemoryManager` is unwilling to give it any memory, as the `TaskMemoryManager` is still holding on to the memory it got for the new large array. 6. `TaskMemoryManager` again asks `UnsafeExternalSorter` to spill, but this time there is nothing to spill. 7. `UnsafeInMemorySorter` receives less memory than it requested, and causes a `SparkOutOfMemoryError` to be thrown, which causes the current task to fail. With the changes in the PR the following will happen instead: 1. `UnsafeExternalSorter` runs out of space in its pointer array and attempts to allocate a new large array to replace the old one. 2. `TaskMemoryManager` tries to allocate the memory backing the new large array using `MemoryManager`, but `MemoryManager` is only willing to return most but not all of the memory requested. 3. `TaskMemoryManager` asks `UnsafeExternalSorter` to spill, which causes `UnsafeExternalSorter` to spill the current run to disk, to free its record pages and to reset its `UnsafeInMemorySorter`. 4. `UnsafeInMemorySorter` frees the old pointer array. 5. `TaskMemoryManager` returns control to `UnsafeExternalSorter.growPointerArrayIfNecessary` (either by returning the the new large array or by throwing a `SparkOutOfMemoryError`). 6. `UnsafeExternalSorter` either frees the new large array or it ignores the `SparkOutOfMemoryError` depending on what happened in the previous step. 7. `UnsafeExternalSorter` successfully allocates a new small pointer array and operation continues as normal. No Tests were added in `UnsafeExternalSorterSuite` and `UnsafeInMemorySorterSuite`. Closes apache#29910 from tomvanbussel/backport-SPARK-32901. Authored-by: Tom van Bussel <tom.vanbussel@databricks.com> Signed-off-by: herman <herman@databricks.com> Ref: LIHADOOP-58062 RB=2551503 BUG=LIHADOOP-58062 G=spark-reviewers R=mmuralid A=mmuralid

[SPARK-32901][CORE] Do not allocate memory while spilling UnsafeExter…

cb42b93

…nalSorter

probot-autolabeler bot added the CORE label Sep 17, 2020

Remove unused imports

c144104

hvanhovell reviewed Sep 17, 2020

View reviewed changes

core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java Show resolved Hide resolved

hvanhovell reviewed Sep 17, 2020

View reviewed changes

core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java Outdated Show resolved Hide resolved

Address feedback

a74aa0a

hvanhovell reviewed Sep 17, 2020

View reviewed changes

MaxGekk reviewed Sep 17, 2020

View reviewed changes

tomvanbussel requested a review from hvanhovell September 18, 2020 11:51

cloud-fan reviewed Sep 21, 2020

View reviewed changes

core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java Outdated Show resolved Hide resolved

hvanhovell approved these changes Sep 24, 2020

View reviewed changes

cloud-fan reviewed Sep 25, 2020

View reviewed changes

core/src/test/scala/org/apache/spark/memory/TestMemoryManager.scala Show resolved Hide resolved

Clarify comment

fc510eb

hvanhovell closed this in f167002 Sep 29, 2020

tomvanbussel mentioned this pull request Sep 30, 2020

[SPARK-32901][CORE][2.4] Do not allocate memory while spilling UnsafeExternalSorter #29910

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-32901][CORE] Do not allocate memory while spilling UnsafeExternalSorter #29785

[SPARK-32901][CORE] Do not allocate memory while spilling UnsafeExternalSorter #29785

tomvanbussel commented Sep 17, 2020 •

edited

Loading

tomvanbussel commented Sep 17, 2020

cloud-fan commented Sep 17, 2020

cloud-fan commented Sep 17, 2020

tomvanbussel commented Sep 17, 2020

hvanhovell Sep 17, 2020

tomvanbussel Sep 17, 2020

MaxGekk left a comment

tomvanbussel commented Sep 17, 2020

dongjoon-hyun commented Sep 20, 2020

SparkQA commented Sep 20, 2020

cloud-fan commented Sep 21, 2020

tomvanbussel commented Sep 21, 2020

Ngone51 commented Sep 22, 2020

tomvanbussel commented Sep 22, 2020

Ngone51 commented Sep 23, 2020

tomvanbussel commented Sep 24, 2020

hvanhovell left a comment

cloud-fan commented Sep 25, 2020 •

edited

Loading

tomvanbussel commented Sep 25, 2020

hvanhovell commented Sep 29, 2020 •

edited

Loading

[SPARK-32901][CORE] Do not allocate memory while spilling UnsafeExternalSorter #29785

[SPARK-32901][CORE] Do not allocate memory while spilling UnsafeExternalSorter #29785

Conversation

tomvanbussel commented Sep 17, 2020 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

tomvanbussel commented Sep 17, 2020

cloud-fan commented Sep 17, 2020

cloud-fan commented Sep 17, 2020

tomvanbussel commented Sep 17, 2020

hvanhovell Sep 17, 2020

Choose a reason for hiding this comment

tomvanbussel Sep 17, 2020

Choose a reason for hiding this comment

MaxGekk left a comment

Choose a reason for hiding this comment

tomvanbussel commented Sep 17, 2020

dongjoon-hyun commented Sep 20, 2020

SparkQA commented Sep 20, 2020

cloud-fan commented Sep 21, 2020

tomvanbussel commented Sep 21, 2020

Ngone51 commented Sep 22, 2020

tomvanbussel commented Sep 22, 2020

Ngone51 commented Sep 23, 2020

tomvanbussel commented Sep 24, 2020

hvanhovell left a comment

Choose a reason for hiding this comment

cloud-fan commented Sep 25, 2020 • edited Loading

tomvanbussel commented Sep 25, 2020

hvanhovell commented Sep 29, 2020 • edited Loading

tomvanbussel commented Sep 17, 2020 •

edited

Loading

cloud-fan commented Sep 25, 2020 •

edited

Loading

hvanhovell commented Sep 29, 2020 •

edited

Loading