[SPARK-28881][PYTHON][TESTS] Add a test to make sure toPandas with Arrow optimization throws an exception per maxResultSize #25594

HyukjinKwon · 2019-08-27T06:25:11Z

What changes were proposed in this pull request?

This PR proposes to add a test case for:

./bin/pyspark --conf spark.driver.maxResultSize=1m
spark.conf.set("spark.sql.execution.arrow.enabled",True)

spark.range(10000000).toPandas()

Empty DataFrame
Columns: [id]
Index: []

which can result in partial results (see #25593 (comment)). This regression was found between Spark 2.3 and Spark 2.4, and accidentally fixed.

Why are the changes needed?

To prevent the same regression in the future.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Test was added.

HyukjinKwon · 2019-08-27T07:18:51Z

retest this please

dongjoon-hyun · 2019-08-27T07:38:41Z

Ur, one of the pythons (python2/python3/pypy) seems to fail due to Pandas.

Traceback (most recent call last):
  File "/home/jenkins/workspace/SparkPullRequestBuilder/python/pyspark/sql/tests/test_arrow.py", line 443, in test_exception_by_max_results
    self.spark.range(0, 10000, 1, 100).toPandas()
AssertionError: "is bigger than" does not match "Pandas >= 0.23.2 must be installed; however, it was not found."

…ception per maxResultSize

dongjoon-hyun · 2019-08-27T07:55:20Z

python/pyspark/sql/tests/test_arrow.py

+
+        # Explicitly enable Arrow and disable fallback.
+        cls.spark.conf.set("spark.sql.execution.arrow.pyspark.enabled", "true")
+        cls.spark.conf.set("spark.sql.execution.arrow.pyspark.fallback.enabled", "false")


Oh, do we need these for test cases? This are opposite to the default values. In the reported JIRA's scenario, we didn't change the default configuration. Did we change the default values for those configurations in 3.0.0?

spark.sql.execution.arrow.enabled now has an alias spark.sql.execution.arrow.pyspark.enabled as of d6632d1

And, spark.sql.execution.arrow.pyspark.fallback.enabled was just set to narrow down test scope. Correct behaviour should be a failure in Arrow optimized code path (although the JIRAs' case, it fails in Arrow optimized path first, and then fails again in non-Arrow optimized path).

I think it's better to have a test that fails with default settings, for branch 2.4.

(the issue SPARK-28881 happened because it passed in Arrow optimized code path, and it returned partial data or empty data).

I am going to not use aliases when I port to branch-2.4 because techincally spark.sql.execution.arrow.enabled and spark.sql.execution.arrow.fallback.enabled are supposed to be removed out in the future.

These configurations just narrow down the testing scope, and test_arrow.py is supposed to test this scope (as can be seen ArrowTests).

SparkQA · 2019-08-27T08:18:48Z

Test build #109799 has finished for PR 25594 at commit 55bed86.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-08-27T08:28:09Z

Test build #109800 has finished for PR 25594 at commit 87840b8.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class MaxResultArrowTests(unittest.TestCase):

HyukjinKwon · 2019-08-27T08:30:03Z

Merged to master.

It will be backported together at #25593

HyukjinKwon · 2019-08-27T08:30:23Z

Thanks, @dongjoon-hyun and @cloud-fan

…row optimization throws an exception per maxResultSize This PR proposes to add a test case for: ```bash ./bin/pyspark --conf spark.driver.maxResultSize=1m spark.conf.set("spark.sql.execution.arrow.enabled",True) ``` ```python spark.range(10000000).toPandas() ``` ``` Empty DataFrame Columns: [id] Index: [] ``` which can result in partial results (see apache#25593 (comment)). This regression was found between Spark 2.3 and Spark 2.4, and accidentally fixed. To prevent the same regression in the future. No. Test was added. Closes apache#25594 from HyukjinKwon/SPARK-28881. Authored-by: HyukjinKwon <gurwls223@apache.org> Signed-off-by: HyukjinKwon <gurwls223@apache.org>

…nection thread to propagate errors ### What changes were proposed in this pull request? This PR proposes to backport #24834 with minimised changes, and the tests added at #25594. #24834 was not backported before because basically it targeted a better exception by propagating the exception from JVM. However, actually this PR fixed another problem accidentally (see #25594 and [SPARK-28881](https://issues.apache.org/jira/browse/SPARK-28881)). This regression seems introduced by #21546. Root cause is that, seems https://github.com/apache/spark/blob/23bed0d3c08e03085d3f0c3a7d457eedd30bd67f/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala#L3370-L3384 `runJob` with `resultHandler` seems able to write partial output. JVM throws an exception but, since the JVM exception is not propagated into Python process, Python process doesn't know if the exception is thrown or not from JVM (it just closes the socket), which results as below: ``` ./bin/pyspark --conf spark.driver.maxResultSize=1m ``` ```python spark.conf.set("spark.sql.execution.arrow.enabled",True) spark.range(10000000).toPandas() ``` ``` Empty DataFrame Columns: [id] Index: [] ``` With this change, it lets Python process catches exceptions from JVM. ### Why are the changes needed? It returns incorrect data. And potentially it returns partial results when an exception happens in JVM sides. This is a regression. The codes work fine in Spark 2.3.3. ### Does this PR introduce any user-facing change? Yes. ``` ./bin/pyspark --conf spark.driver.maxResultSize=1m ``` ```python spark.conf.set("spark.sql.execution.arrow.enabled",True) spark.range(10000000).toPandas() ``` ``` Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/.../pyspark/sql/dataframe.py", line 2122, in toPandas batches = self._collectAsArrow() File "/.../pyspark/sql/dataframe.py", line 2184, in _collectAsArrow jsocket_auth_server.getResult() # Join serving thread and raise any exceptions File "/.../lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__ File "/.../pyspark/sql/utils.py", line 63, in deco return f(*a, **kw) File "/.../lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 328, in get_return_value py4j.protocol.Py4JJavaError: An error occurred while calling o42.getResult. : org.apache.spark.SparkException: Exception thrown in awaitResult: ... Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Total size of serialized results of 1 tasks (6.5 MB) is bigger than spark.driver.maxResultSize (1024.0 KB) ``` now throws an exception as expected. ### How was this patch tested? Manually as described above. unittest added. Closes #25593 from HyukjinKwon/SPARK-27992. Lead-authored-by: HyukjinKwon <gurwls223@apache.org> Co-authored-by: Bryan Cutler <cutlerb@gmail.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

…nection thread to propagate errors ### What changes were proposed in this pull request? This PR proposes to backport apache#24834 with minimised changes, and the tests added at apache#25594. apache#24834 was not backported before because basically it targeted a better exception by propagating the exception from JVM. However, actually this PR fixed another problem accidentally (see apache#25594 and [SPARK-28881](https://issues.apache.org/jira/browse/SPARK-28881)). This regression seems introduced by apache#21546. Root cause is that, seems https://github.com/apache/spark/blob/23bed0d3c08e03085d3f0c3a7d457eedd30bd67f/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala#L3370-L3384 `runJob` with `resultHandler` seems able to write partial output. JVM throws an exception but, since the JVM exception is not propagated into Python process, Python process doesn't know if the exception is thrown or not from JVM (it just closes the socket), which results as below: ``` ./bin/pyspark --conf spark.driver.maxResultSize=1m ``` ```python spark.conf.set("spark.sql.execution.arrow.enabled",True) spark.range(10000000).toPandas() ``` ``` Empty DataFrame Columns: [id] Index: [] ``` With this change, it lets Python process catches exceptions from JVM. ### Why are the changes needed? It returns incorrect data. And potentially it returns partial results when an exception happens in JVM sides. This is a regression. The codes work fine in Spark 2.3.3. ### Does this PR introduce any user-facing change? Yes. ``` ./bin/pyspark --conf spark.driver.maxResultSize=1m ``` ```python spark.conf.set("spark.sql.execution.arrow.enabled",True) spark.range(10000000).toPandas() ``` ``` Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/.../pyspark/sql/dataframe.py", line 2122, in toPandas batches = self._collectAsArrow() File "/.../pyspark/sql/dataframe.py", line 2184, in _collectAsArrow jsocket_auth_server.getResult() # Join serving thread and raise any exceptions File "/.../lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__ File "/.../pyspark/sql/utils.py", line 63, in deco return f(*a, **kw) File "/.../lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 328, in get_return_value py4j.protocol.Py4JJavaError: An error occurred while calling o42.getResult. : org.apache.spark.SparkException: Exception thrown in awaitResult: ... Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Total size of serialized results of 1 tasks (6.5 MB) is bigger than spark.driver.maxResultSize (1024.0 KB) ``` now throws an exception as expected. ### How was this patch tested? Manually as described above. unittest added. Closes apache#25593 from HyukjinKwon/SPARK-27992. Lead-authored-by: HyukjinKwon <gurwls223@apache.org> Co-authored-by: Bryan Cutler <cutlerb@gmail.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

…row optimization throws an exception per maxResultSize ### What changes were proposed in this pull request? This PR proposes to add a test case for: ```bash ./bin/pyspark --conf spark.driver.maxResultSize=1m spark.conf.set("spark.sql.execution.arrow.enabled",True) ``` ```python spark.range(10000000).toPandas() ``` ``` Empty DataFrame Columns: [id] Index: [] ``` which can result in partial results (see apache#25593 (comment)). This regression was found between Spark 2.3 and Spark 2.4, and accidentally fixed. ### Why are the changes needed? To prevent the same regression in the future. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Test was added. Closes apache#25594 from HyukjinKwon/SPARK-28881. Authored-by: HyukjinKwon <gurwls223@apache.org> Signed-off-by: HyukjinKwon <gurwls223@apache.org>

HyukjinKwon changed the title ~~[SPARK-27992][PYTHON][TESTS] Add a test to make sure toPandas with Arrow optimization throws an exception per maxResultSize~~ [SPARK-27881][PYTHON][TESTS] Add a test to make sure toPandas with Arrow optimization throws an exception per maxResultSize Aug 27, 2019

dongjoon-hyun mentioned this pull request Aug 27, 2019

[SPARK-27992][SPARK-28881][PYTHON][2.4] Allow Python to join with connection thread to propagate errors #25593

Closed

HyukjinKwon changed the title ~~[SPARK-27881][PYTHON][TESTS] Add a test to make sure toPandas with Arrow optimization throws an exception per maxResultSize~~ [SPARK-28881][PYTHON][TESTS] Add a test to make sure toPandas with Arrow optimization throws an exception per maxResultSize Aug 27, 2019

This comment has been minimized.

Sign in to view

cloud-fan approved these changes Aug 27, 2019

View reviewed changes

This comment has been minimized.

Sign in to view

Add a test to make sure toPandas with Arrow optimization throws an ex…

87840b8

…ception per maxResultSize

HyukjinKwon force-pushed the SPARK-28881 branch from 2403991 to 87840b8 Compare August 27, 2019 07:48

dongjoon-hyun reviewed Aug 27, 2019

View reviewed changes

dongjoon-hyun approved these changes Aug 27, 2019

View reviewed changes

HyukjinKwon closed this in 00cb2f9 Aug 27, 2019

HyukjinKwon deleted the SPARK-28881 branch March 3, 2020 01:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-28881][PYTHON][TESTS] Add a test to make sure toPandas with Arrow optimization throws an exception per maxResultSize #25594

[SPARK-28881][PYTHON][TESTS] Add a test to make sure toPandas with Arrow optimization throws an exception per maxResultSize #25594

HyukjinKwon commented Aug 27, 2019

This comment has been minimized.

This comment has been minimized.

HyukjinKwon commented Aug 27, 2019

This comment has been minimized.

dongjoon-hyun commented Aug 27, 2019

dongjoon-hyun Aug 27, 2019 •

edited

Loading

HyukjinKwon Aug 27, 2019

cloud-fan Aug 27, 2019

HyukjinKwon Aug 27, 2019

HyukjinKwon Aug 27, 2019

HyukjinKwon Aug 27, 2019

SparkQA commented Aug 27, 2019

SparkQA commented Aug 27, 2019

HyukjinKwon commented Aug 27, 2019

HyukjinKwon commented Aug 27, 2019

[SPARK-28881][PYTHON][TESTS] Add a test to make sure toPandas with Arrow optimization throws an exception per maxResultSize #25594

[SPARK-28881][PYTHON][TESTS] Add a test to make sure toPandas with Arrow optimization throws an exception per maxResultSize #25594

Conversation

HyukjinKwon commented Aug 27, 2019

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

This comment has been minimized.

This comment has been minimized.

HyukjinKwon commented Aug 27, 2019

This comment has been minimized.

dongjoon-hyun commented Aug 27, 2019

dongjoon-hyun Aug 27, 2019 • edited Loading

Choose a reason for hiding this comment

HyukjinKwon Aug 27, 2019

Choose a reason for hiding this comment

cloud-fan Aug 27, 2019

Choose a reason for hiding this comment

HyukjinKwon Aug 27, 2019

Choose a reason for hiding this comment

HyukjinKwon Aug 27, 2019

Choose a reason for hiding this comment

HyukjinKwon Aug 27, 2019

Choose a reason for hiding this comment

SparkQA commented Aug 27, 2019

SparkQA commented Aug 27, 2019

HyukjinKwon commented Aug 27, 2019

HyukjinKwon commented Aug 27, 2019

dongjoon-hyun Aug 27, 2019 •

edited

Loading