[SPARK-32161] [PYTHON] Removing JVM logs from SparkUpgradeException#34275
[SPARK-32161] [PYTHON] Removing JVM logs from SparkUpgradeException#34275pralabhkumar wants to merge 2 commits intoapache:masterfrom
Conversation
|
ok to test |
|
|
||
| try: | ||
| import xmlrunner # type: ignore[import] | ||
|
|
python/pyspark/sql/utils.py
Outdated
| raise converted from None | ||
| else: | ||
| raise | ||
|
|
| except AnalysisException as e: | ||
| self.assertRegex(str(e), "Column '`中文字段`' does not exist") | ||
|
|
||
| def test_spark_upgrade_exception(self): |
There was a problem hiding this comment.
Let's add a short comment like # SPARK-XXXXX: .... see also pull request in https://spark.apache.org/contributing.html
| df = self.spark.createDataFrame([("2014-31-12",)], ['date_str']) | ||
| df2 = df.select('date_str', | ||
| to_date(from_unixtime(unix_timestamp('date_str', 'yyyy-dd-aa')))) | ||
| self.assertRaises(SparkUpgradeException, lambda: df2.show(1, False)) |
There was a problem hiding this comment.
| self.assertRaises(SparkUpgradeException, lambda: df2.show(1, False)) | |
| self.assertRaises(SparkUpgradeException, df2.collect) |
| self.assertRegex(str(e), "Column '`中文字段`' does not exist") | ||
|
|
||
| def test_spark_upgrade_exception(self): | ||
| from pyspark.sql.functions import to_date, unix_timestamp, from_unixtime |
There was a problem hiding this comment.
feel free to import this on the top of this file
|
Test build #144219 has finished for PR 34275 at commit
|
|
Kubernetes integration test starting |
|
Kubernetes integration test status failure |
|
Test build #144239 has finished for PR 34275 at commit
|
|
Kubernetes integration test starting |
|
Kubernetes integration test status failure |
|
Merged to master. |
|
Thx @HyukjinKwon for reviewing and merging it . |
4 participants
What changes were proposed in this pull request?
Hide JVM traceback for SparkUpgradeException
Following PR will result into
Why are the changes needed?
This change will remove JVM traceback for pyspark in SparkUpgradeException. This will help to have stack trace more pythonic way
Does this PR introduce any user-facing change?
Yes user will be able to see only python stacktrace
How was this patch tested?
unit tests