Skip to content

Comments

[SPARK-43206] [SS] [CONNECT] [DRAFT] [DO-NOT-REVIEW] StreamingQuery exception() include stack trace#40935

Closed
WweiL wants to merge 1 commit intoapache:masterfrom
WweiL:SPARK-43206-exception-stk-trace
Closed

[SPARK-43206] [SS] [CONNECT] [DRAFT] [DO-NOT-REVIEW] StreamingQuery exception() include stack trace#40935
WweiL wants to merge 1 commit intoapache:masterfrom
WweiL:SPARK-43206-exception-stk-trace

Conversation

@WweiL
Copy link
Contributor

@WweiL WweiL commented Apr 24, 2023

What changes were proposed in this pull request?

Add stack trace to streamingQuery's exception() method. Following a5c8a3c#diff-98baf452f0352c75a39f39351c5f9e656675810b6d4cfd178f1b0bae9751495b

Add to both python client and scala client

Why are the changes needed?

Including stack trace is helpful in debugging

Does this PR introduce any user-facing change?

No

How was this patch tested?

Manually tested:

  1. Python:
JVM stacktrace:
org.apache.spark.sql.streaming.StreamingQueryException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 1 times, most recent failure: Lost task 0.0 in stage 1.0 (TID 1) (ip-10-110-19-234.us-west-2.compute.internal executor driver): org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  File "/home/wei.liu/oss-spark/python/lib/pyspark.zip/pyspark/worker.py", line 850, in main
    process()
  1. Scala: TODO

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant