Skip to content

Commit

Permalink
[SPARK-37253][PYTHON] try_simplify_traceback should not fail when `…
Browse files Browse the repository at this point in the history
…tb_frame.f_lineno` is None

### What changes were proposed in this pull request?

This PR aims to handle the corner case when `tb_frame.f_lineno` is `None` in `try_simplify_traceback` which was added by #30309 at Apache Spark 3.1.0.

### Why are the changes needed?

This will handle the following corner case.
```python
Traceback (most recent call last):
  File "/Users/dongjoon/APACHE/spark-merge/python/lib/pyspark.zip/pyspark/worker.py", line 630, in main
    tb = try_simplify_traceback(sys.exc_info()[-1])
  File "/Users/dongjoon/APACHE/spark-merge/python/lib/pyspark.zip/pyspark/util.py", line 217, in try_simplify_traceback
    new_tb = types.TracebackType(
TypeError: 'NoneType' object cannot be interpreted as an integer
```

Python GitHub Repo also has the test case for this corner case.
- https://github.com/python/cpython/blob/main/Lib/test/test_exceptions.py#L2373
```python
None if frame.f_lineno is None else
```

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

N/A

Closes #34530 from dongjoon-hyun/SPARK-37253.

Authored-by: Dongjoon Hyun <dongjoon@apache.org>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
  • Loading branch information
dongjoon-hyun authored and HyukjinKwon committed Nov 9, 2021
1 parent 47ceae4 commit 8ae88d0
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion python/pyspark/util.py
Expand Up @@ -218,7 +218,7 @@ def try_simplify_traceback(tb):
tb_next=tb_next,
tb_frame=cur_tb.tb_frame,
tb_lasti=cur_tb.tb_frame.f_lasti,
tb_lineno=cur_tb.tb_frame.f_lineno)
tb_lineno=cur_tb.tb_frame.f_lineno if cur_tb.tb_frame.f_lineno is not None else -1)
tb_next = new_tb
return new_tb

Expand Down

0 comments on commit 8ae88d0

Please sign in to comment.