Considering improving how python exceptions are presented in reticulate

[skyeturriff]

By default, repl_python() only prints the exception message which may not include much helpful information. The full python traceback is available via a call to reticulate::py_last_error(), however there may be an opportunity to improve how python exceptions are initially presented to reticulate users when they are first raised.

This came in from a support ticket, where an RSW customer noticed that the error output in reticulate was not as detailed as a regular python session.

[t-kalinowski]

More detail: If you have a file stack_trace.py with contents:

def test(a,b):
    print(a+b)

If we run the following in reticulate, we get the error but no further info:

> reticulate::source_python("~/test/stack_trace.py")
> repl_python()
>>> from stack_trace import *
>>> test("1",0)
TypeError: can only concatenate str (not "int") to str

Compared to the more detailed traceback when using python in terminal, which includes the file and function that raised the exception:

>>> from stack_trace import *
>>> test("1",0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/parallels/test/stack_trace.py", line 2, in test
    print(a+b)
TypeError: can only concatenate str (not "int") to str

[jonasViehweger]

I also very much support extending how python exceptions are reported. As is, it is impossible or at the very least very cumbersome for me to debug Python code in RMarkdown or Quarto documents.

[jobrachem]

I'd like to give this a little bump - having a more extensive default Python traceback would be awesome!

[t-kalinowski]

There have been substantial improvements on this front in the development branch already, with more slated to land later this week. It'd be great if you could try it out and give some early feedback!

remotes::install_github("rstudio/reticulate")

[jobrachem]

That's great to hear, thank you so much!

Something seems to not work quite as intended when I try it. I installed reticulate from the main branch. Here's an excerpt from devtools::session_info(), confirming the version.

> devtools::session_info("reticulate")
─ Packages ────────────────────────────────────────────────────────────────────────────────
 package    * version   date (UTC) lib source
 here         1.0.1     2020-12-13 [1] CRAN (R 4.2.0)
 jsonlite     1.8.4     2022-12-06 [1] CRAN (R 4.2.0)
 lattice      0.20-45   2021-09-22 [1] CRAN (R 4.2.0)
 Matrix       1.5-3     2022-11-11 [1] CRAN (R 4.2.0)
 png          0.1-8     2022-11-29 [1] CRAN (R 4.2.0)
 rappdirs     0.3.3     2021-01-31 [1] CRAN (R 4.2.0)
 Rcpp         1.0.10    2023-01-22 [1] CRAN (R 4.2.0)
 RcppTOML     0.2.2     2023-01-29 [1] CRAN (R 4.2.0)
 reticulate   1.28-9000 2023-04-27 [1] Github (rstudio/reticulate@63a5c3e)
 rlang        1.1.0     2023-03-14 [1] CRAN (R 4.2.0)
 rprojroot    2.0.3     2022-04-02 [1] CRAN (R 4.2.0)
 withr        2.5.0     2022-03-03 [1] CRAN (R 4.2.0)

I still only get the error message as the traceback:

> reticulate::repl_python()

>>> import numpy as np
>>> np.ara(1.0)
AttributeError: module 'numpy' has no attribute 'ara'

[jobrachem]

Manually leaving the Python REPL and calling reticulate::py_last_error() leads to a nicer error message now, though :)

> reticulate::repl_python()
>>> import numpy as np
>>> np.ara(1.0)
AttributeError: module 'numpy' has no attribute 'ara'
>>> quit
> reticulate::py_last_error()

── Python Exception Message ─────────────────────────────────────────────────────────────────
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/johannesbrachem/.pyenv/versions/3.10.3/envs/code/lib/python3.10/site-packages/numpy/__init__.py", line 315, in __getattr__
    raise AttributeError("module {!r} has no attribute "
AttributeError: module 'numpy' has no attribute 'ara'

── R Traceback ──────────────────────────────────────────────────────────────────────────────
     ▆
  1. └─reticulate::repl_python()
  2.   ├─base::tryCatch(repl(), interrupt = identity)
  3.   │ └─base (local) tryCatchList(expr, classes, parentenv, handlers)
  4.   │   └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]])
  5.   │     └─base (local) doTryCatch(return(expr), name, parentenv, handler)
  6.   └─reticulate (local) repl()
  7.     ├─base::tryCatch(...)
  8.     │ └─base (local) tryCatchList(expr, classes, parentenv, handlers)
  9.     │   ├─base (local) tryCatchOne(...)
 10.     │   │ └─base (local) doTryCatch(return(expr), name, parentenv, handler)
 11.     │   └─base (local) tryCatchList(expr, names[-nh], parentenv, handlers[-nh])
 12.     │     └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]])
 13.     │       └─base (local) doTryCatch(return(expr), name, parentenv, handler)
 14.     └─reticulate:::py_compile_eval(code, capture = FALSE)
 15.       └─builtins$eval(compiled, globals, locals)
 16.         └─reticulate:::py_call_impl(callable, call_args$unnamed, call_args$named)

[t-kalinowski]

If I'm understanding correctly, in repl_python() you would prefer to see something like this, yes?

>>> import numpy as np
>>> np.ara(1.0)
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/johannesbrachem/.pyenv/versions/3.10.3/envs/code/lib/python3.10/site-packages/numpy/__init__.py", line 315, in __getattr__
    raise AttributeError("module {!r} has no attribute "
AttributeError: module 'numpy' has no attribute 'ara'

[jobrachem]

Yes, that would be great. In most cases, this would mean that I don't have to leave the Python REPL to see the traceback, which would make debugging smoother.

[t-kalinowski]

Fixed on main now, thanks all for bringing this up!

[jobrachem]

Works. I love it. Thank you so much!