Getting syntax error instead of actual message of a thrown exception in Python

[DonJayamanne]

@wmpauli commented on Thu Oct 18 2018

Issue Type: Bug

I keep on getting very strange behavior when I execute lines of code in the the DEBUG CONSOLE: I get a syntax error, instead of the actual message of the exception. Here is a very simple example of a code snippet, that intentionally has an error in it:

import pandas as pd
# this is incorrect, as it should be "data = {'values': [1]}"
data = {'values': 1}
df = pd.DataFrame(data=data)

If I run this with the debugger, I get the expected ValueError in line 4 ("If using all scalar values, you must pass an index"). However, if I run this in the DEBUG CONSOLE, I get a syntax error instead of the message from the exception:

SyntaxError('invalid syntax', ('<string>', 1, 4, 'df = pd.DataFrame(data=data)'))

Any ideas? I get this all the time, not just for pandas. This behavior makes is really hard to debug code, because of the actual error message, I just get this SyntaxError.

Note, I also posted a question here, but to no avail: https://stackoverflow.com/questions/52861388/debug-console-throws-syntax-error-instead-of-showing-exception-message

Extension version: 2018.9.0
VS Code version: Code - Insiders 1.29.0-insider (9e83209ed59a22df3d593735d04cc963396bb89e, 2018-10-18T05:17:36.900Z)
OS version: Windows_NT x64 10.0.17763

System Info

Item	Value
CPUs	Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz (8 x 3492)
GPU Status	2d_canvas: enabled checker_imaging: disabled_off flash_3d: enabled flash_stage3d: enabled flash_stage3d_baseline: enabled gpu_compositing: enabled multiple_raster_threads: enabled_on native_gpu_memory_buffers: disabled_software rasterization: unavailable_software video_decode: enabled video_encode: enabled webgl: enabled webgl2: enabled
Memory (System)	31.92GB (21.97GB free)
Process Argv
Screen Reader	no
VM	0%

---

@d3r3kk commented on Thu Oct 25 2018

Thanks for raising the issue @wmpauli, I've been able to reproduce your issue and have some extra information for the team.

Note that there is a workaround: do not assign the result of the created DataFrame to a variable.

This does what you expect and shows ValueError in the debug console:

pd.DataFrame(data=data)

This doesn't, in the way you describe:

pf = pd.DataFrame(data=data)

---

Environment

My environment:
OS: Windows 10
VSCode Version: 1.29.0-insider
vscode-python: 2018.10.0-alpha (ptvsd-4.1.3)
Python: 3.7
Virtual Environment: venv
Affected packages: pandas 0.23.4

Sample project

The issue arises as described for the attached sample project.
2949_error_message_incorrect.zip

Repro video

Here's a video of the issue in action:

Note: The problem manifests only when you capture the output of DataFrame(data=data) into a variable.

Relevant output (?)

In the interactive terminal, the following output can be seen:

(.venv) C:\dev\github\test\2949_error_message_incorrect> cd 'c:\dev\github\test\2949_error_message_incorrect'; ${env:PYTHONIOENCODING}='UTF-8'; ${env:PYTHONUNBUFFERED}='1'; & 'c:\dev\github\test\2949_error_message_incorrect\.venv\Scripts\python.exe' 'c:\Users\dekeeler\.vscode-insiders\extensions\vscode-python\pythonFiles\experimental\ptvsd_launcher.py' '53049' 'c:\dev\github\test\2949_error_message_incorrect\error_msg_repro.py'
Traceback (most recent call last):
  File "c:\Users\dekeeler\.vscode-insiders\extensions\vscode-python\pythonFiles\experimental\ptvsd\ptvsd\wrapper.py", line 1292, in done
    fut.result()
  File "c:\Users\dekeeler\.vscode-insiders\extensions\vscode-python\pythonFiles\experimental\ptvsd\ptvsd\futures.py", line 40, in result
    reraise(self._exc_info)
  File "c:\Users\dekeeler\.vscode-insiders\extensions\vscode-python\pythonFiles\experimental\ptvsd\ptvsd\reraise3.py", line 8, in reraise
    raise exc_info[1].with_traceback(exc_info[2])
  File "c:\Users\dekeeler\.vscode-insiders\extensions\vscode-python\pythonFiles\experimental\ptvsd\ptvsd\futures.py", line 155, in callback
    x = it.send(fut.result())
  File "c:\Users\dekeeler\.vscode-insiders\extensions\vscode-python\pythonFiles\experimental\ptvsd\ptvsd\wrapper.py", line 2274, in on_completions
    for item in list(xml.comp):
  File "c:\Users\dekeeler\.vscode-insiders\extensions\vscode-python\pythonFiles\experimental\ptvsd\ptvsd\untangle.py", line 87, in __getattr__
    "'%s' has no attribute '%s'" % (self._name, key)
AttributeError: 'xml' has no attribute 'comp'

---

@brettcannon commented on Fri Oct 26 2018

This is expected behaviour as the debug console is using eval() for evaluating your commands, not exec(). In that instance you will get a syntax error as eval() only accepts expressions not statements like assignment.

---

@wmpauli commented on Fri Oct 26 2018

Also, please note that this SyntaxError doesn't occur in previous versions, in which the error message from pandas (or other modules) is correctly printed on the debug console.

E.g.
Issue Type: Bug

same steps

VS Code version: Code 1.23.1 (d0182c3417d225529c6d5ad24b7572815d0de9ac, 2018-05-10T17:11:17.614Z)
OS version: Windows_NT x64 10.0.17134

System Info

Item	Value
CPUs	Intel(R) Core(TM) i7-6600U CPU @ 2.60GHz (4 x 2808)
GPU Status	2d_canvas: enabled flash_3d: enabled flash_stage3d: enabled flash_stage3d_baseline: enabled gpu_compositing: enabled multiple_raster_threads: enabled_on native_gpu_memory_buffers: disabled_software rasterization: disabled_software video_decode: enabled video_encode: enabled vpx_decode: unavailable_software webgl: enabled webgl2: enabled
Memory (System)	15.93GB (6.98GB free)
Process Argv	C:\Program Files\Microsoft VS Code\Code.exe
Screen Reader	no
VM	0%

Extensions (1)

Extension	Author (truncated)	Version
python	ms-	2018.6.0

---

@jreynolds01 commented on Fri Oct 26 2018

@brettcannon Are you saying that the debug console will (and should) always return a SyntaxError when an assignment is used and another error is raised? The repro video showed successful assignment to 'my_data', so it's allowed when it's correct. Overall, the syntax error seems very limited, and it seems similar to saying 'when using the debug console, you can use assignment as long as it's correct, but you can't use assignment to find out anything useful, because the actual error that occurs will be hidden by a syntax error'. Is that documented anywhere else?

---

@brettcannon commented on Mon Oct 29 2018

@jreynolds01 good catch, I'll re-open the issue.

---

@bersbersbers commented on Wed Mar 13 2019

I am surprised that this issue has not attracted more attention, as I find it super annoying and have been waiting for it to be fixed for months (without even digging up this issue as I was certain there must be enough other people annoyed by this that it would be fixed shortly).

Anyway, to summarize:

On the debug console, a=1 works fine, just as a and 1. 1+ returns a correct SyntaxError ("unexpected EOF while parsing"), while a=1+ returns a less detailed SyntaxError ("invalid syntax"). xyz() returns a NameError. a=xyz() returns a SyntaxError ("invalid syntax") - I guess this is the issue here.

In a Python terminal, a=1 works fine, as do a and 1. Both 1+ and a=1+ return "SyntaxError: invalid syntax". xyz() and a=xyz(): NameError.

In a Python terminal, eval('a') and eval('1') work fine. eval('1+') : "SyntaxError: unexpected EOF while parsing". All this is consistent with the behavior of the debug console.

eval('a=1'), eval('a=1+'), eval('a=xyz()'): "SyntaxError: invalid syntax", while eval('xyz()'): NameError. Comparing these to the behavior of the debug console, all except eval('a=1') behave the same - just a=1 seems be go some other route than eval(...) - obviously, as that is working.

I have however been unable to find the piece of code that makes a=1 work, and which fails to send a=xyz() the same way. I begin to think that might be evaluate_expression (defined at https://github.com/Microsoft/ptvsd/blob/60c3480b26e42fc7500c5b4a4f9438a3e1ad0209/src/ptvsd/_vendored/pydevd/_pydevd_bundle/pydevd_vars.py#L243 and used at https://github.com/Microsoft/ptvsd/blob/11ffd749be56fd9c523b0ea0bda792abacc0d7ac/src/ptvsd/_vendored/pydevd/_pydevd_bundle/pydevd_comm.py#L889), is that an indication this issue could be better filed at ptvsd?

---

@brettcannon commented on Wed Mar 13 2019

@DonJayamanne is this on ptvsd or PVSC side of things?

---

@DonJayamanne commented on Wed Mar 13 2019

Ptvsd

[karthiknadig]

I can repro this with 4.2.4. Looks like with current master, evaluations are broken. In master with the above repro i don't see either ValueError or 'SyntaxError`.

/cc @fabioz

[int19h]

The new JSON-based implementation of evaluate seems to be ignoring exceptions for exec but not eval. When expression is coming from debug console (context='repl'), it tries eval first, and falls back to exec if it fails for any reason, so effectively all errors get swallowed. The success path works fine though.