Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Debug console: '\n' not shown as new line for evaluation returning a variablesReference #73845

Closed
yzhang-gh opened this issue May 16, 2019 · 23 comments
Assignees
Labels
debug Debug viewlet, configurations, breakpoints, adapter issues on-release-notes Issue/pull request mentioned in release notes under-discussion Issue is under discussion for relevance, priority, approach
Milestone

Comments

@yzhang-gh
Copy link
Contributor

Issue Type: Bug

Type these code in the debug console

import numpy as np

np.random.rand(5, 5)

image


VS Code version: Code 1.33.1 (51b0b28, 2019-04-11T08:27:14.102Z)
OS version: Windows_NT x64 10.0.18898

System Info
Item Value
CPUs Intel(R) Core(TM) i5-6200U CPU @ 2.30GHz (4 x 2400)
GPU Status 2d_canvas: enabled
checker_imaging: disabled_off
flash_3d: enabled
flash_stage3d: enabled
flash_stage3d_baseline: enabled
gpu_compositing: enabled
multiple_raster_threads: enabled_on
native_gpu_memory_buffers: disabled_software
rasterization: enabled
surface_synchronization: enabled_on
video_decode: enabled
webgl: enabled
webgl2: enabled
Memory (System) 7.90GB (3.58GB free)
Process Argv D:\Workspace\vscode-test
Screen Reader no
VM 0%
Extensions (20)
Extension Author (truncated) Version
spellright ban 3.0.38
vscode-custom-css be5 3.0.4
log-wrapper-for-vscode chr 1.0.2
vsc-material-theme Equ 2.8.2
vsc-material-theme-icons equ 0.12.0
vscode-commands fab 1.2.5
vscode-github-notifications-bell fab 2.1.5
vscode-highlight fab 1.3.4
vscode-open-in-application fab 1.0.5
latex-workshop Jam 6.5.1
vscode-fix-checksums leh 1.1.0
rainbow-csv mec 1.0.0
git-graph mhu 1.5.0
python ms- 2019.4.12954
vscode-sort-json ric 1.13.0
code-settings-sync Sha 3.2.9
dictionary-completion yzh 0.8.2
find-word-at-cursor yzh 0.1.1
markdown-all-in-one yzh 2.3.1
vscode-jupyter yzh 1.2.3
@crock
Copy link

crock commented May 16, 2019

This is not an issue with VSCode. You have a new line inside a Python list. If you want each item in the list on a new line you need to use a loop and print out each line followed by the new line character.

@yzhang-gh
Copy link
Contributor Author

I don't get the point. It is a numpy ndarray (not Python list).

>>> import numpy as np
>>> a = np.arange(15).reshape(3, 5)
>>> a
array([[ 0,  1,  2,  3,  4],
       [ 5,  6,  7,  8,  9],
       [10, 11, 12, 13, 14]])

@yzhang-gh yzhang-gh changed the title Debug console: \n not shown as new line Debug console: \n not shown as new line (NumPy ndarray) May 17, 2019
@roblourens roblourens assigned isidorn and unassigned roblourens May 17, 2019
@roblourens
Copy link
Member

Possibly an issue for the Python extension?

@isidorn isidorn added the *caused-by-extension Issue identified to be caused by an extension label May 17, 2019
@vscodebot
Copy link

vscodebot bot commented May 17, 2019

This issue is caused by an extension, please file it with the repository (or contact) the extension has linked in its overview in VS Code or the marketplace for VS Code. See also our issue reporting guidelines.

Happy Coding!

@fabioz
Copy link
Contributor

fabioz commented May 23, 2019

What seems to be happening is that in the case where an expression is evaluated and returns a variablesReference, the variablesReference is shown the same way it's shown in the variables view (where new lines are changed for a \n representation).

Some more info from the debugger backend:

Given the following Python code (not using numpy to make it simpler):

class Foo(object):

    def __repr__(self):
        return 'aa\nbb\ncc'
    __str__ = __repr__

foo = Foo()
print('add breakpoint here')

Pausing at the breakpoint and sending foo to the repl (by typing foo in the Debug Console) the request sent to the debugger is:

{
    "arguments": {
        "context": "repl",
        "expression": "foo",
        "format": {},
        "frameId": 22
    },
    "command": "evaluate",
    "seq": 1000000016,
    "type": "request"
}

and the response is:

{
    "type": "response", 
    "request_seq": 1000000016, 
    "success": true, 
    "command": "evaluate", 
    "body": {
        "result": "aa\nbb\ncc", 
        "variablesReference": 4, 
        "type": "Foo", 
        "presentationHint": {}
    }, 
    "seq": 42
}

So, you can see that the result has proper new lines and VSCode chose to show it in a single line and not expanded in the debug console (so, there's not much to be done on the extension as it seems to be doing the proper thing).

So, this would be a request (and not a bug since it seems it's deliberately doing the same thing done in the variables view) for VSCode to show new lines expanded in the debug console for a given variablesReference.

As a note, if print(foo) is done, the result is shown expanded -- it'll actually return a variablesReference to None in that case as that's the return of print(foo) but then the representation will be printed as a string afterwards with proper new lines.

@fabioz
Copy link
Contributor

fabioz commented May 23, 2019

A better title for this issue is probably:

Debug console: '\n' not shown as new line for evaluation returning a variablesReference.

-- note that if variablesReference==0 in the return then the Debug Console properly shows the result expanded.

@isidorn isidorn changed the title Debug console: \n not shown as new line (NumPy ndarray) Debug console: '\n' not shown as new line for evaluation returning a variablesReference May 23, 2019
@isidorn isidorn reopened this May 23, 2019
@isidorn isidorn added debug Debug viewlet, configurations, breakpoints, adapter issues debug-console and removed *caused-by-extension Issue identified to be caused by an extension labels May 23, 2019
@isidorn
Copy link
Contributor

isidorn commented May 24, 2019

I pushed a fix for this. Please try it out in monday's vscode insiders and let me know if it behaves better for you. Thanks
https://code.visualstudio.com/insiders/

@isidorn isidorn added this to the May 2019 milestone May 24, 2019
@isidorn isidorn added the bug Issue identified by VS Code Team member as probable bug label May 24, 2019
@jeanp413
Copy link
Contributor

In my opinion, the initial behavior with the newline characters escaped is the correct one because that value is just a (one-liner preferably) preview representation of a composite object that you can click to expand the tree node.
With 1113272 applied now I'm getting this while debugging node which feels odd.
Before:
image
After:
image

Also I could argue that the output from the repl in the terminal doesn't need to match the output in the debug console for example in node:
From terminal

> Array.from({length: 10}, () => Array.from({length: 5}, () => Math.floor(Math.random() * 100)))
[ [ 39, 42, 89, 80, 35 ],
  [ 59, 34, 2, 99, 30 ],
  [ 48, 15, 39, 73, 91 ],
  [ 77, 27, 14, 52, 87 ],
  [ 87, 68, 69, 98, 54 ],
  [ 18, 55, 87, 90, 82 ],
  [ 13, 82, 74, 62, 65 ],
  [ 59, 62, 67, 65, 4 ],
  [ 31, 99, 56, 45, 38 ],
  [ 35, 22, 84, 66, 13 ] ]

Debug Console:
image

So I think this should be fixed in the python extension.
@isidorn @fabioz

@fabioz
Copy link
Contributor

fabioz commented May 25, 2019

As a note, in the Python extension the user could already see inside as a structured object as you're doing in node (so, there's nothing to actually be fixed in the Python extension, that behavior already existed), and if it wasn't a structured object then it'd already appear in multiple lines.

I think it's more about expectations... when using an interactive console in Python users are used to see things printed (so, in the terminal using Python interactively if you wanted to see the \r\n\t of the numpy array you'd do a repr(arr) instead of just arr as just arr would show the new lines expanded)...

Now, this change empowers extensions to do as they please (if the node plugin wants, they can show the actual \n char and not put a new line there by doing a replace('\n', '\\n') if an evaluate request was done for the "repl" context for a structured object), whereas without this patch the behavior can't be customized by the extension.

I personally prefer the current version instead of having to click through to see things in the terminal, but I also agree it's pretty much a matter of personal taste.

@jeanp413
Copy link
Contributor

I agree its a matter of taste, but I have a few observations:

  1. (I think) vscode debug console is modeled after the chrome developer tools console where only primitives get its real value printed, and composite objects are showed as interactive tree hierarchies where tree node values are (one-liner) preview values containing ellipses ... when the preview is too long.
  2. Python, Node, etc interactive console are just a stream of text (doesn't have the ability to collapse or expand a composite variable value) that pretty print the value of variables, this could be really long string for example printing an array of 1000 elements.
  3. We are mixing 1 and 2, showing the pretty-printed value from 2 as the preview value for the tree node which in my opinion doesn't look good.

So I think we should print variable values following 1 or 2 but not mixing both.

@isidorn
Copy link
Contributor

isidorn commented May 28, 2019

Both points are valid. And yes we were inspired by the chrome dev tools console.
However when there is no clear incentive to change behavior I prefer to leave it as it is in the current stable.
Due to that reverting my commit and reopening this issue.

@isidorn isidorn reopened this May 28, 2019
@isidorn isidorn removed this from the May 2019 milestone May 28, 2019
@isidorn isidorn reopened this May 28, 2019
@yzhang-gh
Copy link
Contributor Author

Thank you for all the discussion. (Sorry for being late to reply)

However, it is sad to see this change being reverted. We were in the right direction—Let the extension determine what to show (i.e. the repr function), rather than arbitrarily decide to escape certain characters \n, \t etc.

The original motivation for the escape is "we use \n and \t in the variables view to save on space".

  • For this purpose, this escape behaviour doesn't do its job. For "an array of 1000 elements", escaping \n doesn't make things better. This is even not as effective as direct cutting the long string and appending ....
  • Even worse, this behaviour prevents potential pretty-print with \n. And there is no workaround.
    (Strictly speaking, Python extension can replace \n with \\n in advance as we know what vscode will do next. I believe you agree this adaption is weird)

Next, I will address the specific concerns mentioned above.

Previously Object {test6: "\n\n\n\n"} , now ugly non-inline string

As @fabioz said, this can (IMHO, should) be handled by the node extension (replace('\n', '\\n')).
The old behaviour looks right because vscode happened to do it for the node extension. It is not the duty of vscode.

"argue that the output from the repl in the terminal doesn't need to match the output in the debug console"

Making them matched is not my request. The point is, "Now, this change (do not escape) empowers extensions to do as they please, ..., whereas without this patch the behaviour can't be customized by the extension"
(@fabioz's comment is so clear)

The mixing of Chrome dev console and Python/Node interactive console

It has nothing to do with the Python/Node interactive console. It is all about the repl/evaluate function which is responsible for generating presenting text for non-primitive values. Python also uses ellipses if needed. (shown below a 100x100 NumPy array/matrix)

>>> np.random.rand(100, 100)   
array([[0.99050621, 0.75406745, 0.09582186, ..., 0.29125127, 0.24070286,
        0.35042311],
       [0.50863412, 0.78621844, 0.29135548, ..., 0.70344945, 0.39070587,
        0.95326837],
       [0.04844976, 0.1526166 , 0.21778417, ..., 0.95142566, 0.37265694,
        0.21514417],
       ...,
       [0.16014808, 0.87034416, 0.10492746, ..., 0.74535543, 0.77259506,
        0.87129383],
       [0.07699896, 0.57801489, 0.50589809, ..., 0.55967099, 0.00415744,
        0.95969444],
       [0.80943893, 0.78358082, 0.56240486, ..., 0.97081048, 0.80283494,
        0.81190555]])

The text can actually be array([[...], ..., [...]]) or click to expand or anything if the debugger extension wants.


To summarise, currently, vscode do an extra \n\t escape after the evaluate call.
This doesn't make much sense as (1) it cannot do what is expected and (2) takes over the duty of the debugger extension

@jamesstidard
Copy link

From a practical point of view, it's vary hard to inspect the python data science structures from numpy, xarray, and pandas in VSCode at all at the moment. The structures are too complex for the Debuger's variable inspector to really be that productive to use.

Python encourages for these things to produce formatted strings to be more human readable, which these libraries do. So even if it's decided that it's best that the debug console not allow this, there should be some way to get to this representation in a VSCode debug session. Have even tried to print(my_array) and switching to the terminal tab to try and get some clue as to the shape of these data structures.

This feature: https://github.com/microsoft/vscode-python/issues/5113 may end up solving this, but I don't believe it is currently possible to use this in debug or would be available that soon.

Mainly feel that leaving it to the extension to decide, as mentioned above, so it can be done on a per-language basis would be great. And I think it does in python.

Also, it is a pain point in using VSCode with these data science libraries and debugging data structures, when all you are looking for is if a matrix has values in the diagonal. So if the opposition to this is somewhat notional, I think the user experience here may trump that.

Cheers.

@isidorn isidorn added the *out-of-scope Posted issue is not in scope of VS Code label Oct 9, 2019
@vscodebot
Copy link

vscodebot bot commented Oct 9, 2019

This issue is being closed to keep the number of issues in our inbox on a manageable level, we are closing issues that are not going to be addressed in the foreseeable future: We look at the number of votes the issue has received and the number of duplicate issues filed. More details here. If you disagree and feel that this issue is crucial: We are happy to listen and to reconsider.

If you wonder what we are up to, please see our roadmap and issue reporting guidelines.

Thanks for your understanding and happy coding!

@vscodebot vscodebot bot closed this as completed Oct 9, 2019
@isidorn
Copy link
Contributor

isidorn commented Oct 9, 2019

Clossing as out of scope.
Please provide the feedback to the Python extension. Then they can think about a solution for the issue and they can dicsuss it with us (vscode team) on how to proceed. Thanks for understanding

@teichert
Copy link

Maybe off-topic, but for those of us that want to pause the python debugger in VS Code and then pretty-print numpy arrays (or other expressions containing newlines required for legibility) from the debug console, one available work-around is to print the expression to a text file (which could conveniently be kept open for viewing in a separate editor pane).

For example, to print numpy array variable a to debug.txt with newlines rendered as expected, execute this from the debug console:

with open('debug.txt', 'w') as out: print(a, file=out)

Or append to the end of the file:

with open('debug.txt', 'a') as out: print(a, file=out)

@isidorn
Copy link
Contributor

isidorn commented Jan 23, 2020

Thank you very much for your feedback.

After discussing with @weinand we came to the following conclusion:

The VS Code frontend should not escape newlines. It is up to the debug extension to decide how the data should be presented and the debug extension can decide if a newline shuold be escaped or not.
VS Code is agnostic with regards to the data it is showing, and the extensions are already adding quotes to strings and other to make the data more presentable.

The downside of this approach is that in the Variables view the multine values will from now on only show the first line of the value. However it is up to the debug extensions to change their responses to escape new lines. Due to that we plan to make this change start of next milestone so extensions are given enough time to adopt. Thus reopening.

We do not want to add a setting for this behavior as it does not feel right for the user to control this and adding a capability also requires adoption by extensions (a similar work as our escaping approach requires).

@weinand please add more details or correct if I captured something wrongly from our discussion
fyi @connor4312 @roblourens

@isidorn isidorn reopened this Jan 23, 2020
@isidorn isidorn removed the *out-of-scope Posted issue is not in scope of VS Code label Jan 23, 2020
@isidorn isidorn modified the milestones: Backlog, February 2020 Jan 23, 2020
@yzhang-gh
Copy link
Contributor Author

Thanks for the update.

@isidorn isidorn closed this as completed in f647e12 Feb 5, 2020
@isidorn
Copy link
Contributor

isidorn commented Feb 5, 2020

As announced in my previous comment we have changed that VS Code debugger is no longer escaping whitespaces. Now it is up to the debugger to handle this on their side.
I pushed this early in the milestone so we catch potential corner cases where this new strategy does not work and to give enough time for debuggers to adopt this.

fyi @roblourens @connor4312

@DanTup
Copy link
Contributor

DanTup commented Mar 10, 2020

The downside of this approach is that in the Variables view the multine values will from now on only show the first line of the value. However it is up to the debug extensions to change their responses to escape new lines.

If a debugger decides to "escape" newlines here, won't it break the "Copy value" functionality? The user will then end up with a string with these characters escaped, rather than literal?

@isidorn
Copy link
Contributor

isidorn commented Mar 10, 2020

@DanTup No, I do not think so. Since the copy value functionality invokes the evaluation with the context variables. So the debug adapter can distinguish this on their side.

@DanTup
Copy link
Contributor

DanTup commented Mar 10, 2020

@isidorn but the above text specifically mentions variables:

The downside of this approach is that in the Variables view the multine values will from now on only show the first line of the value

In order to fix this, the DA needs to escape newlines to \n when it gets an evaluateRequest for variables, and that means Copy Value will get the escaped version too. The DA cannot distinguish the copy requests from display (though supporting this has come up before - but been rejected - at #81051 (comment)).

@actboy168
Copy link

I agree that Copy Value should have its own context. A year ago, Copy Value still used null as its context. Although it is not documented, at least it works. I have also requested that it be documented #69768.

@vscodebot vscodebot bot locked and limited conversation to collaborators Mar 21, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
debug Debug viewlet, configurations, breakpoints, adapter issues on-release-notes Issue/pull request mentioned in release notes under-discussion Issue is under discussion for relevance, priority, approach
Projects
None yet
Development

No branches or pull requests

10 participants