Debug Console does not show output from the c/c++ streams

[bersbersbers]

Environment data

• PTVSD version: ??? (VS Code Python extension 2019.8.30787)
• OS and version: Win 10
• Python version (& distribution if applicable, e.g. Anaconda): pure Python 3.7.4
• Using VS Code or Visual Studio: VS Code 1.38.0-insider

Actual behavior

Debug Console shows

Hi!
Bye!

Expected behavior

Debug Console should show

Hi!
2019-08-28 10:18:05.755130: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
Bye!

Steps to reproduce:

Run this code (Bug.py) with "console": "internalConsole":

from tensorflow.keras import layers

print("Hi!")
output = layers.Conv1D(1, 1)(layers.Input([1, 1]))
print("Bye!")

Then, run py Bug.py on the console, seeing the expected output.

This sounds related to #1658, but the "subProcess": true workaround suggested there does not do anything.

[fabioz]

I believe the reason this happens is because the console output just redirects sys.stdout and sys.stderr, whereas this output seems to be written directly from the c stream.

This may be fixed by the refactor branch where ptvsd will launch a process externally and thus can be used to monitor the process as a whole (which is not the case right now as ptvsd is embedded on the target process) -- for cases where the launch is done externally we may have to take a look on redirecting the c standard streams and not only sys.stdout / sys.stderr for this to work.

[bersbersbers]

Here's just one more example:

from tensorflow.keras import layers, models
import tensorflow.keras.backend as K

def loss(y_true, y_pred):
    K.print_tensor('Hello!')
    return 0.0

x = layers.Input(1)
model = models.Model(inputs=x, outputs=x)
model.compile(loss=loss)
print('Hi')
model.fit([1], [1])
print('Bye')

It's not just unimportant information, it's also actual K.print_tensor() calls that get lost.

[int19h]

This is fixed by ptvsd 5, at least for "launch" scenarios where we can redirect the process output from the outside - it now captures everything that is written to stdout or stderr, regardless of how that is done.

[bersbersbers]

Should this be fixed in VS Code release (or insider) builds yet?

The release notes for vscode-python 2020.1.0 (6 January 2020) say:

Updated ptvsd debugger to version v5.0.0a9. (#8930)
https://github.com/microsoft/vscode-python/blob/master/CHANGELOG.md

but I am still seeing this issue with 2020.1.58038 (13 January 2020) on VS Code 1.42.0,
and with 2020.3.62207-dev (Daily Insider) on VS Code 1.43.0-insider (beaf391bc53136912e4f0bbdfb7844ad954247db).

Edit: I noticed that you need to be in an experiment for this, microsoft/vscode-python#9320 (comment), which I seem to be unable to join (in fact, I don't arrive at joining any experiment succesfully). So regarding this issue, I probably have to be patient.

[fabioz]

Instructions for joining the experiment are at: #1706 (comment) (note that you have to get the insiders version, change the settings as described and then restart vscode).

[bersbersbers]

Instructions for joining the experiment are at: #1706 (comment) (note that you have to get the insiders version, change the settings as described and then restart vscode).

Thanks, @fabioz. I had tried using

    "python.insidersChannel": "daily", // also tried "weekly"
    "python.experiments.optInto": [
        "AlwaysDisplayTestExplorer - experiment",
        "DebugAdapterFactory - experiment",
        "PtvsdWheels37 - experiment",
    ],

before and I also tried the VSIX way now, but I am simply not seeing the expected

User belongs to experiment group

and also, I am not seeing the test explorer (I added that experiment because it seemed easier to check if experiment opt-in is working).

I have 2020.3.62207-dev installed now (same as the daily insider before), and python.experiments.optInto is recognized as an existing option. Have restarted VS Code Insiders multiple times, too, and also tried VS Code Release. Just in all cases, the Python output is

> conda --version
> pyenv root
> python3.7 -c "import sys;print(sys.executable)"
...

[fabioz]

Which settings.json are you editing? (it should be the one from user settings)

[bersbersbers]

Which settings.json are you editing? (it should be the one from user settings)

@fabioz correct, yes, I am editing C:\Users\bers\AppData\Roaming\Code - Insiders\User\settings.json.

[fabioz]

One thing different in my case is that those settings are in C:\Users\bers\AppData\Roaming\Code\User\settings.json (without the Code - Insiders path, but I guess that's because I'm using the release version -- can you try getting the Code release version with the python daily and changing the settings on that path to see if it makes a difference?)

@karthiknadig @int19h do you have any suggestions here? (is there some way to diagnose it/get logs for that?)

[bersbersbers]

One thing different in my case is that those settings are in C:\Users\bers\AppData\Roaming\Code\User\settings.json (without the Code - Insiders path, but I guess that's because I'm using the release version -- can you try getting the Code release version with the python daily and changing the settings on that path to see if it makes a difference?)

Done that: VS Code 1.42.0 using 2020.3.62207-dev, editing C:\Users\bers\AppData\Roaming\Code\User\settings.json. python.experiments.optInto is recognized and smart enough to check the items ("PtvsdWheels37 - experimen" is underlined as incorrect, "PtvsdWheels37 - experiment" is not). Restarted VS Code, opened a Python file:

> conda --version
> pyenv root
> python3.7 -c "import sys;print(sys.executable)"
...

[karthiknadig]

@bersbersbers The only time I see this is if experiments are disabled via "python.experiments.enabled": false setting. Can you check if you have something like this set? both in user and workspace settings.

[bersbersbers]

@karthiknadig I have double-checked that I don't have this set anywhere; I have now also set "python.experiments.enabled": true in user settings (where it's accepted) and in workspace settings (where it's grayed out saying that "This setting can be applied only in application user settings"). So no, that's not the reason.

I have also disabled all other extensions and restarted VS Code - same thing.

I noticed that I had Python 3.8.1 and still Python 3.7.5. installed, so I uninstalled Python 3.7.5 - no luck.

[karthiknadig]

@bersbersbers can you file a bug on the extension. This looks like an issue with how experiments are handled. Please share your settings file contents when you file the issue.

[bersbersbers]

@karthiknadig done.

[bersbersbers]

I am able to join experiments again (needed to set telemetry.enableTelemetry to True), and can confirm that the Debug Console now shows the same output as the Terminal. Hence, "console": "internalConsole" is making sense again. Thanks!