Ubuntu 20.04.1 - multiple processes launched and memory full using docker container with python and ROS1

[goupil35000]

• VSCode Version: 1.51.1
• OS Version: Ubuntu 20.04.1
• Docker version 19.03.13, build 4484c46d9d

Docker container: python / ROS 1 noetic full.

Steps to Reproduce (2. is independant of problem presented in 1. I had the same problem before when the problem 1. was not occuring).

1. Open a docker container. Select python 3.8.5 in menu at the top. But nothing changes (no reload of window with python written in the bottom). Add workspace selecting catkin_ws for ROS. At this time window is reloaded and i can see python and ROS versions written at the bottom. This problem is new these last days.
2. Launch a launch file for ros (roslaunch ...). If roscore was launched before, no problem. If not, multiple process launched, and memory increase to full use -- in my case 32GB -- the system becomes irresponsive. It's hard to be able to do something after. But after clicking multiple times to close vscode, I was able to get the normal use of Ubuntu and the memory use decrases. After that, docker is not working: I have no answer with docker ps -a. To solve this, I have to destroy docker daemon and restart it.

Does this issue occur when all extensions are disabled?: I need to have extensions opened because the problem occurs for a docker container.

---

We don't know if vscode is the problem or an extension. But it seems that the problems occur after the last updates of vscode (but docker extension also changes theses last days). The problem was tested on 4 differents hardware with Ubuntu 20.04.1 and tested by 2 different users. The problem does not occur in Windows (insider -- wsl2 with gpu support).

---

Full configuration (tested on AMD 3700X/32GB and I9 9900K/32GB/RTX2070).
Docker 1.8.0 (also tested with 1.7.0 version)
Jupyter v2020.11.3728319
Python v2020.11.3715265
Pylance v2020.11.2
Remote Containers v0.148.1
Remote SSH v0.56.0
Remote - SSH: Editing Configuration Files v0.56.0
Remote - WSL v0.51.4
Remote Development v0.20.0
ROS 0.6.4 [Edit]

-------------- [Edit] -------------
Line of the Processes launched:
/home/util1/.vscode-server/bin/e5a624b788d92b8d34d1392e4c4d9789406efe8f/node -e ????const net = require('net'); ????process.stdin.pause(); ????const client = net.createConnection({ port: 34751 }, () => { ?????c

[vkachurka]

I would like to double on this bug: it is still happening. It seems this is a bug of two extensions clashing, Remote-Containers and ROS. Also, I use no Python, but I develop C++ nodes, so, between me and the topicstarter, this bug is not connected, apparently, neither to Python extensions nor to C++ extensions.

Steps to reproduce:

1. Create a docker container with ROS 1 in it (I created it outside of VS Code).
2. Use Remote-Containers extension to connect to the container, and work happily inside it. At this moment, no bad stuffis happening.
3. Install and enable ROS extension, and be sure that roscore is not running at this moment inside the container. If roscore is running, there will be no trouble; if it is not running - ROS extension, apparently, will start to try to find and/or run roscore, consuming all the available memory and making docker daemon totally non-responsive.

Workaround to those who love to develop ROS nodes in VS Code inside docker containers:
Just start roscore in a teminal in container before launching VS Code and/or enabling ROS extension!

My software versions, on which this stuff does happen:

• Ubuntu 20.04.1 LTS
• Docker 20.10.5 build 55c4c88
• ROS Melodic 1.14.10
• VS Code 1.55.0 c185983a...
• ROS Extension 0.6.7
• Remote-Containers Extension 0.166.1

[ooeygui]

I'm the development lead for the Azure Edge Robotics team at Microsoft- who maintain the VSCode ROS extension. I'm currently tracking this down and will follow up internally with the VSCode team if needed.

[ooeygui]

I confirmed that this occurs without the ROS extension.
I've attached a simple repro with the hello world sample.

Repro Steps:

1. Create a container, and use the docker extension to launch vscode from the container.
2. In the container open vscode on this extension helloworld-sample-port-forward.zip
3. Use the VSCode port forward tool to forward port 11311.
4. Build the extension and launch
5. Ctrl-Shift-P then select hello world

Result - memory growth goes exponential.

Expect - no memory growth

[chrmarti]

@ooeygui Thanks for helping to track this down. What are your OS, VS Code and Remote-Containers versions and which Docker image are you using? (I tried Ubuntu 20.04, VS Code 1.55.2, Remote-Containers 0.166.1 with the node:latest image and could not reproduce the problem.)

Maybe related: #4079

[goupil35000]

Hello @chrmarti,
ooeygui is from Microsoft too.
I posted this problem some months ago and also in ms-iot/vscode-ros#393. You will find there many details of the problem.

I can confirm that the problem occurs on all on my computers with my students. But the configuration is always the same:

• Ubuntu 20.04.2
• Vscode 1.55.2 (but the problem was present with earlier version)
• Extensions in remote container: python, pylance, ROS, Jupyter. All in the last versions.

I had also the problems many months ago with WSL2 on windows 10 insiders. With no solution for this problem, I remove Vscode from Windows 10 and launch Vscode directly inside the container. So sorry, I no longer have the configuration in Windows 10. I think I had Remote-Containers, Remote-ssh, Remote-WSL and Remote Development + python, pylance, ROS, Jupyter

I tried 2 days ago to find the easiest way for Lou to reproduce the problem. See details on ms-iot/vscode-ros#393

Hope this helps

Goupil

[chrmarti]

Please provide the command line you are using to start the container.

[goupil35000]

@chrmarti

I used this command:

docker run -it -e DISPLAY=$DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix -v /tmp/.docker.xauth:/tmp/.docker.xauth -e XAUTHORITY=/tmp/.docker.xauth --net=host ros:noetic-ros-base

[chrmarti]

The host network might trigger this on Linux because the automatic port forwarding sees its own ports open and go into an infinity loop. Could you retry when disabling automatic port forwarding in the user settings and see if that works around the issue?

[ooeygui]

I disabled Autoforward ports, which did not fix this issue - still see the rapid memory growth.

[chrmarti]

I can confirm that the automatic port forwarding does not trigger on its own ports when running with net=host.

What seems to happen is that the port forwarding for the 11311 port starts listening on 11311 on localhost and with net=host that is also the container's port 11311. So when a client connects to 11311, the forwarding connects to itself, resulting in an endless loop of connections.

[goupil35000]

Hello Christof,

Thanks for finding the problem of port 11311. But I'm not sure if it's related to ROS extension of Vscode or related to option net=host.

In practice when I'm using Vscode on the host, I have no problem until I open the catkin_ws directory in Vscode with ROS extension loaded. The memory grows when ROS appearing in the bottom of Vscode window.

If I connect from another terminal using docker exec -it $CONTAINER_NAME /bin/bash, I have no problem.

So it seems that for me the problem only occurs in that conditions:

• net=host in the docker run line;
• attach visual code to container;
• a catkin_ws directory is defined in container;
• and ROS extension recognize the catkin_ws directory and put ROS (or ROS1.noetic) in the bottom of VScode window.

Is there a solution for this problem ?

Goupil

[chrmarti]

@goupil35000 We need to track this down further. I think the 11311 port repro might give us the right lead. Could you try the following:

• Disable automatic port forwarding:
  
• Disable restoring forwarded ports:
  
• When attached use F1 > Remote-Containers: Open Container Configuration File to look at the config file and remove the "forwardPorts" property (if there is one).
• Reload with F1 > Developer: Reload Window and check in the Ports view that there no forwarded ports.
• Open the log with F1 > Remote-Containers: Show Log.
• Reproduce the issue and attach the log here.

[goupil35000]

@chrmarti

I only find Remote-Containers: Open Attached Container Configuration File, but I think it's the same. I opened the image for ros:noetic-ros-base. I remove in the ros:noetic-ros-base.json
"forwardPorts": [
42161,
41363,
41247,
35721,
35283,
38305,
44933,
39527,
42119,
43719,
58185,
11311,
11345,
36377,
35035,
46363,
39457
]

I disable automatic port forwarding and disable restoring forwarded ports.
I reloaded with F1 > Developer: Reload Window. I have no port tab in the bottom, but I'm outside a container at this time.

I then "Attach Visual Code" to my running container.
In the bottom line I have ROS1.noetic and no increase in memory use. I have the PORTS tab, but nothing in it.

Here the result in this Terminal after typing: F1 > Remote-Containers: Show Log

[3 ms] Remote-Containers 0.166.1 in VS Code 1.55.2 (3c4e3df9e89829dce27b7b5c24508306b151f30d).
[44 ms] Start: Resolving Remote
[46 ms] Setting up container: /magical_cohen
[47 ms] Start: Run: docker inspect --type container /magical_cohen
[77 ms] Start: Check Docker is running
[78 ms] Start: Run: docker version --format {{.Server.APIVersion}}
[111 ms] Server API version: 1.41
[111 ms] Start: Inspecting container
[112 ms] Start: Run: docker inspect --type container 09e2088020abd258398bcddd96c7617df5d4f8473ca8f31dbcefdd073f11f8e7
[144 ms] Start: Run in container: /bin/sh
[147 ms] Start: Run in container: uname -m
[244 ms] x86_64
[244 ms]
[244 ms] Start: Run in container: (cat /etc/os-release || cat /usr/lib/os-release) 2>/dev/null
[248 ms] NAME="Ubuntu"
VERSION="20.04.2 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.2 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
[248 ms]
[249 ms] Start: Run in container: cat /etc/passwd
[251 ms] Start: Run in container: test -d /root/.vscode-server
[251 ms]
[252 ms]
[252 ms] Start: Run in container: set -o noclobber ; mkdir -p '/root/.vscode-server/data/Machine' && { > '/root/.vscode-server/data/Machine/.writeMachineSettingsMarker' ; } 2> /dev/null
[254 ms]
[254 ms]
[254 ms] Exit code 2
[254 ms] Start: Run in container: cat /root/.vscode-server/data/Machine/settings.json
[255 ms]
[255 ms] cat: /root/.vscode-server/data/Machine/settings.json: No such file or directory
[255 ms] Exit code 1
[256 ms] Start: Run in container: test -d /root/.vscode-server/bin/3c4e3df9e89829dce27b7b5c24508306b151f30d
[256 ms]
[256 ms]
[256 ms] Start: Launching Remote-Containers helper.
[257 ms] Start: Run: gpgconf --list-dir agent-extra-socket
[260 ms] /run/user/1000/gnupg/S.gpg-agent.extra
[260 ms]
[260 ms] Start: Run in container: gpgconf --list-dir agent-socket
[262 ms] /root/.gnupg/S.gpg-agent
[262 ms]
[262 ms] Start: Run in container: gpgconf --list-dir homedir
[264 ms] /root/.gnupg
[264 ms]
[264 ms] Start: Run in container: ls '/root/.gnupg/private-keys-v1.d' 2>/dev/null
[265 ms]
[265 ms]
[265 ms] Exit code 2
[265 ms] Start: Run in container: mkdir -p -m 700 '/root/.gnupg'
[267 ms]
[267 ms]
[267 ms] Start: Run: gpgconf --list-dir homedir
[270 ms] /home/util1/.gnupg
[270 ms]
[270 ms] Start: Run in container: gpgconf --list-dir homedir
[272 ms] /root/.gnupg
[272 ms]
[272 ms] Start: Run in container: cat <<'EOF-/tmp/vscode-remote-containers-6bbbe32cfe857b94a58405c512cc52aaf05cdbe9.js' >/tmp/vscode-remote-containers-6bbbe32cfe857b94a58405c512cc52aaf05cdbe9.js
[273 ms]
[273 ms]
[273 ms] Start: Run in container: # Test for /root/.ssh/known_hosts and ssh
[274 ms] /root/.ssh/known_hosts exists
[274 ms]
[274 ms] Exit code 1
[274 ms] Start: Run in container: # Test for /root/.gnupg/pubring.kbx and gpg
[275 ms] /root/.gnupg/pubring.kbx exists
[275 ms]
[275 ms] Exit code 1
[275 ms] Start: Run in container: cat <<'EOF-/tmp/vscode-remote-containers-server-6bbbe32cfe857b94a58405c512cc52aaf05cdbe9.js' >/tmp/vscode-remote-containers-server-6bbbe32cfe857b94a58405c512cc52aaf05cdbe9.js
[276 ms]
[277 ms]
[277 ms] Start: Run in container: command -v git >/dev/null 2>&1 && git config --system credential.helper '!f() { /root/.vscode-server/bin/3c4e3df9e89829dce27b7b5c24508306b151f30d/node /tmp/vscode-remote-containers-6bbbe32cfe857b94a58405c512cc52aaf05cdbe9.js $*; }; f' || true
[277 ms] Start: Run in container: /root/.vscode-server/bin/3c4e3df9e89829dce27b7b5c24508306b151f30d/node /tmp/vscode-remote-containers-server-6bbbe32cfe857b94a58405c512cc52aaf05cdbe9.js
[283 ms]
[283 ms]
[283 ms] Start: Run in container: # Test for /root/.gnupg/trustdb.gpg and gpg
[284 ms] Start: Preparing Extensions
[284 ms] /root/.gnupg/trustdb.gpg exists
[284 ms]
[284 ms] Exit code 1
[284 ms] Start: Run in container: set -o noclobber ; mkdir -p '/root/.vscode-server/data/Machine' && { > '/root/.vscode-server/data/Machine/.installExtensionsMarker' ; } 2> /dev/null
[284 ms] Start: Run: gpg-connect-agent updatestartuptty /bye
[287 ms]
[287 ms]
[287 ms] Exit code 2
[288 ms] Start: Run in container: test -d /root/.vscode-server/extensionsCache && ls /root/.vscode-server/extensionsCache || true
[290 ms] ms-iot.vscode-ros-0.6.7
ms-python.python-2021.4.765268190
ms-python.vscode-pylance-2021.4.2
ms-toolsai.jupyter-2021.5.745244803
ms-vscode.cpptools-1.3.1
[290 ms]
[292 ms] Start: Run in container: for pid in cd /proc && ls -d [0-9]*; do { echo $pid ; readlink /proc/$pid/cwd ; readlink /proc/$pid/ns/mnt ; cat /proc/$pid/stat | tr "
[314 ms] Start: Starting VS Code Server
[315 ms] Start: Run in container: /root/.vscode-server/bin/3c4e3df9e89829dce27b7b5c24508306b151f30d/server.sh --force-disable-user-env --use-host-proxy --port 0 --extensions-download-dir /root/.vscode-server/extensionsCache --start-server
[443 ms]

• Visual Studio Code Server
• Reminder: You may only use this software with Visual Studio family products,
• as described in the license https://aka.ms/vscode-remote/license

Extension host agent listening on 41379

[443 ms] Start: Run in container: echo 41379 >/root/.vscode-server/data/Machine/.devport
[444 ms]
[444 ms]
[445 ms] Forwarding local port 33963 to container port 41379
[446 ms] Start: Run in container: /bin/sh -c # Watch installed extensions
[447 ms] Start: Run in container: /bin/sh -c # Watch machine settings
[457 ms] Start: Run in container: command -v git >/dev/null 2>&1 && git config --global credential.helper '!f() { /root/.vscode-server/bin/3c4e3df9e89829dce27b7b5c24508306b151f30d/node /tmp/vscode-remote-containers-6bbbe32cfe857b94a58405c512cc52aaf05cdbe9.js $*; }; f' || true
[460 ms]
[461 ms]
[481 ms] Forwarding connection from port 51218 on local port 33963 to container port 41379
[481 ms] Start: Run in container: /root/.vscode-server/bin/3c4e3df9e89829dce27b7b5c24508306b151f30d/node -e
[550 ms] [13:18:13] Extension host agent started.
[749 ms] Forwarding connection from port 51230 on local port 33963 to container port 41379
[749 ms] Start: Run in container: /root/.vscode-server/bin/3c4e3df9e89829dce27b7b5c24508306b151f30d/node -e
[752 ms] [13:18:13] [::ffff:127.0.0.1][b1c8b213][ManagementConnection] New connection established.
[774 ms] [IPC Library: Pty Host] INFO Listing 0 persistent terminals, 0 total terminals
[1245 ms] [13:18:14] [::ffff:127.0.0.1][bf0caf93][ExtensionHostConnection] New connection established.
[1251 ms] [13:18:14] [::ffff:127.0.0.1][bf0caf93][ExtensionHostConnection] <207> Launched Extension Host Process.
[1520 ms] Start: Run in container: cat /proc/218/environ
[60735 ms] Start: Run in container: /bin/sh -c # Watch installed extensions
[60865 ms] Start: Run in container: /bin/sh -c # Watch machine settings
[121053 ms] Start: Run in container: /bin/sh -c # Watch installed extensions
[121158 ms] Start: Run in container: /bin/sh -c # Watch machine settings
[181348 ms] Start: Run in container: /bin/sh -c # Watch installed extensions
[181446 ms] Start: Run in container: /bin/sh -c # Watch machine settings
[241650 ms] Start: Run in container: /bin/sh -c # Watch installed extensions
[241761 ms] Start: Run in container: /bin/sh -c # Watch machine settings
[301963 ms] Start: Run in container: /bin/sh -c # Watch installed extensions
[302054 ms] Start: Run in container: /bin/sh -c # Watch machine settings
[362261 ms] Start: Run in container: /bin/sh -c # Watch installed extensions
[362362 ms] Start: Run in container: /bin/sh -c # Watch machine settings
[422575 ms] Start: Run in container: /bin/sh -c # Watch installed extensions
[422671 ms] Start: Run in container: /bin/sh -c # Watch machine settings
[482875 ms] Start: Run in container: /bin/sh -c # Watch installed extensions
[482978 ms] Start: Run in container: /bin/sh -c # Watch machine settings
[543186 ms] Start: Run in container: /bin/sh -c # Watch installed extensions
[543278 ms] Start: Run in container: /bin/sh -c # Watch machine settings

---

I include below the same before disabling automatic port forwarding and disabling restoring forwarded ports

============ Before unclick Disable automatic port forwarding and Disable restoring forwarded ports========
I'm unable to get the beginning of the lines in terminal, it's too fast.
But you can see below that many forwarding connections are opened and it doesn't stop.

[18377 ms] Forwarding connection from port 37554 on local port 35259 to container port 41379
[18377 ms] Start: Run in container: /root/.vscode-server/bin/3c4e3df9e89829dce27b7b5c24508306b151f30d/node -e
[18394 ms] Forwarding connection from port 37558 on local port 35259 to container port 41379
[18395 ms] Start: Run in container: /root/.vscode-server/bin/3c4e3df9e89829dce27b7b5c24508306b151f30d/node -e
[18424 ms] Forwarding connection from port 37564 on local port 35259 to container port 41379
[18425 ms] Start: Run in container: /root/.vscode-server/bin/3c4e3df9e89829dce27b7b5c24508306b151f30d/node -e
[18453 ms] Forwarding connection from port 37570 on local port 35259 to container port 41379
[18453 ms] Start: Run in container: /root/.vscode-server/bin/3c4e3df9e89829dce27b7b5c24508306b151f30d/node -e
[18488 ms] Forwarding connection from port 37576 on local port 35259 to container port 41379
[18488 ms] Start: Run in container: /root/.vscode-server/bin/3c4e3df9e89829dce27b7b5c24508306b151f30d/node -e
[18522 ms] Forwarding connection from port 37582 on local port 35259 to container port 41379
[18523 ms] Start: Run in container: /root/.vscode-server/bin/3c4e3df9e89829dce27b7b5c24508306b151f30d/node -e
[18571 ms] Forwarding connection from port 37588 on local port 35259 to container port 41379

[ooeygui]

I just want to confirm - 11311 is the default ROS port, but this repros without ROS in the picture.

[chrmarti]

@goupil35000 This doesn’t repro without —net=host, correct? Could you retry with VS Code Insiders and the latest Remote-Containers this comes with and set Remote-Containers Log Level in the user settings to trace? The log output from that might give us additional information on which process is initiating the connection.

[goupil35000]

@chrmarti
Sorry for delay, I was testing the new wslg in Windows with docker integrated in Windows.

---

Yes it seems the problem appears only when you have the option --net=host.

I installed last VS Code Insiders 1.56.0-insider ant latest Remote-Containers 0.171.0
Remote>Containers>Log Level to trace

In containers I installed these extensions: C/C++, Jupyter, Pylance, Python and ROS.
The system is working now with ROS1.noetic in the bottom and I can use ROSlaunch from VScode without any problem.
Small amount of memory used when I open this new window. So it seems that the memory growth is no longer a problem.
Here the logs(Remote Server):

---

[2021-04-25 09:57:36.624] [remoteagent] [trace] Remote configuration data at /root/.vscode-server-insiders
[2021-04-25 09:57:36.624] [remoteagent] [trace] process arguments: {"_":[],"port":"0","start-server":true,"print-startup-performance":false,"print-ip-address":false,"enable-remote-auto-shutdown":false,"remote-auto-shutdown-without-delay":false,"without-browser-env-var":false,"disable-telemetry":false,"extensions-download-dir":"/root/.vscode-server-insiders/extensionsCache","list-extensions":false,"force":false,"show-versions":false,"do-not-sync":false,"force-disable-user-env":true,"use-host-proxy":"","enable-sync":false,"log":"trace","user-data-dir":"/root/.vscode-server-insiders/data","builtin-extensions-dir":"/root/.vscode-server-insiders/bin/0310f02dc5d834bb3a7cc421ea5374aec8d011f1/extensions","extensions-dir":"/root/.vscode-server-insiders/extensions","logsPath":"/root/.vscode-server-insiders/data/logs/20210425T095736"}
[2021-04-25 09:57:36.624] [remoteagent] [info]
*

• Visual Studio Code Server
• Reminder: You may only use this software with Visual Studio family products,
• as described in the license https://aka.ms/vscode-remote/license

[2021-04-25 09:57:36.624] [remoteagent] [info] Extension host agent started.
[2021-04-25 09:57:36.624] [remoteagent] [trace] Started scanning user extensions
[2021-04-25 09:57:36.697] [remoteagent] [trace] Scanned user extensions: 5
[2021-04-25 09:57:36.794] [remoteagent] [info] [::ffff:127.0.0.1][325314c3][ManagementConnection] New connection established.
[2021-04-25 09:57:36.816] [remoteagent] [trace] Started scanning system extensions
[2021-04-25 09:57:36.816] [remoteagent] [trace] Started scanning user extensions
[2021-04-25 09:57:36.817] [remoteagent] [trace] Started scanning user extensions
[2021-04-25 09:57:36.964] [remoteagent] [info] [::ffff:127.0.0.1][159686b6][ExtensionHostConnection] New connection established.
[2021-04-25 09:57:36.968] [remoteagent] [info] [::ffff:127.0.0.1][159686b6][ExtensionHostConnection] <208> Launched Extension Host Process.

---

Tell me if you need other tests.
Thanks for your work.

Goupil

[chrmarti]

Does it occur again when you enable auto port forwarding or does that not affect it?

[goupil35000]

@chrmarti
I confirm that in Windows insiders "Port Forwarding" and "Restore Forwarded Ports" are activated. And that it's working.

When will it be available to everyone ?

Thanks for all.

[chrmarti]

VS Code Insiders and the latest Remote-Containers will become stable releases when VS Code 1.56 ships. That should be first or second week of May.

Closing as fixed. Thanks!