Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Self-signed certificate error when installing Python support in WSL in spite of custom root certificate correctly installed in WSL #131836

Open
srinathh opened this issue Aug 28, 2021 · 20 comments
Assignees
Labels
bug Issue identified by VS Code Team member as probable bug proxy Issues regarding network proxies

Comments

@srinathh
Copy link

srinathh commented Aug 28, 2021

Does this issue occur when all extensions are disabled?: Yes/No

  • VS Code Version: 1.59.1
  • OS Version: Windows 10.0.19042, WSL 5.10.16.3-microsoft-standard-WSL2
  • Python Language extension : v2021.8.1159798656

My company uses an SSL inspection on company devices and provides a custom root CA certificate. In Windows, this is already pre-installed in the system certificate store and VS Code works fine in Windows where there are no problems installing & using Python language support. For WSL, the certificates & a process to install them in the system certificate store are provided (below) However, when trying to setup Python Language support in WSL I get a self-signed certificate error in spite of following the process to install the custom root certificates in WSL.

Steps to Reproduce:

  1. Copy custom certificates to WSL CA-Certificates folder
sudo cp *.crt /usr/local/share/ca-certificates/
  1. Install
sudo update-ca-certificates
  1. Launch VS Code and connect to WSL project
  2. Try to install Python support in WSL
  3. Get error - self signed certificate in certificate chain
@srinathh
Copy link
Author

srinathh commented Aug 28, 2021

Hello! I've developed a work around.

The core issue here seems to be that the certificate store passed by NodeJS running the back-end server to the OpenSSL library it uses to deal with encryption is not the linux system certificate store updated by update-ca-certificates. However, NodeJS has an environment variable configuration option NODE_EXTRA_CA_CERTS in which we can pass it the custom certificate. We can setup this environment variable in ~/.vscode-server/server-env-setup in WSL.

Something like this works

echo "export NODE_EXTRA_CA_CERTS=<PATH TO SELF SIGNED ROOT>" >>  ~/.vscode-server/server-env-setup

However, ideally the solution should be to have NodeJS use the system certificate store

@aeschli
Copy link
Contributor

aeschli commented Aug 31, 2021

@chrmarti Are you familiar with this?

@chrmarti
Copy link
Contributor

chrmarti commented Sep 1, 2021

We read /etc/ssl/certs/ca-certificates.crt and /etc/ssl/certs/ca-bundle.crt and add these to all requests made with Node.js' https module from the remote extension host.

Maybe the failing request is made somewhere else. Could you attach a screenshot from when the error occurs?

@srinathh
Copy link
Author

srinathh commented Sep 1, 2021

Hello - here's the relevant failure screenshot.
image

I also tried installing the extension manually & got the same error.

Installing extensions...
self signed certificate in certificate chain
Failed Installing Extensions: file:///home/<my home dir>/Downloads/ms-python.python-2021.8.1159798656.vsix
(node:394) UnhandledPromiseRejectionWarning: Canceled: Canceled
    at n (/home/<my home dir>/.vscode-server/bin/3866c3553be8b268c8a7f8c0482c0c0177aa8bfa/out/vs/server/remoteExtensionHostAgent.js:8:1157)
    at /home/<my home dir>/.vscode-server/bin/3866c3553be8b268c8a7f8c0482c0c0177aa8bfa/out/vs/server/remoteExtensionHostAgent.js:68:11867
    at b.fire (/home/<my home dir>/.vscode-server/bin/3866c3553be8b268c8a7f8c0482c0c0177aa8bfa/out/vs/server/remoteExtensionHostAgent.js:60:1712)
    at D.cancel (/home/<my home dir>/.vscode-server/bin/3866c3553be8b268c8a7f8c0482c0c0177aa8bfa/out/vs/server/remoteExtensionHostAgent.js:60:5929)
    at I.cancel (/home/<my home dir>/.vscode-server/bin/3866c3553be8b268c8a7f8c0482c0c0177aa8bfa/out/vs/server/remoteExtensionHostAgent.js:60:6470)
    at Object.cancel (/home/<my home dir>/.vscode-server/bin/3866c3553be8b268c8a7f8c0482c0c0177aa8bfa/out/vs/server/remoteExtensionHostAgent.js:68:11996)
    at Q.cancel (/home/<my home dir>/.vscode-server/bin/3866c3553be8b268c8a7f8c0482c0c0177aa8bfa/out/vs/server/remoteExtensionHostAgent.js:99:17267)
    at /home/<my home dir>/.vscode-server/bin/3866c3553be8b268c8a7f8c0482c0c0177aa8bfa/out/vs/server/remoteExtensionHostAgent.js:99:8033
    at Array.forEach (<anonymous>)
    at k.installExtension (/home/<my home dir>/.vscode-server/bin/3866c3553be8b268c8a7f8c0482c0c0177aa8bfa/out/vs/server/remoteExtensionHostAgent.js:99:8011)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at async d.installVSIX (/home/<my home dir>/.vscode-server/bin/3866c3553be8b268c8a7f8c0482c0c0177aa8bfa/out/vs/server/remoteExtensionHostAgent.js:77:260)
    at async /home/<my home dir>/.vscode-server/bin/3866c3553be8b268c8a7f8c0482c0c0177aa8bfa/out/vs/server/remoteExtensionHostAgent.js:76:40837
    at async Promise.all (index 0)
    at async d.installExtensions (/home/<my home dir>/.vscode-server/bin/3866c3553be8b268c8a7f8c0482c0c0177aa8bfa/out/vs/server/remoteExtensionHostAgent.js:76:40791)
    at async /home/<my home dir>/.vscode-server/bin/3866c3553be8b268c8a7f8c0482c0c0177aa8bfa/out/vs/server/remoteExtensionHostAgent.js:116:110599
    at async k.run (/home/<my home dir>/.vscode-server/bin/3866c3553be8b268c8a7f8c0482c0c0177aa8bfa/out/vs/server/remoteExtensionHostAgent.js:116:110463)
    at async Y (/home/<my home dir>/.vscode-server/bin/3866c3553be8b268c8a7f8c0482c0c0177aa8bfa/out/vs/server/remoteExtensionHostAgent.js:116:112805)

Additionally, based on your comment - I confirmed using grep that the update-ca-certificates does indeed successfully add my custom root certificate to /etc/ssl/certs/ca-certificates.crt & it's there

grep <part of custom root cert text> /etc/ssl/certs/ca-certificates.crt

So net net, I think somewhere in VSCode when extensions are being downloaded, the certs in /etc/ssl/certs/ca-certificates.crt are not actually being passed through. However, if I explicitly set the environment variable NODE_EXTRA_CA_CERTS when starting VSCode in ~/.vscode-server/server-env-setup it does work.

@srinathh
Copy link
Author

srinathh commented Sep 4, 2021

Hi @chrmarti @aeschli - any luck identifying where the issue might be? Any other outputs I can pull out or test to help identify? We've now confirmed multiple folks in my company faced this issue but have been able to use the work-around and we've documented it internally.

@chrmarti
Copy link
Contributor

chrmarti commented Sep 9, 2021

@sandy081 Are you downloading the VSIX using the Node.js https module in the renderer/shared process? Did this change from using the Chromium network stack?

@sandy081
Copy link
Member

I use the request service to download and it seems this is the request service that is being registered in renderer -

https://github.com/microsoft/vscode-distro/blob/731402dce225ce25d42a7b173361650a54bf52ad/src/vs/workbench/services/request/electron-sandbox/requestService.ts#L13-L13

It seems this uses chrome stack.

@aeschli
Copy link
Contributor

aeschli commented Sep 17, 2021

Looks like various issues here:

  • on remote, the process that manages the extensions (if I read the code correctly that the agent) needs the same treatment as the ext host. @chrmarti Could you do that?
  • downloading on local also has issue (although @srinathh say that installing extensions locally works)

@aeschli aeschli assigned chrmarti and sandy081 and unassigned aeschli Sep 17, 2021
@sandy081 sandy081 removed their assignment Sep 17, 2021
@chrmarti chrmarti added bug Issue identified by VS Code Team member as probable bug proxy Issues regarding network proxies labels Sep 20, 2021
@gbeal-smsc
Copy link

Hello! I've developed a work around.

The core issue here seems to be that the certificate store passed by NodeJS running the back-end server to the OpenSSL library it uses to deal with encryption is not the linux system certificate store updated by update-ca-certificates. However, NodeJS has an environment variable configuration option NODE_EXTRA_CA_CERTS in which we can pass it the custom certificate. We can setup this environment variable in ~/.vscode-server/server-env-setup in WSL.

Something like this works

echo "export NODE_EXTRA_CA_CERTS=<PATH TO SELF SIGNED ROOT>" >>  ~/.vscode-server/server-env-setup

However, ideally the solution should be to have NodeJS use the system certificate store

You are the hero I needed, thanks for this!

@shsuman
Copy link
Member

shsuman commented Oct 5, 2021

One of our customers is also impacted by the same issue when connecting to Azure ML Compute instances through VSCode. Azure ML - Remote extension tried to install the Python extension as part of setting up the VSCode server and that's where they get this error:
[2021-10-04 17:03:27.801] [renderer6] [error] Error while installing 'ms-python.python' extension in the remote server. self signed certificate in certificate chain

@J0F3
Copy link

J0F3 commented Nov 5, 2021

@chrmarti and @sandy081 Can you again look into that. It seems to be pop up again with the 1.62 version of VS Code (tough with all extension, not just the python). I have tested it by rolling back to 1.61.2 and then I can install all extensions again in WSL. But then as soon as update to 1.62 the installation of any extension in WSL fails. (The version "Remote - WSL" extensions is in both cases still the same same version - 0.58.5) When looking into the log it is quite obvious what happens different:

In version 1.61.2 the cert error also happens but then it automatically downloads the extensions locally instead (line 1 & 2):

[2021-11-05 14:12:42.022] [renderer1] [error] Error while installing 'dbaeumer.vscode-eslint' extension in the remote server. self signed certificate in certificate chain**
[2021-11-05 14:12:42.022] [renderer1] [info] Downloading the 'dbaeumer.vscode-eslint' extension locally and install
[2021-11-05 14:12:42.067] [renderer1] [info] Downloaded extension: dbaeumer.vscode-eslint /c:/Users/j0f3/AppData/Local/Temp/1e84c682-6562-432a-8e63-78d8303469ef
[2021-11-05 14:12:43.133] [renderer1] [info] Successfully installed 'dbaeumer.vscode-eslint' extension

But now with version 1.62 the process just stops at the cert error:

[2021-11-05 14:14:53.702] [renderer1] [error] ["Error: self signed certificate in certificate chain","    at TLSSocket.onConnectSecure (_tls_wrap.js:1497:34)","    at TLSSocket.emit (events.js:315:20)","    at TLSSocket._finishInit (_tls_wrap.js:932:8)","    at TLSWrap.ssl.onhandshakedone (_tls_wrap.js:706:12)"]
[2021-11-05 14:14:53.714] [renderer1] [error] self signed certificate in certificate chain: Error: self signed certificate in certificate chain
    at TLSSocket.onConnectSecure (_tls_wrap.js:1497:34)
    at TLSSocket.emit (events.js:315:20)
    at TLSSocket._finishInit (_tls_wrap.js:932:8)
    at TLSWrap.ssl.onhandshakedone (_tls_wrap.js:706:12)

I can workaround it for now by settings the remote.downloadExtensionsLocally setting to true or by using the method @gbeal-smsc has describe above. But while looking at the solution from @gbeal-smsc I think it would be the best if the vscode-server would somehow automatically inherit the NODE_EXTRA_CA_CERTS environment variable if it is set in WSL already.
Because of our corporate Proxy with TLS-Inspection I have to set the NODE_EXTRA_CA_CERTS environment variable anyway in WSL but it is then not very intuitive that the vscode-server does not know anything about it by default.

So not sure which one would actually be the "correct" solution but I think either the extension installation process should automatically fail back again to local download (as it was in 1.61.2) or the vscode-server should automatically pickup the NODE_EXTRA_CA_CERTS environment variable if it is set in WSL. But just fail with the cert error and leave the user with the problem, as it is now in 1.62, is IHMO the worst option. 😉

Thanks!

@sandy081
Copy link
Member

sandy081 commented Nov 9, 2021

In 1.62.0, I have made retry downloading (fallback) from local bit more strict and is triggered only when we see specific errors as it does not makes sense to retry if it failed for some obvious reasons.

case ExtensionManagementErrorCode.Download:
case ExtensionManagementErrorCode.Internal:
try {
this.logService.error(`Error while installing '${extension.identifier.id}' extension in the remote server.`, toErrorMessage(error));
return await this.downloadAndInstall(extension, installOptions || {});
} catch (e) {
this.logService.error(e);
throw e;
}
default:
this.logService.debug(error.name);
throw error;

I would be interested to know what error (name) is thrown in your case. So I added a debug statement for that. Can you please try with tomorrow's insiders and let me know that.

For those users, who have connectivity issues on remote, there is a setting you can use to always download the extension locally - remote.downloadExtensionsLocally. I would recommend to set this setting instead of relying on fallback we have.

@J0F3
Copy link

J0F3 commented Nov 9, 2021

In 1.62.0, I have made retry downloading (fallback) from local bit more strict and is triggered only when we see specific errors as it does not makes sense to retry if it failed for some obvious reasons.

Ah yes that makes sense.

I would be interested to know what error (name) is thrown in your case. So I added a debug statement for that. Can you please try with tomorrow's insiders and let me know that.

Sure, I will do that and keep you posted. 👍

@J0F3
Copy link

J0F3 commented Nov 10, 2021

I have just tested with the insider version (2021-11-10T07:59:05.913Z, Commit: bef4dba) but I am afraid that there is not really more to see in the log:

[2021-11-10 12:15:38.172] [renderer1] [debug] Remote Install Error Name Error
[2021-11-10 12:15:38.202] [renderer1] [error] ["Error: self signed certificate in certificate chain","    at TLSSocket.onConnectSecure (_tls_wrap.js:1497:34)","    at TLSSocket.emit (events.js:315:20)","    at TLSSocket._finishInit (_tls_wrap.js:932:8)","    at TLSWrap.ssl.onhandshakedone (_tls_wrap.js:706:12)"]
[2021-11-10 12:15:38.230] [renderer1] [error] self signed certificate in certificate chain: Error: self signed certificate in certificate chain
    at TLSSocket.onConnectSecure (_tls_wrap.js:1497:34)
    at TLSSocket.emit (events.js:315:20)
    at TLSSocket._finishInit (_tls_wrap.js:932:8)
    at TLSWrap.ssl.onhandshakedone (_tls_wrap.js:706:12)

Further I have noticed also problem installing extensions during dev container build and startup. I have a dev container setup wich should install the ms-vscode.powershell when the container is created. But after the container is created and running the extension is not installed. In the Remote Server Log i see the following:

[2021-11-10 11:06:19.699] [remoteagent] [info] 

*
* Visual Studio Code Server
*
* Reminder: You may only use this software with Visual Studio family products,
* as described in the license https://aka.ms/vscode-remote/license
*


[2021-11-10 11:06:19.699] [remoteagent] [info] Extension host agent started.
[2021-11-10 11:06:21.679] [remoteagent] [error] Error: self signed certificate in certificate chain
    at TLSSocket.onConnectSecure (_tls_wrap.js:1497:34)
    at TLSSocket.emit (events.js:315:20)
    at TLSSocket._finishInit (_tls_wrap.js:932:8)
    at TLSWrap.ssl.onhandshakedone (_tls_wrap.js:706:12)
[2021-11-10 11:06:21.787] [remoteagent] [info] [::ffff:127.0.0.1][5b720351][ManagementConnection] New connection established.

So it seems to be because of the same error as in WSL. That this is also when I have "remote.downloadExtensionsLocally": true in my settings. Installing the extension then manually after the container has started works. But it is not installed automatically during container creation.

@sandy081
Copy link
Member

Thanks for the logs @J0F3. I got what I am looking for

[2021-11-10 12:15:38.172] [renderer1] [debug] Remote Install Error Name Error

It seems this is an error that I am not handling properly so that the fallback can be triggered - I will be fixing this here - #136710, So please follow up there.

So it seems to be because of the same error as in WSL. That this is also when I have "remote.downloadExtensionsLocally": true in my settings. Installing the extension then manually after the container has started works. But it is not installed automatically during container creation.

While container is being created, extensions are downloaded and installed on remote server/container. Since the VS Code client is not yet connected to server, it is not possible to download on client and install on remote. CC @chrmarti

@J0F3
Copy link

J0F3 commented Nov 10, 2021

It seems this is an error that I am not handling properly so that the fallback can be triggered - I will be fixing this here - #136710, So please follow up there.

Perfect! Thx!👍

So it seems to be because of the same error as in WSL. That this is also when I have "remote.downloadExtensionsLocally": true in my settings. Installing the extension then manually after the container has started works. But it is not installed automatically during container creation.

While container is being created, extensions are downloaded and installed on remote server/container. Since the VS Code client is not yet connected to server, it is not possible to download on client and install on remote. CC @chrmarti

Aha that makes sense. Thanks to your explanation I found what the problem was here. The container did not use the custom Root CA Certs of our company firewall. When I correctly define the NODE_EXTRA_CA_CERTS environment variable in my dev container configuration then the extension get installed automatically again. This one was then more a missing config from my side.

Thank you for your help!

@megakid
Copy link

megakid commented Mar 5, 2024

Had this issue with my new WSL2 environment.

We add two self signed root CAs to our trusted root certificates (via update-ca-certificates on the WSL side and added to Windows certificate store on the Windows side). I fixed the error by adding this file:

❯ cat ~/.vscode-server/server-env-setup
export NODE_EXTRA_CA_CERTS=/etc/ssl/certs/ca-certificates.crt

❯ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.3 LTS
Release: 22.04
Codename: jammy

@chrmarti
Copy link
Contributor

chrmarti commented Mar 6, 2024

VS Code 1.85 and later should automatically load the remote OS certificates for remote extensions. For WSL and local dev containers the local OS certificates are also loaded.

@megakid Have you tried with VS Code 1.85 or later without setting NODE_EXTRA_CA_CERTS?

@michaelteo
Copy link

VS Code 1.85 and later should automatically load the remote OS certificates for remote extensions. For WSL and local dev containers the local OS certificates are also loaded.

@megakid Have you tried with VS Code 1.85 or later without setting NODE_EXTRA_CA_CERTS?

I needed to do the same workaround to install Jupyter Extension on WSL2 Ubuntu22.04. VSCode version is 1.87.2

@mrhorvath
Copy link

VS Code 1.85 and later should automatically load the remote OS certificates for remote extensions. For WSL and local dev containers the local OS certificates are also loaded.
@megakid Have you tried with VS Code 1.85 or later without setting NODE_EXTRA_CA_CERTS?

I needed to do the same workaround to install Jupyter Extension on WSL2 Ubuntu22.04. VSCode version is 1.87.2

Yes, I had the exact same issue installing the Jupyter Extension (Id: ms-toolsai.jupyter) extension in WSL2 Ubuntu22.04.

Version: 1.87.2 (user setup)
Commit: 863d258
Date: 2024-03-08T15:20:17.278Z
Electron: 27.3.2
ElectronBuildId: 26836302
Chromium: 118.0.5993.159
Node.js: 18.17.1
V8: 11.8.172.18-electron.0
OS: Windows_NT x64 10.0.19045

Adding the environmental variable NODE_EXTRA_CA_CERTS to ~/.vscode-server/server-env-setup fixed the issue for me for now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Issue identified by VS Code Team member as probable bug proxy Issues regarding network proxies
Projects
None yet
Development

No branches or pull requests

10 participants