Can't install remote extensions

[bdkuhman]

Is there an existing issue for this?

• I have searched the existing issues

OS/Web Information

• Web Browser: Firefox 125.0.2

• Local OS: Windows 10

• Remote OS: Ubuntu

• Remote Architecture: amd64

• code-server --version:

  1.88.1
  9a28bc2
  x64

• Running in Kubernetes k3s v1.28.7+k3s1

• Node is Ubuntu 22.04.1 kernel 6.5.0-28

• Image (latest) docker.io/codercom/code-server@sha256:897b358eed18114061b371f2b2f1416bc7acbed9442877428ecc0855523031ad

• I do not have persistence setup

Steps to Reproduce

1. Open code-server
2. Attempt to install extensions (seems like most of them, Tried python and kubernetes, among others)

Expected

Extension installs

Actual

Install hangs indefinitely

Logs

coder@code-server-5f58c85dcc-gp7h4:~$ code-server --verbose --install-extension ms-python.python
{
  "type": "extensionManagement",
  "install": [
    "ms-python.python"
  ],
  "force": false
}
^C
coder@code-server-5f58c85dcc-gp7h4:~$ code-server --verbose --install-extension does-not-exist
{
  "type": "extensionManagement",
  "install": [
    "does-not-exist"
  ],
  "force": false
}
Installing extensions on code.cluster.local...
Extension 'does-not-exist' not found.
Make sure you use the full extension ID, including the publisher, e.g.: ms-dotnettools.csharp
Failed Installing Extensions: does-not-exist

Tried installing through vsix, but also does not work (both cli and UI).
first try with verbose on and using the direct path got 503, but second try just hangs.

usr/bin/code-server --verbose --install-extension ms-python.python
[2024-05-02T15:41:17.626Z] debug parsed command line {"args":{"verbose":true,"install-extension":["ms-python.python"]}}
[2024-05-02T15:41:17.642Z] debug parsed config {"args":{"bind-addr":"127.0.0.1:8080","auth":"password","password":"<redacted>"}}
[2024-05-02T15:41:17.643Z] debug Found VS Code arguments; spawning VS Code CLI
[2024-05-02T15:41:17.644Z] debug Running Code CLI
Installing extensions...
Error while installing extensions: Server returned 503
Server returned 503
[2024-05-02T15:41:20.344Z] debug parent:3125 disposing {}
coder@code-server-86b59f55d-jk2z4:~$ /usr/bin/code-server --verbose --install-extension ms-python.python
[2024-05-02T15:42:04.160Z] debug parsed command line {"args":{"verbose":true,"install-extension":["ms-python.python"]}}
[2024-05-02T15:42:04.261Z] debug parsed config {"args":{"bind-addr":"127.0.0.1:8080","auth":"password","password":"<redacted>"}}
[2024-05-02T15:42:04.262Z] debug Found VS Code arguments; spawning VS Code CLI
[2024-05-02T15:42:04.263Z] debug Running Code CLI
Installing extensions...

^C[2024-05-02T15:43:51.542Z] debug parent:3250 disposing {"code":"SIGINT"}
coder@code-server-86b59f55d-jk2z4:~$ curl https://open-vsx.org/api/ms-python/python/2024.4.1/file/ms-python.python-2024.4.1.vsix
coder@code-server-86b59f55d-jk2z4:~$ ls
coder@code-server-86b59f55d-jk2z4:~$ wget https://open-vsx.org/api/ms-python/python/2024.4.1/file/ms-python.python-2024.4.1.vsix
--2024-05-02 15:44:04--  https://open-vsx.org/api/ms-python/python/2024.4.1/file/ms-python.python-2024.4.1.vsix
Resolving open-vsx.org (open-vsx.org)... 198.41.30.195
Connecting to open-vsx.org (open-vsx.org)|198.41.30.195|:443... connected.
HTTP request sent, awaiting response... 302 
Location: https://openvsxorg.blob.core.windows.net/resources/ms-python/python/2024.4.1/ms-python.python-2024.4.1.vsix [following]
--2024-05-02 15:44:06--  https://openvsxorg.blob.core.windows.net/resources/ms-python/python/2024.4.1/ms-python.python-2024.4.1.vsix
Resolving openvsxorg.blob.core.windows.net (openvsxorg.blob.core.windows.net)... 20.60.40.4
Connecting to openvsxorg.blob.core.windows.net (openvsxorg.blob.core.windows.net)|20.60.40.4|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 15917438 (15M) [application/octet-stream]
Saving to: ‘ms-python.python-2024.4.1.vsix’

ms-python.python-2024.4.1.vsix                            100%[=====================================================================================================================================>]  15.18M  4.84MB/s    in 3.1s    

2024-05-02 15:44:11 (4.84 MB/s) - ‘ms-python.python-2024.4.1.vsix’ saved [15917438/15917438]

coder@code-server-86b59f55d-jk2z4:~$ /usr/bin/code-server --verbose --install-extension ./ms-python.python-2024.4.1.vsix 
[2024-05-02T15:44:19.363Z] debug parsed command line {"args":{"verbose":true,"install-extension":["./ms-python.python-2024.4.1.vsix"]}}
[2024-05-02T15:44:19.373Z] debug parsed config {"args":{"bind-addr":"127.0.0.1:8080","auth":"password","password":"<redacted>"}}
[2024-05-02T15:44:19.374Z] debug Found VS Code arguments; spawning VS Code CLI
[2024-05-02T15:44:19.374Z] debug Running Code CLI
Installing extensions...

Screenshot/Video

No response

Does this bug reproduce in native VS Code?

I did not test native VS Code

Does this bug reproduce in GitHub Codespaces?

I did not test GitHub Codespaces

Are you accessing code-server over a secure context?

• I am using a secure context.

Notes

It seems that browser extensions do install. (k--kato.intellij-idea-keybindings works) but only by clicking install, not through the command line.

[code-asher]

Is there a proxy involved? http_proxy or something? I am not sure why wget would work but not code-server. But 503 seems like a weird error if it was a proxy issue.

Also interesting that it hangs even when installing from a .vsix. I believe when installing from a .vsix it still makes a request to the marketplace (maybe to check for the latest version).

[bdkuhman]

Is there a proxy involved? http_proxy or something? I am not sure why wget would work but not code-server. But 503 seems like a weird error if it was a proxy issue.

Also interesting that it hangs even when installing from a .vsix. I believe when installing from a .vsix it still makes a request to the marketplace (maybe to check for the latest version).

I don't have any forward proxies going out of the cluster; I do have two reverse proxies -- HAProxy outside and Traefik inside the cluster; though I can't imagine they'd cause this.

I gave openvscode-server a try, but same issue was happening.

It's possible it's a network/firewall issue but so far it's evaded me.

In the past I've had problems with not being able to get to windows smartscreen, along with failing to download vscode extensions- but that currently hasn't been happening. I've usually solved it with a router restart but I haven't been able to try yet, along with not really being able to determine that cause when it happens.

Edit:
Restarting my router didn't change anything.

[code-asher]

Yeah I agree a reverse proxy seems unlikely to be related.

What about trying with --disable-telemetry? Maybe it is trying to make some telemetry call that is being blocked.

[bdkuhman]

Yeah I agree a reverse proxy seems unlikely to be related.

What about trying with --disable-telemetry? Maybe it is trying to make some telemetry call that is being blocked.

no dice.

coder@code-server-78944b7c69-45sgq:~$ /usr/bin/code-server --disable-telemetry --verbose --install-extension ms-kubernetes-tools.vscode-kubernetes-tools
[2024-05-03T21:18:52.492Z] debug parsed command line {"args":{"disable-telemetry":true,"verbose":true,"install-extension":["ms-kubernetes-tools.vscode-kubernetes-tools"]}}
[2024-05-03T21:18:52.516Z] debug parsed config {"args":{"bind-addr":"127.0.0.1:8080","auth":"password","password":"<redacted>"}}
[2024-05-03T21:18:52.517Z] debug Found VS Code arguments; spawning VS Code CLI
[2024-05-03T21:18:52.517Z] debug Running Code CLI
Installing extensions...
^C[2024-05-03T21:19:42.371Z] debug parent:4385 disposing {"code":"SIGINT"}
coder@code-server-78944b7c69-45sgq:~$ code-server --verbose --disable-telemetry --install-extension ms-kubernetes-tools.vscode-kubernetes-tools
Ignoring option 'disable-telemetry': not supported for code-server.
{
  "type": "extensionManagement",
  "install": [
    "ms-kubernetes-tools.vscode-kubernetes-tools"
  ],
  "force": false
}
^C

[bdkuhman]

Tried investigating with a tcpdump, so far haven't noticed anything. All the domains it's trying to reach out to resolve successfully.

[code-asher]

I tried the (latest) Docker image just in case it was easily reproducible, but unfortunately it all worked. Got to be some network hang somewhere, feels like, but maybe VS Code dropped an exception somewhere and this is a hanging promise.

$ docker run --rm --name code-server codercom/code-server
[2024-05-03T22:38:50.411Z] info  Wrote default config file to /home/coder/.config/code-server/config.yaml
[2024-05-03T22:38:50.685Z] info  code-server 4.23.1 9a28bc29dbddb6886dfe03dc1c31320249a901ce
[2024-05-03T22:38:50.686Z] info  Using user-data-dir /home/coder/.local/share/code-server
[2024-05-03T22:38:50.698Z] info  Using config file /home/coder/.config/code-server/config.yaml
[2024-05-03T22:38:50.698Z] info  HTTP server listening on http://0.0.0.0:8080/
[2024-05-03T22:38:50.698Z] info    - Authentication is enabled
[2024-05-03T22:38:50.698Z] info      - Using password from /home/coder/.config/code-server/config.yaml
[2024-05-03T22:38:50.698Z] info    - Not serving HTTPS
[2024-05-03T22:38:50.698Z] info  Session server listening on /home/coder/.local/share/code-server/code-server-ipc.sock

$ docker exec -it code-server bash
coder@4f8432bc2752:~$ code-server --install-extension ms-kubernetes-tools.vscode-kubernetes-tools
Installing extensions...
Installing extension 'ms-kubernetes-tools.vscode-kubernetes-tools'...
Extension 'redhat.vscode-yaml' v1.14.0 was successfully installed.
Extension 'ms-kubernetes-tools.vscode-kubernetes-tools' v1.3.16 was successfully installed.

[code-asher]

Another idea for a test: if you download official VS Code and then use code --install-extension, does it have the same problem? If so maybe we can leverage that to open an issue upstream where they probably have more ideas.

[bdkuhman]

Another idea for a test: if you download official VS Code and then use code --install-extension, does it have the same problem? If so maybe we can leverage that to open an issue upstream where they probably have more ideas.

I went ahead and downloaded the latest (code_1.89.0-1714530869_amd64.deb) vscode on this node,

First test gave me

code --verbose --install-extension redhat.vscode-yaml
...
Installing extensions...
...
TRACE [File Watcher (node.js)] Started watching: '/home/user/.vscode/extensions/extensions.json'
DEBUG ComputeTargetPlatform: linux-x64
TRACE resolveShellEnv(): skipped (VSCODE_CLI is set)
TRACE Scanned user extensions: 0
TRACE [File Watcher (node.js)]  >> normalized [ADDED] /home/user/.vscode/extensions/extensions.json.vsctmp
DEBUG User data changed
Error while installing extensions: getaddrinfo EAI_AGAIN marketplace.visualstudio.com
getaddrinfo EAI_AGAIN marketplace.visualstudio.com

which looked promising, though a second test (and others) give me

...
Installing extensions...
...
TRACE Started scanning user extensions c {
...
}
DEBUG ComputeTargetPlatform: linux-x64
TRACE Scanned user extensions: 0
TRACE Scanned user extensions: 0
DEBUG ComputeTargetPlatform: linux-x64
TRACE resolveShellEnv(): skipped (VSCODE_CLI is set)
TRACE resolveShellEnv(): skipped (VSCODE_CLI is set)
TRACE resolveShellEnv(): skipped (VSCODE_CLI is set)

and hangs from there.

I gave the node a reboot, to see if that fixes anything. It seems sluggish, so there might just be some other system issue at play. Though I believe I was having issues even when this pod was scheduled on another node.

[bdkuhman]

I resolved an issue in /etc/hosts on that node, And it seems I can install extensions in vscode through the UI, but the cli just hangs.
both methods still hang in code-server

[code-asher]

Interesting. I am not sure why that would be, unfortunately.

Maybe setting "extensions.verifySignature": false will work, although maybe not since I did not see anything about a signature in your logs. Got the idea from microsoft/vscode#175705 and microsoft/vscode-remote-release#9628.

There are some other issues that look like they might be related, but it is hard to tell.

[bdkuhman]

Sure sounds similar.

Reading through the issues, it looks like "extensions.verifySignature": false needs to be set in .devcontainer/devcontainer.json and the user settings json won't work.
Is there a way to configure this in code-server?
I can fork if need be, just don't see devcontainer in the source tree.

[code-asher]

Ah sorry I missed that, I thought it was a regular setting. Looks like it is unrelated in this case after all, but if you want you could check CPU usage and processes for vsce-sign to confirm, I think.

Another idea I had is to launch code-server with NODE_DEBUG. For example:

NODE_DEBUG=http,https code-server --disable-telemetry --install-extension <ext>

Maybe something will stand out in the output.

[code-asher]

Here is what I get as far as host names, interesting there is a windows.net domain in there, maybe Azure or something.

$ NODE_DEBUG=http,https code-server --disable-telemetry --install-extension vscodevim.vim 2>&1 | grep hostname | sort | uniq
  hostname: 'open-vsx.org',
  hostname: 'openvsxorg.blob.core.windows.net',