Access code-server from coder response 502 Gad Gateway

[lvchenyang]

502 - Bad Gateway
Failed to proxy request to application: proxyconnect tcp: dial context: connect tcp [fd7a:115c:a1e0:42ba:af8f:1cde:1c68:a816]:7890: connection was refused

+ mkdir -p /tmp/code-server/lib /tmp/code-server/bin
+ tar -C /tmp/code-server/lib -xzf ~/.cache/code-server/code-server-4.19.1-linux-arm64.tar.gz
+ mv -f /tmp/code-server/lib/code-server-4.19.1-linux-arm64 /tmp/code-server/lib/code-server-4.19.1
+ ln -fs /tmp/code-server/lib/code-server-4.19.1/bin/code-server /tmp/code-server/bin/code-server
Standalone release has been installed into /tmp/code-server/lib/code-server-4.19.1
Extend your path to use code-server:
  PATH="/tmp/code-server/bin:$PATH"
Then run with:
  code-server
Deploy code-server for your team with Coder: https://github.com/coder/coder

[matifali]

Are you sure code-server is running? Please share your startup_script or how you are installing/starting code-server.

Additionally, try code-server module from Coder registry.

[bfqrst]

I think this hit us too! For some additional context, it seems to have appeared in v2.8.0 and it's still around in the latest v2.9.x. The v2.7 versions are not affected by this. Running v2.7.3 at the moment.

[jacobwi]

Same issue after the update

[bfqrst]

So a couple of things we noticed:

1. The IPv6 stuff is similar in all screenshots and seems to be related to the Tailscale networking stuff, so that is where I would focus the hunt
2. The port in our screenshot is 3128 which is our corporate proxy port. That seems to get picked up along the way. Unsure if 7890 is also some proxy in case of @lvchenyang
3. We made sure that code-server comes up correctly with our configured port and is curl-able from inside the machine

Hope that helps!
Cheers

[bfqrst]

Update: latest v2.10.0 unsurprisingly also does not work!

[spikecurtis]

@coadler this could be related to #11810

@lvchenyang @bfqrst @jacobwi is code-server listening port bound to IPv4, IPv6 or both?

[bfqrst]

So this is the command that starts code-server: /usr/lib/code-server/lib/node /usr/lib/code-server --auth none --port 8080. Not sure that if you don't explicitly configure something it's both stacks... In terms of netstat, code-server only listens to IPv4 as far as I can tell:

netstat -tulpn 
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:8787            0.0.0.0:*               LISTEN      2277/rserver        
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -                   
tcp        0      0 127.0.0.1:6060          0.0.0.0:*               LISTEN      1823/./coder        
tcp        0      0 127.0.0.1:8080          0.0.0.0:*               LISTEN      2349/node           
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      -                   
tcp        0      0 127.0.0.1:2112          0.0.0.0:*               LISTEN      1823/./coder        
tcp        0      0 127.0.0.1:2113          0.0.0.0:*               LISTEN      1823/./coder        
tcp6       0      0 :::111                  :::*                    LISTEN      -                   
tcp6       0      0 :::22                   :::*                    LISTEN      -                   
udp        0      0 127.0.0.53:53           0.0.0.0:*                           -                   
udp        0      0 10.194.73.0:68          0.0.0.0:*                           -                   
udp        0      0 0.0.0.0:111             0.0.0.0:*                           -                   
udp        0      0 127.0.0.1:323           0.0.0.0:*                           -                   
udp        0      0 0.0.0.0:41581           0.0.0.0:*                           1823/./coder        
udp6       0      0 :::40399                :::*                                1823/./coder        
udp6       0      0 :::111                  :::*                                -                   
udp6       0      0 ::1:323                 :::*                                -

Was that your question @spikecurtis?

[matifali]

I am getting into this too on v2.9.0+ while using vscode-web module. Wildcard domains are correctly configured.

Failed to proxy request to application: dial context: connect tcp [fd7a:115c:a1e0:4337:9c08:29d0:7c2c:6274]:13338: connection was refused

while app is running fine,

ps aux | grep /tmp/vscode-web
coder        469  0.0  0.0   2892  1600 ?        S    Apr03   0:00 sh /tmp/vscode-web/bin/code-server serve-local --port 13338 --accept-server-license-terms serve-local --without-connection-token --telemetry-level error
coder        529  0.0  0.0 920252 70856 ?        Sl   Apr03   0:23 /tmp/vscode-web/node /tmp/vscode-web/out/server-main.js serve-local --port 13338 --accept-server-license-terms serve-local --without-connection-token --telemetry-level error

And the coder_script log is.

cat /tmp/vscode-web.log 
*
* Visual Studio Code Server
*
* By using the software, you agree to
* the Visual Studio Code Server License Terms (https://aka.ms/vscode-server-license) and
* the Microsoft Privacy Statement (https://privacy.microsoft.com/en-US/privacystatement).
*
Server bound to ::1:13338 (IPv6)
Extension host agent listening on 13338

[18:41:03] 




Web UI available at http://localhost:13338/
[18:41:03] Extension host agent started.
[18:41:03] Started initializing default profile extensions in extensions installation folder. file:///home/coder/.vscode-server/extensions
[18:41:03] Completed initializing default profile extensions in extensions installation folder. file:///home/coder/.vscode-server/extensions

When I do ss -tlpn, I do not see port 13338

State      Recv-Q     Send-Q         Local Address:Port           Peer Address:Port     Process                                   
LISTEN     0          4096               127.0.0.1:6060                0.0.0.0:*         users:(("coder",pid=49,fd=7))            
LISTEN     0          4096               127.0.0.1:2112                0.0.0.0:*         users:(("coder",pid=49,fd=9))            
LISTEN     0          4096               127.0.0.1:2113                0.0.0.0:*         users:(("coder",pid=49,fd=8))            
LISTEN     0          4096               127.0.0.1:13339               0.0.0.0:*         users:(("filebrowser",pid=216,fd=7))     
LISTEN     0          128                  0.0.0.0:19999               0.0.0.0:*         users:(("jupyter-lab",pid=224,fd=6))     
LISTEN     0          511                    [::1]:13338                  [::]:*         users:(("node",pid=529,fd=19))

[coadler]

@bfqrst does the port shown in the error screen say [ipv6 address]:8080? Could you also confirm the url in your coder_app is http://localhost:8080?

[bfqrst]

So a couple of things we noticed:

1. The IPv6 stuff is similar in all screenshots and seems to be related to the Tailscale networking stuff, so that is where I would focus the hunt
2. The port in our screenshot is 3128 which is our corporate proxy port. That seems to get picked up along the way. Unsure if 7890 is also some proxy in case of @lvchenyang
3. We made sure that code-server comes up correctly with our configured port and is curl-able from inside the machine

Hope that helps! Cheers

The above is our error screenshot. It shows the IPv6 address in brackets, but the port is our corporate proxy port 3128. So that somehow enters the mix.

And yes, code-server runs under: url = "http://localhost:8080/?folder=/home/${local.linux_user}".

[coadler]

Sorry, completely missed that, thanks for the info. Last question, is coder being accessed over the proxy port you mentioned? i.e., is it somewhere in the browser URL?

[bfqrst]

No worries, thanks for looking into it!

The 3128 HAS to be the port we're setting as our HTTP[S]_PROXY and NO_PROXY vars on the respective workspaces/machines. In other words from the machine to the outside world. The other way around is: browser --> ingress (Traefik) --> Coder --> workspace from template (AWS EC2 based). It's not in the browser URL.

Does that make sense?

[coadler]

That makes sense. So is the HTTP[S]_PROXY env also set in the coder pods themselves as well?

EDIT: Yeah, this definitely seems to be the case. The HTTP client we use to dial workspaces over the tailnet will respect these envs. This is definitely a bug on our end.

[bfqrst]

That makes sense. So is the HTTP[S]_PROXY env also set in the coder pods themselves as well?

In our case it's Nomad jobs not pods, but your assessment seems to be true nonetheless!

Thanks @coadler

[coadler]

This is fixed now and should be included in the next 2.10.x release