bug: error occured when upstream application exists websocket

[totemofwolf]

Issue description

Environment

• apisix version (cmd: apisix version): apisix:2.*-alpine
• OS (cmd: uname -a): bash-5.0# uname -a Linux server 4.14.105-19-0019 #1 SMP Fri Jan 15 11:39:34 CST 2021 x86_64 Linux
• OpenResty / Nginx version (cmd: nginx -V or openresty -V):

# nginx -V
nginx version: openresty/1.19.3.1
built by gcc 10.2.1 20201203 (Alpine 10.2.1_pre1) 
built with OpenSSL 1.1.1i  8 Dec 2020
TLS SNI support enabled
configure arguments: --prefix=/usr/local/openresty/nginx --with-cc-opt='-O2 -DNGX_LUA_ABORT_AT_PANIC -I/usr/local/openresty/pcre/include -I/usr/local/openresty/openssl/include' --add-module=../ngx_devel_kit-0.3.1 --add-module=../echo-nginx-module-0.62 --add-module=../xss-nginx-module-0.06 --add-module=../ngx_coolkit-0.2 --add-module=../set-misc-nginx-module-0.32 --add-module=../form-input-nginx-module-0.12 --add-module=../encrypted-session-nginx-module-0.08 --add-module=../srcache-nginx-module-0.32 --add-module=../ngx_lua-0.10.19 --add-module=../ngx_lua_upstream-0.07 --add-module=../headers-more-nginx-module-0.33 --add-module=../array-var-nginx-module-0.05 --add-module=../memc-nginx-module-0.19 --add-module=../redis2-nginx-module-0.15 --add-module=../redis-nginx-module-0.3.7 --add-module=../rds-json-nginx-module-0.15 --add-module=../rds-csv-nginx-module-0.09 --add-module=../ngx_stream_lua-0.0.9 --with-ld-opt='-Wl,-rpath,/usr/local/openresty/luajit/lib -L/usr/local/openresty/pcre/lib -L/usr/local/openresty/openssl/lib -Wl,-rpath,/usr/local/openresty/pcre/lib:/usr/local/openresty/openssl/lib' --with-pcre --with-compat --with-file-aio --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_geoip_module=dynamic --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module=dynamic --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-http_xslt_module=dynamic --with-ipv6 --with-mail --with-mail_ssl_module --with-md5-asm --with-pcre-jit --with-sha1-asm --with-stream --with-stream_ssl_module --with-threads --with-stream --with-stream_ssl_preread_module

bash-5.0# openresty -v
nginx version: openresty/1.19.3.1

• etcd version, if have (cmd: run curl http://127.0.0.1:9090/v1/server_info to get the info from server-info API):

bash-5.0# curl http://127.0.0.1:9090/v1/server_info
{"version":"2.4","hostname":"server","id":"2ba49791-ce28-48c4-b03c-7a6d56d0d8c8","boot_time":1617158058,"etcd_version":"3.4.0","up_time":2737,"last_report_time":1617160758}

• apisix-dashboard version, if have: nil

Minimal test code / Steps to reproduce the issue

1. add upstream , route for confluence

# route
{
    "code":0,
    "message":"",
    "data":{
        "id":"00000000000000000190",
        "update_time":1617038159,
        "uris":[
            "/*"
        ],
        "name":"https_wiki",
        "desc":"https://wiki.xxx.cn",
        "priority":20,
        "hosts":[
            "wiki.xxx.cn"
        ],
        "vars":[

        ],
        "upstream_id":"2",
        "status":1
    },
    "request_id":"5add5e62-df98-4cc7-a691-87da8dd3f9b1"
}

# upstream
{
    "code":0,
    "message":"",
    "data":{
        "id":"2",
        "update_time":1617032983,
        "nodes":[
            {
                "host":"127.0.0.1",
                "port":56381,
                "weight":10
            }
        ],
        "timeout":{
            "connect":6000,
            "read":6000,
            "send":6000
        },
        "type":"roundrobin",
        "pass_host":"pass",
        "name":"frps proxy",
        "desc":"frps http proxy upstream"
    },
    "request_id":"0a464ece-3b29-4870-b051-815633836c33"
}

2. click edit page button in confluence
3. open Chrome - F12 - Network - WS

What's the actual result? (including assertion message & call stack if applicable)

Left part of pic

What's the expected result?

Right part of pic

GET wss://wiki.xxx.cn/synchrony-proxy/v1/bayeux-sync1 HTTP/1.1
Host: wiki.xxx.cn
Connection: Upgrade
Pragma: no-cache
Cache-Control: no-cache
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.104 Safari/537.36
Upgrade: websocket
Origin: https://wiki.xxx.cn
Sec-WebSocket-Version: 13
Accept-Encoding: gzip, deflate, br
Accept-Language: zh-CN,zh;q=0.9
Cookie: JSESSIONID=9293EFA6EF5C6F861A
x-forwarded-for: 127.0.0.1
ali-cdn-real-ip: 127.0.0.1
Sec-WebSocket-Key: L3uBKeBsGQ==
Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits


HTTP/1.1 101 Switching Protocols
Connection: upgrade
Upgrade: websocket
Sec-WebSocket-Accept: j3AFiIvXaRg+Aac=
Sec-WebSocket-Extensions: permessage-deflate;client_max_window_bits=15
Date: Wed, 31 Mar 2021 03:10:28 GMT
Server: APISIX web server

[spacewander]

You need to set enable_websocket in the route:
https://github.com/apache/apisix/blob/master/docs/en/latest/admin-api.md#route

[totemofwolf]

You need to set enable_websocket in the route:
https://github.com/apache/apisix/blob/master/docs/en/latest/admin-api.md#route

Does the enable_websocket suitable for all requests ? My case is that, the major requests (for this route) are https and wss was used only when Collaborative editing page in confluence.

[tokers]

You need to set enable_websocket in the route:
https://github.com/apache/apisix/blob/master/docs/en/latest/admin-api.md#route

Does the enable_websocket suitable for all requests ? My case is that, the major requests (for this route) are https and wss was used only when Collaborative editing page in confluence.

Does the Collaborative editing page stuff can be decoupled into a single route?

[totemofwolf]

You need to set enable_websocket in the route:
https://github.com/apache/apisix/blob/master/docs/en/latest/admin-api.md#route

Does the enable_websocket suitable for all requests ? My case is that, the major requests (for this route) are https and wss was used only when Collaborative editing page in confluence.

Does the Collaborative editing page stuff can be decoupled into a single route?

They have the same host, but differ in uri

[totemofwolf]

You need to set enable_websocket in the route:
https://github.com/apache/apisix/blob/master/docs/en/latest/admin-api.md#route

Problem solved by adding an individual host like this >

{
  "uris": [
    "/synchrony-proxy/v1/*"
  ],
  "name": "wss_wiki",
  "desc": "wss://wiki.xxx.cn",
  "priority": 10,
  "hosts": [
    "wiki.xxx.cn"
  ],
  "vars": [
    [
      "scheme",
      "==",
      "https"
    ]
  ],
  "upstream_id": "2",
  "enable_websocket": true,
  "status": 1
}

why does APISIX 2.* not transparent proxy requests like 1.* (as it worked before)

[spacewander]

why does APISIX 2.* not transparent proxy requests like 1.* (as it worked before)

@membphis
Wait for your answer...

[membphis]

I just chatted with @totemofwolf through IM, we need to add some configuration items to APISIX 1.* to support it.

He will post more information later.

[totemofwolf]

I just chatted with @totemofwolf through IM, we need to add some configuration items to APISIX 1.* to support it.

He will post more information later.

@spacewander
for APISIX 1.* nginx.conf should be this (to transparent proxy requests, include ws , wss..) (it is from the nginx ingress' default nginx.conf)

http {
    ...
    
    + # See https://www.nginx.com/blog/websocket-nginx
    + map $http_upgrade $connection_upgrade {
    +     default          upgrade;

    +     # See http://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive
    +     ''               '';
    + }

    ...
}

server {
            + # Allow websocket connections
            + proxy_set_header Upgrade           $http_upgrade;
            + proxy_set_header Connection        $connection_upgrade;

            set $upstream_host               $host;
            + # set $upstream_upgrade            '';
            + # set $upstream_connection         '';
            set $upstream_uri                '';

            # proxy_http_version 1.1;
            proxy_set_header   Host              $upstream_host;
            + # proxy_set_header   Upgrade           $upstream_upgrade;
            + # proxy_set_header   Connection        $upstream_connection;
}