nginx: stream error received: unspecific protocol error detected #27

shumvgolove · 2023-11-06T14:50:30Z

Hey 👋 and thanks for the great project!

After several attempts to figure out how to make it work with nginx reverse-proxy, I've created half-working solution. I am providing below steps to reproduce the issue and ultimately figure out what is the problem.

`sshx-server` setup

Create and enter temp folder:

mkdir /tmp/reproduce-issue && cd /tmp/reproduce-issue

Clone and enter the sshx repository:

git clone https://github.com/ekzhang/sshx.git && cd sshx

Build frontend:

npm ci && npm run build

Build backend:

cargo build --release --bin sshx-server

Launch sshx (which will also serve frontend).:

mv target/release/sshx-server . && ./sshx-server --secret dev-secret --listen :: --override-origin https://localhost:8080

`nginx` setup

Create and enter nginx folder:

mkdir /tmp/reproduce-issue/nginx && cd /tmp/reproduce-issue/nginx

Create nginx.conf with the following content:

# Run nginx using:
#     nginx -p $PWD -e stderr -c nginx.conf

daemon off;  # run in foreground

events {}

pid nginx.pid;

http {
    access_log /dev/stdout;

    # Locally generated SSL certificates from mkcert
    ssl_certificate ./localhost+1.pem;
    ssl_certificate_key ./localhost+1-key.pem;
    ssl_protocols	TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers on;
    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;

    # Directories nginx needs configured to start up.
    client_body_temp_path .;
    proxy_temp_path .;
    fastcgi_temp_path .;
    uwsgi_temp_path .;
    scgi_temp_path .;
    
    ###################
    client_body_timeout 5;  # <------ makes sshx client spitting error log every 5 seconds
    ###################

    # Connection upgrade variable for websocket
    map $http_upgrade $connection_upgrade {  
        default upgrade;
        ''      close;
    }

    server {
        server_name localhost 127.0.0.1;
        listen 8080 ssl http2;

        location / {
            proxy_pass   http://127.0.0.1:8051;
        }

        location /api {
            # Websocket headers
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection $connection_upgrade;
            proxy_set_header Host $http_host;
            proxy_pass   http://127.0.0.1:8051;
        }
    }
}

Setup local certificates with mkcert (otherwise grpc/websockets won't work) and assign correct permissions:

sudo mkcert -install && sudo mkcert localhost 127.0.0.1 && sudo chown $USER:$USER *.pem

Launch nginx:

nginx -p $PWD -e stderr -c nginx.conf

`sshx` setup

The default sshx binaries won't work, because tonic is compiled with tls-webpki-roots feature, which means that sshx will not lookup OS ca-certificates bundle, making our self-signed certificates useless and throwing the following error:

❯ ./sshx --server https://localhost:8080
2023-11-06T12:15:26.058671Z ERROR sshx: transport error

Caused by:
    0: error trying to connect: invalid peer certificate: UnknownIssuer
    1: invalid peer certificate: UnknownIssuer

In order to fix that we need to recompile sshx binary with tls-roots feature (which will lookup OS ca-certificates):

Enter sshx cloned git repository:

cd /tmp/reproduce-issue/sshx

Save the following content as tonic_fix.patch:

diff --git a/Cargo.toml b/Cargo.toml
index 4b0c58e..c028b8d 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -19,7 +19,7 @@ rand = "0.8.5"
 serde = { version = "1.0.188", features = ["derive", "rc"] }
 tokio = { version = "1.32.0", features = ["full"] }
 tokio-stream = { version = "0.1.14", features = ["sync"] }
-tonic = { version = "0.10.0", features = ["tls", "tls-webpki-roots"] }
+tonic = { version = "0.10.0", features = ["tls", "tls-roots"] }
 tracing = "0.1.37"
 tracing-subscriber = { version = "0.3.17", features = ["env-filter"] }

and apply it like so:

git apply tonic_fix.patch

Compile modified sshx:

cargo build --release --bin sshx

Launch client:

mv target/release/sshx . && ./sshx --server https://localhost:8080

Now, here's the issue: there's no connection between server and shell and I'm unable to spawn terminal:

And once in a while sshx produces the following log:

2023-11-06T14:25:05.152762Z ERROR sshx::controller: disconnected, retrying in 1s... err=status: Internal, message: "h2 protocol error: http2 error: stream error received: unspecific protocol error detected", details: [], metadata: MetadataMap { headers: {} }

Here's the log with RUST_LOG=trace: trace_sshx.log

Any help would be much appreciated!

The text was updated successfully, but these errors were encountered:

Hugo1380 · 2023-11-06T17:57:33Z

Adding this to the server section seems to fix it:

location /sshx.SshxService/Channel {
   grpc_pass      grpc://127.0.0.1:8051;
}

shumvgolove · 2023-11-06T18:57:57Z

Yep, this directive fixes the issue. Thanks a lot!

Maybe we should add the working nginx.conf example to wiki?

natarajan0007 · 2023-11-07T19:23:30Z

release]$ ./sshx --server https://localhost:8003
2023-11-07T19:15:11.596704Z ERROR sshx: transport error

Caused by:
0: error trying to connect: invalid peer certificate: UnknownIssuer
1: invalid peer certificate: UnknownIssuer
[muthUn@brlb release]$

As per your suggestion i was changed the below line of code in cargo.toml file still i am getting certificate error message.

diff --git a/Cargo.toml b/Cargo.toml index 4b0c58e..c028b8d 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -19,7 +19,7 @@ rand = "0.8.5" serde = { version = "1.0.188", features = ["derive", "rc"] } tokio = { version = "1.32.0", features = ["full"] } tokio-stream = { version = "0.1.14", features = ["sync"] } -tonic = { version = "0.10.0", features = ["tls", "tls-webpki-roots"] } +tonic = { version = "0.10.0", features = ["tls", "tls-roots"] } tracing = "0.1.37" tracing-subscriber = { version = "0.3.17", features = ["env-filter"] }

@shumvgolove

ekzhang · 2023-11-08T22:10:20Z

Yes, you'll have to grpc_pass all requests to suffixes of /sshx.SshxService.

Actually on the server it internally just checks if the content-type is application/grpc to decide whether to handle it as an HTTP request or a gRPC request. But since all gRPC requests go to SshxService, matching by path would work fine.

sshx/crates/sshx-server/src/listen.rs

Lines 51 to 60 in e01b75a

    
           let svc = Steer::new( 
        
               [http_service, grpc_service], 
        
               |req: &Request<Body>, _services: &[_]| { 
        
                   let headers = req.headers(); 
        
                   match headers.get(CONTENT_TYPE) { 
        
                       Some(content) if content == "application/grpc" => 1, 
        
                       _ => 0, 
        
                   } 
        
               }, 
        
           );

To clarify, you don't necessarily need to detect gRPC when you're putting a reverse proxy in front of sshx. You just need to forward HTTP/2 connections to it as the backend protocol. But Nginx doesn't let you do h2c (HTTP/2 cleartext) backend for its reverse proxy; it only has grpc_pass as a special case.

Sorry about this, Nginx is just eternally difficult to configure 😅

natarajan0007 · 2023-11-09T09:57:03Z

this is what i have in my nginx.conf . I am still facing certificate issue. would you mind sharing your nginx setup @shumvgolove

$ ./sshx --server https://localhost:8002
2023-11-09T09:39:28.982734Z ERROR sshx: transport error

Caused by:
0: error trying to connect: invalid peer certificate: UnknownIssuer
1: invalid peer certificate: UnknownIssuer

`events {}

pid nginx.pid;

http {
access_log /dev/stdout;

ssl_certificate ./localhost.pem;
ssl_certificate_key ./localhost-key.pem;
ssl_protocols	TLSv1.2 TLSv1.3;
ssl_prefer_server_ciphers on;
ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;

client_body_temp_path .;
proxy_temp_path .;
fastcgi_temp_path .;
uwsgi_temp_path .;
scgi_temp_path .;

client_body_timeout 5;

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

server {
    server_name localhost 127.0.0.1;
    listen 8003 ssl http2;

    location /sshx.SshxService {
        grpc_pass      grpc://127.0.0.1:8051;
    }

    location /api {
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_set_header Host $http_host;
        proxy_pass   http://127.0.0.1:8051;
    }
}

}
`

natarajan0007 · 2023-11-09T12:05:28Z

I tried without nginx . Its working fine. But i will try to see if i can able to make with nginx.

juiiyang · 2023-11-24T17:10:43Z

Hey 👋 and thanks for the great project!

After several attempts to figure out how to make it work with nginx reverse-proxy, I've created half-working solution. I am providing below steps to reproduce the issue and ultimately figure out what is the problem.

`sshx-server` setup

Create and enter temp folder:

mkdir /tmp/reproduce-issue && cd /tmp/reproduce-issue

Clone and enter the sshx repository:

git clone https://github.com/ekzhang/sshx.git && cd sshx

Build frontend:

npm ci && npm run build

Build backend:

cargo build --release --bin sshx-server

Launch sshx (which will also serve frontend).:

mv target/release/sshx-server . && ./sshx-server --secret dev-secret --listen :: --override-origin https://localhost:8080

`nginx` setup

Create and enter nginx folder:

mkdir /tmp/reproduce-issue/nginx && cd /tmp/reproduce-issue/nginx

Create nginx.conf with the following content:

# Run nginx using:
#     nginx -p $PWD -e stderr -c nginx.conf

daemon off;  # run in foreground

events {}

pid nginx.pid;

http {
    access_log /dev/stdout;

    # Locally generated SSL certificates from mkcert
    ssl_certificate ./localhost+1.pem;
    ssl_certificate_key ./localhost+1-key.pem;
    ssl_protocols	TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers on;
    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;

    # Directories nginx needs configured to start up.
    client_body_temp_path .;
    proxy_temp_path .;
    fastcgi_temp_path .;
    uwsgi_temp_path .;
    scgi_temp_path .;
    
    ###################
    client_body_timeout 5;  # <------ makes sshx client spitting error log every 5 seconds
    ###################

    # Connection upgrade variable for websocket
    map $http_upgrade $connection_upgrade {  
        default upgrade;
        ''      close;
    }

    server {
        server_name localhost 127.0.0.1;
        listen 8080 ssl http2;

        location / {
            proxy_pass   http://127.0.0.1:8051;
        }

        location /api {
            # Websocket headers
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection $connection_upgrade;
            proxy_set_header Host $http_host;
            proxy_pass   http://127.0.0.1:8051;
        }
    }
}

Setup local certificates with mkcert (otherwise grpc/websockets won't work) and assign correct permissions:

sudo mkcert -install && sudo mkcert localhost 127.0.0.1 && sudo chown $USER:$USER *.pem

Launch nginx:

nginx -p $PWD -e stderr -c nginx.conf

`sshx` setup

The default sshx binaries won't work, because tonic is compiled with tls-webpki-roots feature, which means that sshx will not lookup OS ca-certificates bundle, making our self-signed certificates useless and throwing the following error:

❯ ./sshx --server https://localhost:8080
2023-11-06T12:15:26.058671Z ERROR sshx: transport error

Caused by:
    0: error trying to connect: invalid peer certificate: UnknownIssuer
    1: invalid peer certificate: UnknownIssuer

In order to fix that we need to recompile sshx binary with tls-roots feature (which will lookup OS ca-certificates):

Enter sshx cloned git repository:

cd /tmp/reproduce-issue/sshx

Save the following content as tonic_fix.patch:

diff --git a/Cargo.toml b/Cargo.toml
index 4b0c58e..c028b8d 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -19,7 +19,7 @@ rand = "0.8.5"
 serde = { version = "1.0.188", features = ["derive", "rc"] }
 tokio = { version = "1.32.0", features = ["full"] }
 tokio-stream = { version = "0.1.14", features = ["sync"] }
-tonic = { version = "0.10.0", features = ["tls", "tls-webpki-roots"] }
+tonic = { version = "0.10.0", features = ["tls", "tls-roots"] }
 tracing = "0.1.37"
 tracing-subscriber = { version = "0.3.17", features = ["env-filter"] }

and apply it like so:

git apply tonic_fix.patch

Compile modified sshx:

cargo build --release --bin sshx

Launch client:

mv target/release/sshx . && ./sshx --server https://localhost:8080

Now, here's the issue: there's no connection between server and shell and I'm unable to spawn terminal:

And once in a while sshx produces the following log:

2023-11-06T14:25:05.152762Z ERROR sshx::controller: disconnected, retrying in 1s... err=status: Internal, message: "h2 protocol error: http2 error: stream error received: unspecific protocol error detected", details: [], metadata: MetadataMap { headers: {} }

Here's the log with RUST_LOG=trace: trace_sshx.log

Any help would be much appreciated!

Thanks for your guide, my problem is solved.

By the way, self signed cert need to add into keychain in Mac OS

shumvgolove mentioned this issue Nov 7, 2023

Allow specification of trusted certificates / CAs #31

Open

ekzhang closed this as completed Nov 16, 2023

muellerj mentioned this issue Nov 17, 2023

Add minimal nginx config #54

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

nginx: stream error received: unspecific protocol error detected #27

nginx: stream error received: unspecific protocol error detected #27

shumvgolove commented Nov 6, 2023 •

edited

Loading

Hugo1380 commented Nov 6, 2023 •

edited

Loading

shumvgolove commented Nov 6, 2023

natarajan0007 commented Nov 7, 2023

ekzhang commented Nov 8, 2023 •

edited

Loading

natarajan0007 commented Nov 9, 2023

natarajan0007 commented Nov 9, 2023

juiiyang commented Nov 24, 2023

`sshx-server` setup

`nginx` setup

`sshx` setup

nginx: stream error received: unspecific protocol error detected #27

nginx: stream error received: unspecific protocol error detected #27

Comments

shumvgolove commented Nov 6, 2023 • edited Loading

sshx-server setup

nginx setup

sshx setup

Hugo1380 commented Nov 6, 2023 • edited Loading

shumvgolove commented Nov 6, 2023

natarajan0007 commented Nov 7, 2023

ekzhang commented Nov 8, 2023 • edited Loading

natarajan0007 commented Nov 9, 2023

natarajan0007 commented Nov 9, 2023

juiiyang commented Nov 24, 2023

sshx-server setup

nginx setup

sshx setup

shumvgolove commented Nov 6, 2023 •

edited

Loading

`sshx-server` setup

`nginx` setup

`sshx` setup

Hugo1380 commented Nov 6, 2023 •

edited

Loading

ekzhang commented Nov 8, 2023 •

edited

Loading

`sshx-server` setup

`nginx` setup

`sshx` setup