Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nginx: stream error received: unspecific protocol error detected #27

Closed
shumvgolove opened this issue Nov 6, 2023 · 7 comments
Closed

Comments

@shumvgolove
Copy link

shumvgolove commented Nov 6, 2023

Hey 👋 and thanks for the great project!

After several attempts to figure out how to make it work with nginx reverse-proxy, I've created half-working solution. I am providing below steps to reproduce the issue and ultimately figure out what is the problem.

sshx-server setup

  1. Create and enter temp folder:
mkdir /tmp/reproduce-issue && cd /tmp/reproduce-issue
  1. Clone and enter the sshx repository:
git clone https://github.com/ekzhang/sshx.git && cd sshx
  1. Build frontend:
npm ci && npm run build
  1. Build backend:
cargo build --release --bin sshx-server
  1. Launch sshx (which will also serve frontend).:
mv target/release/sshx-server . && ./sshx-server --secret dev-secret --listen :: --override-origin https://localhost:8080

nginx setup

  1. Create and enter nginx folder:
mkdir /tmp/reproduce-issue/nginx && cd /tmp/reproduce-issue/nginx
  1. Create nginx.conf with the following content:
# Run nginx using:
#     nginx -p $PWD -e stderr -c nginx.conf

daemon off;  # run in foreground

events {}

pid nginx.pid;

http {
    access_log /dev/stdout;

    # Locally generated SSL certificates from mkcert
    ssl_certificate ./localhost+1.pem;
    ssl_certificate_key ./localhost+1-key.pem;
    ssl_protocols	TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers on;
    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;

    # Directories nginx needs configured to start up.
    client_body_temp_path .;
    proxy_temp_path .;
    fastcgi_temp_path .;
    uwsgi_temp_path .;
    scgi_temp_path .;
    
    ###################
    client_body_timeout 5;  # <------ makes sshx client spitting error log every 5 seconds
    ###################

    # Connection upgrade variable for websocket
    map $http_upgrade $connection_upgrade {  
        default upgrade;
        ''      close;
    }

    server {
        server_name localhost 127.0.0.1;
        listen 8080 ssl http2;

        location / {
            proxy_pass   http://127.0.0.1:8051;
        }

        location /api {
            # Websocket headers
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection $connection_upgrade;
            proxy_set_header Host $http_host;
            proxy_pass   http://127.0.0.1:8051;
        }
    }
}
  1. Setup local certificates with mkcert (otherwise grpc/websockets won't work) and assign correct permissions:
sudo mkcert -install && sudo mkcert localhost 127.0.0.1 && sudo chown $USER:$USER *.pem
  1. Launch nginx:
nginx -p $PWD -e stderr -c nginx.conf

sshx setup

The default sshx binaries won't work, because tonic is compiled with tls-webpki-roots feature, which means that sshx will not lookup OS ca-certificates bundle, making our self-signed certificates useless and throwing the following error:

❯ ./sshx --server https://localhost:8080
2023-11-06T12:15:26.058671Z ERROR sshx: transport error

Caused by:
    0: error trying to connect: invalid peer certificate: UnknownIssuer
    1: invalid peer certificate: UnknownIssuer

In order to fix that we need to recompile sshx binary with tls-roots feature (which will lookup OS ca-certificates):

  1. Enter sshx cloned git repository:
cd /tmp/reproduce-issue/sshx
  1. Save the following content as tonic_fix.patch:
diff --git a/Cargo.toml b/Cargo.toml
index 4b0c58e..c028b8d 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -19,7 +19,7 @@ rand = "0.8.5"
 serde = { version = "1.0.188", features = ["derive", "rc"] }
 tokio = { version = "1.32.0", features = ["full"] }
 tokio-stream = { version = "0.1.14", features = ["sync"] }
-tonic = { version = "0.10.0", features = ["tls", "tls-webpki-roots"] }
+tonic = { version = "0.10.0", features = ["tls", "tls-roots"] }
 tracing = "0.1.37"
 tracing-subscriber = { version = "0.3.17", features = ["env-filter"] }

and apply it like so:

git apply tonic_fix.patch
  1. Compile modified sshx:
cargo build --release --bin sshx
  1. Launch client:
mv target/release/sshx . && ./sshx --server https://localhost:8080

Now, here's the issue: there's no connection between server and shell and I'm unable to spawn terminal:

image

And once in a while sshx produces the following log:

2023-11-06T14:25:05.152762Z ERROR sshx::controller: disconnected, retrying in 1s... err=status: Internal, message: "h2 protocol error: http2 error: stream error received: unspecific protocol error detected", details: [], metadata: MetadataMap { headers: {} }

Here's the log with RUST_LOG=trace: trace_sshx.log

Any help would be much appreciated!

@Hugo1380
Copy link

Hugo1380 commented Nov 6, 2023

Adding this to the server section seems to fix it:

location /sshx.SshxService/Channel {
   grpc_pass      grpc://127.0.0.1:8051;
}

@shumvgolove
Copy link
Author

Yep, this directive fixes the issue. Thanks a lot!

Maybe we should add the working nginx.conf example to wiki?

@natarajan0007
Copy link

release]$ ./sshx --server https://localhost:8003
2023-11-07T19:15:11.596704Z ERROR sshx: transport error
image

Caused by:
0: error trying to connect: invalid peer certificate: UnknownIssuer
1: invalid peer certificate: UnknownIssuer
[muthUn@brlb release]$

As per your suggestion i was changed the below line of code in cargo.toml file still i am getting certificate error message.

diff --git a/Cargo.toml b/Cargo.toml index 4b0c58e..c028b8d 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -19,7 +19,7 @@ rand = "0.8.5" serde = { version = "1.0.188", features = ["derive", "rc"] } tokio = { version = "1.32.0", features = ["full"] } tokio-stream = { version = "0.1.14", features = ["sync"] } -tonic = { version = "0.10.0", features = ["tls", "tls-webpki-roots"] } +tonic = { version = "0.10.0", features = ["tls", "tls-roots"] } tracing = "0.1.37" tracing-subscriber = { version = "0.3.17", features = ["env-filter"] }

@shumvgolove

@ekzhang
Copy link
Owner

ekzhang commented Nov 8, 2023

Yes, you'll have to grpc_pass all requests to suffixes of /sshx.SshxService.

Actually on the server it internally just checks if the content-type is application/grpc to decide whether to handle it as an HTTP request or a gRPC request. But since all gRPC requests go to SshxService, matching by path would work fine.

let svc = Steer::new(
[http_service, grpc_service],
|req: &Request<Body>, _services: &[_]| {
let headers = req.headers();
match headers.get(CONTENT_TYPE) {
Some(content) if content == "application/grpc" => 1,
_ => 0,
}
},
);

To clarify, you don't necessarily need to detect gRPC when you're putting a reverse proxy in front of sshx. You just need to forward HTTP/2 connections to it as the backend protocol. But Nginx doesn't let you do h2c (HTTP/2 cleartext) backend for its reverse proxy; it only has grpc_pass as a special case.

Sorry about this, Nginx is just eternally difficult to configure 😅

@natarajan0007
Copy link

this is what i have in my nginx.conf . I am still facing certificate issue. would you mind sharing your nginx setup @shumvgolove

$ ./sshx --server https://localhost:8002
2023-11-09T09:39:28.982734Z ERROR sshx: transport error

Caused by:
0: error trying to connect: invalid peer certificate: UnknownIssuer
1: invalid peer certificate: UnknownIssuer

`events {}

pid nginx.pid;

http {
access_log /dev/stdout;

ssl_certificate ./localhost.pem;
ssl_certificate_key ./localhost-key.pem;
ssl_protocols	TLSv1.2 TLSv1.3;
ssl_prefer_server_ciphers on;
ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;

client_body_temp_path .;
proxy_temp_path .;
fastcgi_temp_path .;
uwsgi_temp_path .;
scgi_temp_path .;

client_body_timeout 5;

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

server {
    server_name localhost 127.0.0.1;
    listen 8003 ssl http2;

    location /sshx.SshxService {
        grpc_pass      grpc://127.0.0.1:8051;
    }

    location /api {
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_set_header Host $http_host;
        proxy_pass   http://127.0.0.1:8051;
    }
}

}
`

@natarajan0007
Copy link

I tried without nginx . Its working fine. But i will try to see if i can able to make with nginx.

image

@juiiyang
Copy link

Hey 👋 and thanks for the great project!

After several attempts to figure out how to make it work with nginx reverse-proxy, I've created half-working solution. I am providing below steps to reproduce the issue and ultimately figure out what is the problem.

sshx-server setup

  1. Create and enter temp folder:
mkdir /tmp/reproduce-issue && cd /tmp/reproduce-issue
  1. Clone and enter the sshx repository:
git clone https://github.com/ekzhang/sshx.git && cd sshx
  1. Build frontend:
npm ci && npm run build
  1. Build backend:
cargo build --release --bin sshx-server
  1. Launch sshx (which will also serve frontend).:
mv target/release/sshx-server . && ./sshx-server --secret dev-secret --listen :: --override-origin https://localhost:8080

nginx setup

  1. Create and enter nginx folder:
mkdir /tmp/reproduce-issue/nginx && cd /tmp/reproduce-issue/nginx
  1. Create nginx.conf with the following content:
# Run nginx using:
#     nginx -p $PWD -e stderr -c nginx.conf

daemon off;  # run in foreground

events {}

pid nginx.pid;

http {
    access_log /dev/stdout;

    # Locally generated SSL certificates from mkcert
    ssl_certificate ./localhost+1.pem;
    ssl_certificate_key ./localhost+1-key.pem;
    ssl_protocols	TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers on;
    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;

    # Directories nginx needs configured to start up.
    client_body_temp_path .;
    proxy_temp_path .;
    fastcgi_temp_path .;
    uwsgi_temp_path .;
    scgi_temp_path .;
    
    ###################
    client_body_timeout 5;  # <------ makes sshx client spitting error log every 5 seconds
    ###################

    # Connection upgrade variable for websocket
    map $http_upgrade $connection_upgrade {  
        default upgrade;
        ''      close;
    }

    server {
        server_name localhost 127.0.0.1;
        listen 8080 ssl http2;

        location / {
            proxy_pass   http://127.0.0.1:8051;
        }

        location /api {
            # Websocket headers
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection $connection_upgrade;
            proxy_set_header Host $http_host;
            proxy_pass   http://127.0.0.1:8051;
        }
    }
}
  1. Setup local certificates with mkcert (otherwise grpc/websockets won't work) and assign correct permissions:
sudo mkcert -install && sudo mkcert localhost 127.0.0.1 && sudo chown $USER:$USER *.pem
  1. Launch nginx:
nginx -p $PWD -e stderr -c nginx.conf

sshx setup

The default sshx binaries won't work, because tonic is compiled with tls-webpki-roots feature, which means that sshx will not lookup OS ca-certificates bundle, making our self-signed certificates useless and throwing the following error:

❯ ./sshx --server https://localhost:8080
2023-11-06T12:15:26.058671Z ERROR sshx: transport error

Caused by:
    0: error trying to connect: invalid peer certificate: UnknownIssuer
    1: invalid peer certificate: UnknownIssuer

In order to fix that we need to recompile sshx binary with tls-roots feature (which will lookup OS ca-certificates):

  1. Enter sshx cloned git repository:
cd /tmp/reproduce-issue/sshx
  1. Save the following content as tonic_fix.patch:
diff --git a/Cargo.toml b/Cargo.toml
index 4b0c58e..c028b8d 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -19,7 +19,7 @@ rand = "0.8.5"
 serde = { version = "1.0.188", features = ["derive", "rc"] }
 tokio = { version = "1.32.0", features = ["full"] }
 tokio-stream = { version = "0.1.14", features = ["sync"] }
-tonic = { version = "0.10.0", features = ["tls", "tls-webpki-roots"] }
+tonic = { version = "0.10.0", features = ["tls", "tls-roots"] }
 tracing = "0.1.37"
 tracing-subscriber = { version = "0.3.17", features = ["env-filter"] }

and apply it like so:

git apply tonic_fix.patch
  1. Compile modified sshx:
cargo build --release --bin sshx
  1. Launch client:
mv target/release/sshx . && ./sshx --server https://localhost:8080

Now, here's the issue: there's no connection between server and shell and I'm unable to spawn terminal:

image

And once in a while sshx produces the following log:

2023-11-06T14:25:05.152762Z ERROR sshx::controller: disconnected, retrying in 1s... err=status: Internal, message: "h2 protocol error: http2 error: stream error received: unspecific protocol error detected", details: [], metadata: MetadataMap { headers: {} }

Here's the log with RUST_LOG=trace: trace_sshx.log

Any help would be much appreciated!

Thanks for your guide, my problem is solved.

By the way, self signed cert need to add into keychain in Mac OS

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants