Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Netbird behind nginx: stream terminated by RST_STREAM #651

Open
ykorzikowski opened this issue Jan 11, 2023 · 9 comments
Open

Netbird behind nginx: stream terminated by RST_STREAM #651

ykorzikowski opened this issue Jan 11, 2023 · 9 comments

Comments

@ykorzikowski
Copy link

ykorzikowski commented Jan 11, 2023 •
edited

Hey there,

I am successfully running netbird since couple of months. Some clients cant speak with each other. I am investigating this issue right now and found, that I have errors regarding the signal-service in my client-logs (all clients, also the working ones):

time="2023-01-11T09:50:08+01:00" level=warning msg="disconnected from the Signal service but will retry silently. Reason: rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: PROTOCOL_ERROR" file="grpc.go:144"
time="2023-01-11T09:50:23+01:00" level=info msg="connected to the Signal Service stream" file="grpc.go:136"
time="2023-01-11T09:51:08+01:00" level=warning msg="disconnected from the Signal service but will retry silently. Reason: rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: PROTOCOL_ERROR" file="grpc.go:144"
time="2023-01-11T09:51:23+01:00" level=info msg="connected to the Signal Service stream" file="grpc.go:136"

server{
    listen *:80;
    listen [::]:80;
    server_name  ganymede.redacted.com;

    location / {
        return 301 https://$server_name$request_uri;
    }
}

# Dashboard

server {
    include security.conf;
    listen *:443 ssl http2;
    listen [::]:443 ssl http2;
    server_name ganymede.redacted.com;
    ssl_certificate /etc/letsencrypt/live/ganymede.redacted.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/ganymede.redacted.com/privkey.pem;

    access_log /var/log/nginx/ganymede.redacted.com/netbird.log;
    error_log /var/log/nginx/ganymede.redacted.com/err/netbird.log;

    location / {
        proxy_pass http://10.10.11.199:8080;
        proxy_set_header        X-Real-IP $remote_addr;
        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header        X-Scheme $scheme;
	proxy_set_header        X-Forwarded-Proto https;
        proxy_set_header        X-Forwarded-Host ganymede.redacted.com;
    }
}

# MGMT HTTPS API

server {
    include security.conf;
    listen *:4443 ssl http2;
    listen [::]:4443 ssl http2;
    server_name ganymede.redacted.com;
    ssl_certificate /etc/letsencrypt/live/ganymede.redacted.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/ganymede.redacted.com/privkey.pem;

    access_log /var/log/nginx/ganymede.redacted.com/netbird-api.log;
    error_log /var/log/nginx/ganymede.redacted.com/err/netbird-api.log;

    location / {
        proxy_pass http://10.10.11.199:8081;
        proxy_set_header        X-Real-IP $remote_addr;
        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header        X-Scheme $scheme;
	proxy_set_header        X-Forwarded-Proto https;
        proxy_set_header        X-Forwarded-Host ganymede.redacted.com;
    }
}

# MGMT GRPC API

server {
    include security.conf;
    listen *:33073 ssl http2;
    listen [::]:33073 ssl http2;
    server_name ganymede.redacted.com;
    ssl_certificate /etc/letsencrypt/live/ganymede.redacted.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/ganymede.redacted.com/privkey.pem;

    access_log /var/log/nginx/ganymede.redacted.com/netbird-grpc.log;
    error_log /var/log/nginx/ganymede.redacted.com/err/netbird-grpc.log;

    location / {
        grpc_pass grpc://10.10.11.199:33074;
        grpc_read_timeout 3600s;
    }
}

# Signal GRPC API

server {
    include security.conf;
    listen *:10000 ssl http2;
    listen [::]:10000 ssl http2;
    server_name ganymede.redacted.com;
    ssl_certificate /etc/letsencrypt/live/ganymede.redacted.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/ganymede.redacted.com/privkey.pem;

    access_log /var/log/nginx/ganymede.redacted.com/netbird-signal.log;
    error_log /var/log/nginx/ganymede.redacted.com/err/netbird-signal.log;

    location / {
        grpc_pass grpc://10.10.11.199:10001;
        grpc_read_timeout 3600s;
	grpc_ssl_verify off;
    }
}

Is there a recommendation how to run the netbird service? Like my config or everythin on port 443 like #536 ?

Thank you for any help :)
@mlsmaycon mlsmaycon mentioned this issue Mar 6, 2023
Closed
@mlsmaycon
Copy link
Collaborator

This issue is caused by grpc and its native keepalive not being supported by revere-proxy. We are implementing a application keepalive tin #771

@ykorzikowski
Copy link
Author

Just for reference: Found this: camunda-community-hub/zeebe-client-node-js#101

@mlsmaycon
Copy link
Collaborator

@ykorzikowski, we notice that running Nginx v1.25.1 solved the issue. Can you test it?

@ykorzikowski
Copy link
Author

Hellooo,

I dont know. What helped is setting grpc_read_timeout 3600s; to 1 hour.

I am still using nginx version: nginx/1.18.0 and did not notice this issue since tweaking my config with above parameter.

@devopskupryk
Copy link

Hello,

I'm trying to deploy the netbird management into my k8s cluster v.1.25.12 with the nginx ingress controller 1.19.10

Have specified the annotations in the netbird management and netbird signal ingresses:

      nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
      nginx.ingress.kubernetes.io/configuration-snippet: |
        allow all;
        grpc_read_timeout 3600s;
        grpc_send_timeout 3600s;
        grpc_socket_keepalive on;

Peers are registered and the netbird client connected but vpn doesn't work:

$ netbird status
Daemon version: 0.22.7
CLI version: 0.22.7
Management: Connected
Signal: Connected
FQDN: test.hidden.tech
NetBird IP: 100.77.11.161/16
Interface type: Kernel
Peers count: 0/5 Connected

sudo tail -n 30 /var/log/netbird/client.log
2023-09-27T12:08:47+03:00 WARN client/server/server.go:226: canceling previous waiting execution
2023-09-27T12:09:34+03:00 INFO client/internal/login.go:130: peer has been successfully registered on Management Service
2023-09-27T12:09:35+03:00 INFO client/internal/wgproxy/proxy_ebpf.go:79: local wg proxy listening on: 3128
2023-09-27T12:09:35+03:00 INFO iface/tun_linux.go:15: create tun interface with kernel WireGuard support: wt0
2023-09-27T12:09:50+03:00 INFO signal/client/grpc.go:157: connected to the Signal Service stream
2023-09-27T12:09:50+03:00 INFO client/internal/connect.go:179: Netbird engine started, my IP is: 100.77.11.161/16
2023-09-27T12:09:50+03:00 INFO management/client/grpc.go:143: connected to the Management Service stream
2023-09-27T12:09:50+03:00 INFO client/internal/dns/systemd_linux.go:135: adding 1 search domains and 0 match domains. Search list: [vpn.stage.heddin.tech] , Match list: []
2023-09-27T12:09:50+03:00 INFO client/internal/acl/manager.go:67: ACL rules processed in: 1.365082ms, total rules count: 2
2023-09-27T12:10:35+03:00 WARN signal/client/grpc.go:170: disconnected from the Signal service but will retry silently. Reason: rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: PROTOCOL_ERROR
2023-09-27T12:10:50+03:00 INFO signal/client/grpc.go:157: connected to the Signal Service stream
2023-09-27T12:11:35+03:00 WARN signal/client/grpc.go:170: disconnected from the Signal service but will retry silently. Reason: rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: PROTOCOL_ERROR
2023-09-27T12:11:48+03:00 INFO signal/client/grpc.go:157: connected to the Signal Service stream
2023-09-27T12:12:01+03:00 INFO client/internal/dns/systemd_linux.go:135: adding 1 search domains and 0 match domains. Search list: [vpn.stage.heddin.tech] , Match list: []
2023-09-27T12:12:01+03:00 INFO client/internal/acl/manager.go:67: ACL rules processed in: 461.906µs, total rules count: 2
2023-09-27T12:12:36+03:00 WARN signal/client/grpc.go:170: disconnected from the Signal service but will retry silently. Reason: rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: PROTOCOL_ERROR
2023-09-27T12:12:40+03:00 WARN signal/client/grpc.go:151: disconnected from the Signal Exchange due to an error: rpc error: code = Unavailable desc = connection error: desc = "error reading from server: read tcp 192.168.1.100:59684->hidden:443: read: connection timed out"
2023-09-27T12:12:41+03:00 WARN management/client/grpc.go:158: disconnected from the Management service but will retry silently. Reason: rpc error: code = Unavailable desc = keepalive ping failed to receive ACK within `timeout`

How should I additionally configure my nginx ingress controller and\or ingress resources to solve this issue?

PS. I can't add additionaly:
grpc_pass grpc://$service_name:$service_port;
because nginx config already has:
grpc_pass grpc://$upstream_balancer;
and I get error when try:

Error: cannot patch "netbird-management" with kind Ingress: admission webhook "validate.nginx.ingress.kubernetes.io" denied the request: 
 nginx: [emerg] "grpc_pass" directive is duplicate in /tmp/nginx/nginx-cfg1131325741:4731
 nginx: configuration file /tmp/nginx/nginx-cfg1131325741 test failed

@devopskupryk
Copy link

Any news?

@ykorzikowski
Copy link
Author

Can you post some nginx access / error logs? Maybe this is some ingress configuration issue and noting regarding netbird.

@devopskupryk
Copy link

Nginx ingress controllers' logs don't contain any error:

~ % stern nginx -i netbird
+ nginx-ingress-ingress-nginx-controller-ssvsd › controller
+ nginx-ingress-ingress-nginx-controller-kc929 › controller
+ nginx-ingress-ingress-nginx-controller-znnks › controller
+ nginx-ingress-ingress-nginx-controller-dsphr › controller
+ nginx-ingress-ingress-nginx-controller-qlglz › controller
+ nginx-ingress-ingress-nginx-controller-6jqht › controller
nginx-ingress-ingress-nginx-controller-qlglz controller ip-hidden - - [06/Oct/2023:07:43:22 +0000] "POST /management.ManagementService/GetServerKey HTTP/2.0" 200 61 "-" "grpc-go/1.55.0" 118 0.002 [netbird-netbird-management-80] [] 10.244.3.224:80 91 0.004 200 d93bf7b778f62315eddbce6d13e36540
nginx-ingress-ingress-nginx-controller-qlglz controller ip-hidden - - [06/Oct/2023:07:43:22 +0000] "POST /management.ManagementService/GetPKCEAuthorizationFlow HTTP/2.0" 200 0 "-" "grpc-go/1.55.0" 149 0.002 [netbird-netbird-management-80] [] 10.244.3.224:80 67 0.000 200 55a5a3f501b501a66b96cd41b7a64dec
nginx-ingress-ingress-nginx-controller-qlglz controller ip-hidden - - [06/Oct/2023:07:43:27 +0000] "POST /management.ManagementService/GetServerKey HTTP/2.0" 200 61 "-" "grpc-go/1.55.0" 118 0.002 [netbird-netbird-management-80] [] 10.244.3.224:80 91 0.004 200 2f1504ba52578c47c42aa625646c24b5
nginx-ingress-ingress-nginx-controller-qlglz controller ip-hidden - - [06/Oct/2023:07:43:27 +0000] "POST /management.ManagementService/Login HTTP/2.0" 200 0 "-" "grpc-go/1.55.0" 352 0.002 [netbird-netbird-management-80] [] 10.244.3.224:80 88 0.000 200 3739e538f1cc0de4a30661be8e30c8e8
nginx-ingress-ingress-nginx-controller-dsphr controller ip-hidden - - [06/Oct/2023:07:43:56 +0000] "POST /signalexchange.SignalExchange/Send HTTP/2.0" 200 5 "-" "grpc-go/1.55.0" 254 0.012 [netbird-netbird-signal-80] [] 10.244.3.157:80 47 0.008 200 320af0b7fa14deb408df7a9655d6894e
nginx-ingress-ingress-nginx-controller-dsphr controller ip-hidden - - [06/Oct/2023:07:43:56 +0000] "POST /signalexchange.SignalExchange/Send HTTP/2.0" 200 5 "-" "grpc-go/1.55.0" 217 0.002 [netbird-netbird-signal-80] [] 10.244.3.157:80 47 0.004 200 c9b0464002cf4a0c99090ae3e4dd965b
nginx-ingress-ingress-nginx-controller-dsphr controller ip-hidden - - [06/Oct/2023:07:43:57 +0000] "POST /signalexchange.SignalExchange/Send HTTP/2.0" 200 5 "-" "grpc-go/1.55.0" 217 0.002 [netbird-netbird-signal-80] [] 10.244.3.157:80 47 0.000 200 8ce398045d021141db56fe2d7ab0a40b
nginx-ingress-ingress-nginx-controller-dsphr controller ip-hidden - - [06/Oct/2023:07:44:36 +0000] "POST /signalexchange.SignalExchange/Send HTTP/2.0" 200 5 "-" "grpc-go/1.55.0" 217 0.002 [netbird-netbird-signal-80] [] 10.244.3.157:80 47 0.004 200 89850818ca10896a4cd86d91ca4b0111
nginx-ingress-ingress-nginx-controller-dsphr controller ip-hidden - - [06/Oct/2023:07:44:36 +0000] "POST /signalexchange.SignalExchange/Send HTTP/2.0" 200 5 "-" "grpc-go/1.55.0" 217 0.002 [netbird-netbird-signal-80] [] 10.244.3.157:80 47 0.000 200 97ec32beb45da1830ee71e7062260acf
nginx-ingress-ingress-nginx-controller-dsphr controller ip-hidden - - [06/Oct/2023:07:44:40 +0000] "POST /signalexchange.SignalExchange/ConnectStream HTTP/2.0" 200 0 "-" "grpc-go/1.55.0" 154 60.000 [netbird-netbird-signal-80] [] 10.244.3.157:80 187 60.000 200 33cd1401fdfed21bbafd556ef35317db
nginx-ingress-ingress-nginx-controller-dsphr controller ip-hidden - - [06/Oct/2023:07:45:06 +0000] "POST /signalexchange.SignalExchange/Send HTTP/2.0" 200 5 "-" "grpc-go/1.55.0" 217 0.006 [netbird-netbird-signal-80] [] 10.244.3.157:80 47 0.008 200 5a77a24d9402d063a6a4ecc4a2c99929
nginx-ingress-ingress-nginx-controller-dsphr controller ip-hidden - - [06/Oct/2023:07:45:15 +0000] "POST /signalexchange.SignalExchange/Send HTTP/2.0" 200 5 "-" "grpc-go/1.55.0" 217 0.007 [netbird-netbird-signal-80] [] 10.244.3.157:80 47 0.004 200 1d285753adebe4f1647259ea2d60020c
nginx-ingress-ingress-nginx-controller-dsphr controller ip-hidden - - [06/Oct/2023:07:45:15 +0000] "POST /signalexchange.SignalExchange/Send HTTP/2.0" 200 5 "-" "grpc-go/1.55.0" 217 0.002 [netbird-netbird-signal-80] [] 10.244.3.157:80 47 0.000 200 9fbd16a3312440bcc847bc6f2c93be4e

@devopskupryk
Copy link

Any news?

@xiaolei0125 xiaolei0125 mentioned this issue Dec 6, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@mlsmaycon @ykorzikowski @devopskupryk