Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WebSocket connections being disconnected after 100 seconds #5539

Closed
darnfish opened this issue May 11, 2020 · 1 comment
Closed

WebSocket connections being disconnected after 100 seconds #5539

darnfish opened this issue May 11, 2020 · 1 comment
Labels
kind/bug Categorizes issue or PR as related to a bug.

Comments

@darnfish
Copy link

darnfish commented May 11, 2020

NGINX Ingress controller version: 0.32.0

Kubernetes version (use kubectl version): v1.16.6

Environment: WebSocket server (Mega running on DigitalOcean Kubernetes behind a DO LoadBalancer

  • Cloud provider or hardware configuration: 2 GB Memory / 50 GB Disk + 23 GB / NYC3 - Debian do-kube-1.16.6-do.2
  • OS (e.g. from /etc/os-release): Debian
  • Kernel (e.g. uname -a): Darwin Williams-MacBook-Pro.local 19.4.0 Darwin Kernel Version 19.4.0: Wed Mar 4 22:28:40 PST 2020; root:xnu-6153.101.6~15/RELEASE_X86_64 x86_64 (development)
  • Install tools: kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-0.32.0/deploy/static/provider/do/deploy.yaml

What happened:
The WebSocket was closed with code 1006 after 100 seconds

What you expected to happen:
The WebSocket connection would stay open for longer than 100s

How to reproduce it:
Create a DigitalOcean cluster behind a DigitalOcean LoadBalancer (also behind Cloudflare proxy) and deploy a WebSocket server (specifically Mega) with the following Ingress config:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx

    nginx.org/websocket-services: gateway
    nginx.ingress.kubernetes.io/websocket-services: gateway

    ingress.kubernetes.io/proxy-read-timeout: "3600"
    ingress.kubernetes.io/proxy-send-timeout: "3600"
    ingress.kubernetes.io/proxy-connect-timeout: "3600"

    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "3600"
  name: gateway-redacted-chat
spec:
  rules:
    - host: gateway.redacted.chat
      http:
        paths:
          - path: /
            backend:
              serviceName: gateway
              servicePort: 80

Anything else we need to know:
Initially connections were timing out after 60s. This wasn't fixed by adding nginx.ingress.kubernetes.io/proxy-read-timeout and nginx.ingress.kubernetes.io/proxy-send-timeout, but weirdly it was fixed via ingress.kubernetes.io/proxy-read-timeout, ingress.kubernetes.io/proxy-send-timeout and ingress.kubernetes.io/proxy-connect-timeout. After this, connections started being dropped due to WS code 1006 after 100 seconds.

I have port-forwarded the WebSocket server and connected to it locally without any connection drop issues, so I'm confident the issue resides in either the DigitalOcean LoadBalancer or in ingress-nginx.

/kind bug

@darnfish darnfish added the kind/bug Categorizes issue or PR as related to a bug. label May 11, 2020
@darnfish
Copy link
Author

After further testing it seems like it's an issue with CloudFlare, sorry for the hassle!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Development

No branches or pull requests

1 participant