issue: Exhausting Workers in Nginx Reverse Proxy in 0.6.6

[borysow]

Check Existing Issues

• I have searched the existing issues and discussions.
• I am using the latest version of Open WebUI.

Installation Method

Pip Install

Open WebUI Version

0.6.6

Ollama Version (if applicable)

No response

Operating System

Ubuntu 22.04 Container

Browser (if applicable)

Google Chrome 121.0.6167.139

Confirmation

• I have read and followed all instructions in README.md.
• I am using the latest version of both Open WebUI and Ollama.
• I have included the browser console logs.
• I have included the Docker container logs.
• I have listed steps to reproduce the bug in detail.

Expected Behavior

I have an Open WebUI instance set up via docker compose to bring up Open WebUI on port 8080 on the internal docker network, and have an nginx reverse proxy configured to establish a secure connection to users outside the host. I believe I based the nginx config on an example in the documentation.

I expect to have this configuration still work to allow end users to get a secure connection by going to https://<docker_host_name>:8080.

Everything works happily on OpenWebUI 0.6.5.

Actual Behavior

After updating to 0.6.6, I can log in briefly, but after a short amount of time my nginx server logs report that xxx "worker_connections are not enough while connecting to upstream" and I get either Error 500 or 503 (Bad Gateway).

(I know it's not the proper solution, but I increased the max_num_workers to 4096 as a test, and all it did was make it take longer to have the errors emerge).

I don't know if this is related to the changes in Websocket mentioned in the changelog.

Below are some logs of the issue as well as my docker compose file and nginx.conf.

Steps to Reproduce

With the following nginx and docker compose setup, performing any activity on the OpenWebUI server (maybe opening a few chats and wandering around the configs) very quickly causes this.

docker-compose:

services:
  # nothing weird in the build, just apt installing the same stuff in the official dockerfile, creating an openwebui user
  # and pip installing openwebui 
  openwebui:
    container_name: openwebui-server-${OPENWEBUI_VERSION}-${OPENWEBUI_ENVIRONMENT}
    image: openwebui-server:${OPENWEBUI_VERSION}
    build:
      context: ../openwebui
      dockerfile: ../docker/dockerfiles/dockerfile_openwebui
      args:
        OPENWEBUI_VERSION: ${OPENWEBUI_VERSION}
    volumes:
      - open-webui:/home/openwebui/app:z
    environment:
      HF_DATASETS_ONLINE: 1
      HF_HUB_OFFLINE: 1
      WEBUI_NAME: ${OPENWEBUI_NAME}
      WEBUI_URL: http://0.0.0.0:8080
      PORT: 8080
      GLOBAL_LOG_LEVEL: INFO

    command: 
      - "serve"
    restart: always
  
  # this is just a vanilla nginx image with .conf and certs added (this is a prototype instance they are temporary self-signed certs)
  nginx:
    container_name: nginx-openwebui-server-${OPENWEBUI_VERSION}-${OPENWEBUI_ENVIRONMENT}
    image: nginx-certs:latest
    build:
      context: ../nginx
      dockerfile: ../docker/dockerfiles/dockerfile_nginx
    ports:
      - ${OPENWEBUI_EXPOSED_PORT}:443
    depends_on:
      - openwebui
    restart: always

I believe I basically copied one of these from the OpenWebUI documentation page somewhere.
nginx.conf

worker_processes  1;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;

    sendfile        on;
    #tcp_nopush     on;

    #keepalive_timeout  0;
    keepalive_timeout  120;

    #gzip  on;

    # needed to properly handle websockets (streaming)
    map $http_upgrade $connection_upgrade {
        default upgrade;
        ''      close;
    }

    # Redirect all HTTP traffic to HTTPS
    server {
        listen 80;
        server_name <docker_host_name>

        return 301 https://$host$request_uri;
    }

    # Handle HTTPS traffic
    server {
        listen 443 ssl;
        server_name <docker_host_name>;

        # SSL Settings (ensure paths are correct)

        ssl_certificate /etc/ssl/certs/cert.pem;
        ssl_certificate_key /etc/ssl/private/key.pem;        
        ssl_protocols TLSv1.2 TLSv1.3;
        ssl_ciphers ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256;
        ssl_prefer_server_ciphers on;

        # OCSP Stapling
        #ssl_stapling on;
        #ssl_stapling_verify on;

        # Proxy settings to your local service
        location / {
            # proxy_pass should point to your running localhost version of open-webui
            proxy_pass http://openwebui:8090;

            # Add WebSocket support (Necessary for version 0.5.0 and up)
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection $connection_upgrade;

            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;

            # (Optional) Disable proxy buffering for better streaming response from models
            proxy_buffering off;

            # (Optional) Increase max request size for large attachments and long audio messages
            client_max_body_size 20M;
            proxy_read_timeout 10m;
        }
    }

}

Logs & Screenshots

OpenWebUI log while this is happening:

openwebui-server-0.6.6-prod             | 2025-05-05 16:46:59.625 | INFO     | uvicorn.protocols.http.httptools_impl:send:476 - 10.14.106.146:0 - "GET /api/v1/chats/d31de0e3-db90-400a-8976-ee075eb5551f HTTP/1.1" 401 - {}                                    
n64; x64; rv:136.0) Gecko/20100101 Firefox/136.0"                                                                                                                                                                                                               
openwebui-server-0.6.6-prod             | 2025-05-05 16:46:59.632 | INFO     | uvicorn.protocols.http.httptools_impl:send:476 - 10.14.106.146:0 - "GET /api/v1/chats/d31de0e3-db90-400a-8976-ee075eb5551f HTTP/1.1" 401 - {}                                    
n64; x64; rv:136.0) Gecko/20100101 Firefox/136.0"                                                                                                                                                                                                               
openwebui-server-0.6.6-prod             | 2025-05-05 16:46:59.698 | INFO     | uvicorn.protocols.http.httptools_impl:send:476 - 10.14.106.146:0 - "GET /api/v1/chats/d31de0e3-db90-400a-8976-ee075eb5551f HTTP/1.1" 401 - {}                                    
n64; x64; rv:136.0) Gecko/20100101 Firefox/136.0"                                                                                                                                                                                                               
openwebui-server-0.6.6-prod             | 2025-05-05 16:46:59.715 | INFO     | uvicorn.protocols.http.httptools_impl:send:476 - 10.14.106.146:0 - "GET /api/v1/chats/e932c15b-8fa1-4b8a-b770-03020d7800e3 HTTP/1.1" 401 - {}                                    
n64; x64; rv:136.0) Gecko/20100101 Firefox/136.0"                                                                                                                                                                                                               
openwebui-server-0.6.6-prod             | 2025-05-05 16:47:22.059 | INFO     | uvicorn.protocols.http.httptools_impl:send:476 - 10.14.106.72:0 - "GET /api/config HTTP/1.1" 200 - {}       

Nginx log while this is happening:

nginx-openwebui-server-0.6.6-prod       | 10.14.106.72 - - [05/May/2025:16:47:22 +0000] "GET /api/config HTTP/1.1" 200 2720 "https://<docker_host_name>:8080/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36"      
nginx-openwebui-server-0.6.6-prod       | 2025/05/05 16:47:29 [alert] 29#29: *5946 4096 worker_connections are not enough while connecting to upstream, client: 10.14.106.72, server: <docker_host_name>, request: "GET /api/config HTTP/1.1", upstream: "h
ttp://172.22.0.3:8080/api/config", host: "<docker_host_name>:8080", referrer: "https://<docker_host_name>:8080/"                                                                                                                                                                        
nginx-openwebui-server-0.6.6-prod       | 10.14.106.72 - - [05/May/2025:16:47:29 +0000] "GET /api/config HTTP/1.1" 500 579 "https://external_host:8080/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36"       
nginx-openwebui-server-0.6.6-prod       | 10.14.106.253 - - [05/May/2025:16:47:29 +0000] "GET /ws/socket.io/?EIO=4&transport=websocket HTTP/1.1" 500 579 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 
Safari/537.36 Edg/135.0.0.0"                                                                                                                                                                                                                                    
nginx-openwebui-server-0.6.6-prod       | 2025/05/05 16:47:29 [alert] 29#29: *5948 4096 worker_connections are not enough while connecting to upstream, client: 10.14.106.253, server: <docker_host_name>, request: "GET /ws/socket.io/?EIO=4&transport=web
socket HTTP/1.1", upstream: "http://172.22.0.3:8080/ws/socket.io/?EIO=4&transport=websocket", host: "<docker_host_name>:8080"  
nginx-openwebui-server-0.6.6-prod       | 2025/05/05 16:47:35 [alert] 29#29: 4096 worker_connections are not enough

Additional Information

No response

[borysow]

First off, I appreciate all the work you guys put in. OpenWebUI is a great piece of software and I deeply appreciate it.

I apologize, as this isn't my area of expertise. However, an otherwise working configuration completely fails just by updating OpenWebUI from 0.6.5 to 0.6.6. Has there been a change in how connections are handled on the OpenWebUI side? I'm entirely unfamiliar with Websockets, but is it possible that the new openwebui version causing some sort of connection leakage?

[yshishenya]

I have the same problem after update

[LukeDC76]

I also have the same issue. This change breaks my multi tenant NPM instance for my other working proxied projects after update to 0.6.7

[Ithanil]

Yes, see graph here with before/after update:

Was in the late night, not even any users online.

[wolfaba]

In our installation, we use apache as reverse proxy and a few seconds after the container start the apache process count increases to limit. Connections from apache to container keep open (established), therefore blocking apache for other requests. Downgrade to 0.6.5 solves this issue.

[wolfaba]

[tjbck]

This may have something to do with THREAD_POOL_SIZE introduced, investigating!

[Classic298]

Hmm okay now I understand. This is odd. What would create new connections, if not a user (or service) accessing Open WebUI?

[Ithanil]

Check your access logs and tell me you don't see countless random accesses from bots on the internet 😉

[wolfaba]

@Classic298 We are monitoring our service, so monitoring software creates new connections too. And as @Ithanil said, there are many bots and even script kiddies running scanning scripts create new connections.

[Classic298]

@Ithanil I don't because it's available only on intranet, not internet :)

[Classic298]

Hate to say it: i updated and it seems i again and again get 500 errors. I prepared in advance, optimizing my nginx config and throwing away old connections quickly but... Yeah.. this is a serious issue. 500 errors keep happening regardless of what values you choose.

Even if you set the thread pool size env var to high numbers it keeps happening.

If you didn't upgrade yet: don't.

[Ithanil]

Can anyone tell if #13364 could have something to do with this?

[phpia]

0.6.6 same issue, downgrade to 0.6.5 needed

[borysow]

So, I managed to upgrade. Sonnet made some suggestions, and I have no idea if they actually fix the issue or if it's just masked, but it functions now.

There are the modifications I made to my nginx.conf:

events {
    # worker_connections  1024;
    worker_connections  8192;
    # this next line is new
    multi_accept on;
}

http {
## snip ###

    #keepalive_timeout  0;
    keepalive_timeout  120;

## snip ## 
    # Upstream block with keepalive  (this block is new)
    upstream openwebui_backend {
        server openwebui:8080;
        keepalive 32;  # Adjust based on backend capacity
    }

server {
## snip ###
## modified proxy_pass to point to the upstream

            # proxy_pass http://openwebui:8080;
            proxy_pass http://openwebui_backend;

### set these new timeout values
            # new timeouts
            proxy_connect_timeout 75s;
            proxy_send_timeout 300s;
            proxy_read_timeout 300s;
}

Hope that's parseable...

Effectively:

In the events block:

1. raised worker_connections from 1024 to 8192
2. set multi_accept on

In the http block,

1. changed keepalive_timeout from 0 to 120
2. added this upstream:

    # Upstream block with keepalive  (this block is new)
    upstream openwebui_backend {
        server openwebui:8080;
        keepalive 32;  # Adjust based on backend capacity
    }

In the server_location block:

1. changed proxy_pass to point to the upstream
2. set proxy_connect_timeout to 75s
3. set proxy_send_timeout to 300s
4. set proxy_read_timeout to 300s

---

Again, I'm really new to this, but this at least is allowing me to keep the instance running and up-to-date.

[Classic298]

I did similar changes - it does not fix it, only make the issue smaller and more rarely to occur.

[Classic298]

Does anyone of you have any logs of the 500 error?
Would be very helpful to identify the root cause.
Unfortunately for me, I cannot see any WARNING or ERROR Messages in my logs, that indicate the 500 error message i receive on my client side.

Please share if you have logs!

@borysow @phpia @wolfaba @yshishenya @LukeDC76

[rgaricano]

Nop, fine here.
with redis and postgresql
relevants env vars:

...
# websockets -------------------
ENABLE_WEBSOCKET_SUPPORT="true"
WEBSOCKET_MANAGER="redis"
WEBSOCKET_REDIS_URL="redis://localhost:6379/0"
REDIS_URL="redis://localhost:6379/0"

# databases -------------------

VECTOR_DB="pgvector"
PGVECTOR_DB_URL="postgresql://openwebui:xxx@localhost:5432/openwebuipg_test"
DATABASE_URL="postgresql://openwebui:xxx@localhost:5432/openwebuidb_test"
DATABASE_POOL_SIZE="50"
DATABASE_POOL_MAX_OVERFLOW="50"
DATABASE_POOL_TIMEOUT="30"
...

nginx:

limit_conn_zone $binary_remote_addr zone=perip:15m;
limit_conn_zone $server_name zone=perserver:15m;

server {
	server_name _;
	listen 80 default_server;
	listen [::]:80 default_server;
	return 301 https://$host$request_uri;
}

#  upstreamed openwebui server
    upstream dyn_openwebui {
	    zone upstream_openwebui 64k;
		ip_hash;
#       zone upstreams 64K;
#options if more than 1 server
#    server backend1.example.com weight=5 max_conns=0 max_fails=0 fail_timeout=10s;
        server 127.0.0.1:8080;
        keepalive 64;
		keepalive_time 1h;
		keepalive_requests 100;
		keepalive_timeout 3600s;
    }
#    upstream node_mcpo {
#        zone upstreams 64K;
#        server 127.0.0.1:8000 max_fails=1 fail_timeout=2s;
#        keepalive 2;
#    }


server {
	server_name torre-nueva.org www.torre-nueva.org openwebui.torre-nueva.org;
	listen 443 ssl;
    listen [::]:443 ssl http2;

#add_header Content-Security-Policy "default-src 'self' https: data: 'unsafe-inline' 'unsafe-eval';" always;
# 	add_header Content-Security-Policy "default-src *://*.torre-nueva.org:*; connect-src *; script-src *; unsafe-inline *;" always;
#		add_header Content-Security-Policy "default-src *; connect-src *; script-src *; unsafe-eval *; unsafe-inline *;" always;
#		add_header Content-Security-Policy "script-src self:unsafe-eval:unsafe-inline;" always;
#	add_header Content-Security-Policy "default-src 'self'; connect-src *; script-src 'self'; unsafe-eval 'self'; unsafe-inline 'self';" always;
# managed by Certbot
    ssl_certificate /etc/letsencrypt/live/www.torre-nueva.org/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/www.torre-nueva.org/privkey.pem; # managed by Certbot
 #_
    limit_conn perip 20;
    limit_conn perserver 500;

	location / {
		proxy_pass https://dyn_openwebui;
#		proxy_pass https://127.0.0.1:8080;

		proxy_set_header Host $host;
		proxy_set_header X-Real-IP $remote_addr;
		proxy_set_header Accept-Encoding "";
		proxy_set_header X-Forwarded-Scheme $scheme;
		proxy_set_header X-Forwarded-Proto $scheme;
		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

# Websockets
		proxy_http_version 1.1;
		proxy_set_header Upgrade $http_upgrade;
		proxy_set_header Connection "upgrade";
	##
	# Disable buffering for the streaming responses (SSE)
		chunked_transfer_encoding off;
		proxy_cache off;
		proxy_buffering on;
		proxy_buffers 10 16k;
		proxy_buffer_size 32k;
	##
	# Conections Timeouts (1hr)
        keepalive_timeout 3600s;
		proxy_connect_timeout 3600s;
		proxy_read_timeout 3600s;
		proxy_send_timeout 3600s;
	##
	}
}

nginx.conf

user www-data;
worker_processes auto;
pid /run/nginx.pid;
error_log /var/log/nginx/error.log;
include /etc/nginx/modules-enabled/*.conf;

events {
	worker_connections 768;
	# multi_accept on;
}

http {
client_max_body_size 100M;
	##
	# Basic Settings
	##

	sendfile on;
	tcp_nopush on;
	types_hash_max_size 4096;
	# server_tokens off;

	# server_names_hash_bucket_size 64;
	# server_name_in_redirect off;

	include /etc/nginx/mime.types;
	default_type application/octet-stream;

	##
	# SSL Settings
	##

	ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE
	ssl_prefer_server_ciphers on;

	##
	# Logging Settings
	##

	access_log /var/log/nginx/access.log;

	##
	# Gzip Settings
	##

	gzip on;

	# gzip_vary on;
	# gzip_proxied any;
	# gzip_comp_level 6;
	# gzip_buffers 16 8k;
	# gzip_http_version 1.1;
	# gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;
limit_conn_zone $binary_remote_addr zone=concurrent:10m;

    limit_conn_zone $binary_remote_addr zone=conn_limit_per_ip:5m;
    limit_req_zone $binary_remote_addr zone=req_limit_per_ip:5m rate=5r/s;
            limit_conn concurrent 100;
	##
	# Virtual Host Configs
	##
	include /etc/nginx/conf.d/*.conf;
	include /etc/nginx/sites-enabled/*;
}

[wolfaba]

@Classic298

Does anyone of you have any logs of the 500 error? Would be very helpful to identify the root cause. Unfortunately for me, I cannot see any WARNING or ERROR Messages in my logs, that indicate the 500 error message i receive on my client side.

We get no 500 errors, neither warnings nor errors. The only error message in error.log is AH00161: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting and apache stops responding to new requests because of all available workers depleted and having connections open.

[LukeDC76]

This is how ours looked. Anonymised logs from our NPM instance.

2025/05/07 09:15:33 [alert] 196#196: *11777 512 worker_connections are not enough while connecting to upstream, client: [CLIENT_IP_REDACTED], server: [SERVER_DOMAIN_REDACTED], request: "GET /_app/immutable/chunks/Help.Bj7EGghJ.js HTTP/2.0", upstream: "[UPSTREAM_REDACTED]/_app/immutable/chunks/Help.Bj7EGghJ.js", host: "[HOST_REDACTED]"
2025/05/07 09:15:33 [alert] 196#196: *11777 512 worker_connections are not enough while connecting to upstream, client: [CLIENT_IP_REDACTED], server: [SERVER_DOMAIN_REDACTED], request: "GET /_app/immutable/chunks/FilesOverlay.B8CaAFc5.js HTTP/2.0", upstream: "[UPSTREAM_REDACTED]/_app/immutable/chunks/FilesOverlay.B8CaAFc5.js", host: "[HOST_REDACTED]"
2025/05/07 09:15:33 [alert] 196#196: *11777 512 worker_connections are not enough while connecting to upstream, client: [CLIENT_IP_REDACTED], server: [SERVER_DOMAIN_REDACTED], request: "GET /_app/immutable/chunks/transformers.B_i_4WXY.js HTTP/2.0", upstream: "[UPSTREAM_REDACTED]/_app/immutable/chunks/transformers.B_i_4WXY.js", host: "[HOST_REDACTED]"
2025/05/07 09:15:33 [alert] 196#196: *11777 512 worker_connections are not enough while connecting to upstream, client: [CLIENT_IP_REDACTED], server: [SERVER_DOMAIN_REDACTED], request: "GET /_app/immutable/chunks/User.B9XcpnTG.js HTTP/2.0", upstream: "[UPSTREAM_REDACTED]/_app/immutable/chunks/User.B9XcpnTG.js", host: "[HOST_REDACTED]"
2025/05/07 09:15:33 [alert] 196#196: *11777 512 worker_connections are not enough while connecting to upstream, client: [CLIENT_IP_REDACTED], server: [SERVER_DOMAIN_REDACTED], request: "GET /_app/immutable/chunks/languages.DejWixTG.js HTTP/2.0", upstream: "[UPSTREAM_REDACTED]/_app/immutable/chunks/languages.DejWixTG.js", host: "[HOST_REDACTED]", referrer: "[REFERRER_REDACTED]"
2025/05/07 09:15:33 [alert] 196#196: *11777 512 worker_connections are not enough while connecting to upstream, client: [CLIENT_IP_REDACTED], server: [SERVER_DOMAIN_REDACTED], request: "GET /_app/immutable/chunks/index.B-L1dpqH.js HTTP/2.0", upstream: "[UPSTREAM_REDACTED]/_app/immutable/chunks/index.B-L1dpqH.js", host: "[HOST_REDACTED]"
...

[Classic298]

Related merged PRs (?)
#12958
#13364
#13031

Related unmerged PR (?)
#13027

[Youggls]

PR #12958 was initiated by me, and I tested this modification on my instance without observing similar issues.

[spammenotinoz]

I don't seem impacted, here is an extract from my nginx.conf
Due to the use of websockets, your worker connections seem way to low, I am at 65k for only a small amount of users.

worker_processes auto;
events {
use epoll;
worker_connections 65536;
multi_accept on;
}

[spammenotinoz]

Please see @wolfaba post from 3 days ago. This is something that was introduced through a change between 0.6.5 & 0.6.6 Downgrade to 0.6.5 resolves the workers issue.

I get that release took it over the edge, but would recommend you consider setting worker_processes to auto, increasing the connections and enabling multi_accept.
This application relies on web-sockets, if you don't you will be living on the edge, waiting for the next change to break your deployment.
Even with that massive jump in connections, ram usage is tiny.

[Ithanil]

The point is that that jump occurs without any active users, at least on public facing instances, and that is a bug in my opinion.

[wolfaba]

@LukeDC76 yes, we have downgraded to 0.6.5

[wolfaba]

@spammenotinoz

This application relies on web-sockets, if you don't you will be living on the edge, waiting for the next change to break your deployment.

Using web-sockets is OK. But keeping unsued connection open is not OK. Have been web-sockets activated in 0.6.6 and there are no web-sockets in 0.6.5? Version 0.6.5 works fine. Why cannot next versions work the same way?

Either client can somehow tell the nginx/reverse proxy to reuse already opened connection, or if not possible to reuse it, the openwebui should close the connection. What is the reason to keep connections open without using it?

[wolfaba]

@spammenotinoz one note more: with version 0.6.5 we can still see about 60 long time open connections between apache and openwebui (first column contains $( date +%s) - $( stat --format '%Z' /proc/PID/fd/FD ) seconds). So there are opened connections, but apache can reuse them. Maybe the problem is, that openwebui does not inform client that one connection is long term and client closes the connection to apache and for next request opens new connection, keeping the old connection between apache and openwebui open?

940sec   |  ESTAB  0  0  127.0.0.1:47792  127.0.0.1:11300  users:(("apache2",pid=710904,fd=19))
940sec   |  ESTAB  0  0  127.0.0.1:53496  127.0.0.1:11300  users:(("apache2",pid=710533,fd=20))
940sec   |  ESTAB  0  0  127.0.0.1:54472  127.0.0.1:11300  users:(("apache2",pid=710195,fd=20))
1262sec  |  ESTAB  0  0  127.0.0.1:38712  127.0.0.1:11300  users:(("apache2",pid=710324,fd=20))
1262sec  |  ESTAB  0  0  127.0.0.1:49266  127.0.0.1:11300  users:(("apache2",pid=710413,fd=20))
1340sec  |  ESTAB  0  0  127.0.0.1:38506  127.0.0.1:11300  users:(("apache2",pid=709980,fd=20))
1411sec  |  ESTAB  0  0  127.0.0.1:50304  127.0.0.1:11300  users:(("apache2",pid=709780,fd=20))
1411sec  |  ESTAB  0  0  127.0.0.1:51888  127.0.0.1:11300  users:(("apache2",pid=709782,fd=20))
1615sec  |  ESTAB  0  0  127.0.0.1:54928  127.0.0.1:11300  users:(("apache2",pid=706511,fd=21))
1654sec  |  ESTAB  0  0  127.0.0.1:46014  127.0.0.1:11300  users:(("apache2",pid=708245,fd=19))
1657sec  |  ESTAB  0  0  127.0.0.1:46050  127.0.0.1:11300  users:(("apache2",pid=708204,fd=19))
1760sec  |  ESTAB  0  0  127.0.0.1:35220  127.0.0.1:11300  users:(("apache2",pid=706628,fd=20))
1760sec  |  ESTAB  0  0  127.0.0.1:46788  127.0.0.1:11300  users:(("apache2",pid=706633,fd=20))
1760sec  |  ESTAB  0  0  127.0.0.1:53664  127.0.0.1:11300  users:(("apache2",pid=706634,fd=20))
1760sec  |  ESTAB  0  0  127.0.0.1:56108  127.0.0.1:11300  users:(("apache2",pid=706632,fd=20))
1761sec  |  ESTAB  0  0  127.0.0.1:49954  127.0.0.1:11300  users:(("apache2",pid=706623,fd=20))
1761sec  |  ESTAB  0  0  127.0.0.1:55630  127.0.0.1:11300  users:(("apache2",pid=706626,fd=20))
1762sec  |  ESTAB  0  0  127.0.0.1:35214  127.0.0.1:11300  users:(("apache2",pid=706624,fd=19))
1796sec  |  ESTAB  0  0  127.0.0.1:51820  127.0.0.1:11300  users:(("apache2",pid=706611,fd=19))
1797sec  |  ESTAB  0  0  127.0.0.1:36614  127.0.0.1:11300  users:(("apache2",pid=706615,fd=19))
1797sec  |  ESTAB  0  0  127.0.0.1:50186  127.0.0.1:11300  users:(("apache2",pid=706610,fd=19))
1797sec  |  ESTAB  0  0  127.0.0.1:51766  127.0.0.1:11300  users:(("apache2",pid=706608,fd=19))
1797sec  |  ESTAB  0  0  127.0.0.1:51774  127.0.0.1:11300  users:(("apache2",pid=706607,fd=19))
1797sec  |  ESTAB  0  0  127.0.0.1:51780  127.0.0.1:11300  users:(("apache2",pid=706613,fd=19))
1797sec  |  ESTAB  0  0  127.0.0.1:51784  127.0.0.1:11300  users:(("apache2",pid=706618,fd=19))
1797sec  |  ESTAB  0  0  127.0.0.1:51786  127.0.0.1:11300  users:(("apache2",pid=706614,fd=19))
1797sec  |  ESTAB  0  0  127.0.0.1:51790  127.0.0.1:11300  users:(("apache2",pid=706621,fd=19))
1797sec  |  ESTAB  0  0  127.0.0.1:51806  127.0.0.1:11300  users:(("apache2",pid=706620,fd=19))
1797sec  |  ESTAB  0  0  127.0.0.1:58704  127.0.0.1:11300  users:(("apache2",pid=706612,fd=19))
1798sec  |  ESTAB  0  0  127.0.0.1:50574  127.0.0.1:11300  users:(("apache2",pid=706567,fd=20))
1798sec  |  ESTAB  0  0  127.0.0.1:51610  127.0.0.1:11300  users:(("apache2",pid=706559,fd=19))
1798sec  |  ESTAB  0  0  127.0.0.1:51624  127.0.0.1:11300  users:(("apache2",pid=706562,fd=19))
1798sec  |  ESTAB  0  0  127.0.0.1:51636  127.0.0.1:11300  users:(("apache2",pid=706563,fd=19))
1798sec  |  ESTAB  0  0  127.0.0.1:51652  127.0.0.1:11300  users:(("apache2",pid=706570,fd=19))
1798sec  |  ESTAB  0  0  127.0.0.1:51668  127.0.0.1:11300  users:(("apache2",pid=706561,fd=19))
1798sec  |  ESTAB  0  0  127.0.0.1:51674  127.0.0.1:11300  users:(("apache2",pid=706571,fd=19))
1798sec  |  ESTAB  0  0  127.0.0.1:51706  127.0.0.1:11300  users:(("apache2",pid=706573,fd=19))
1798sec  |  ESTAB  0  0  127.0.0.1:51714  127.0.0.1:11300  users:(("apache2",pid=706574,fd=19))
1798sec  |  ESTAB  0  0  127.0.0.1:51726  127.0.0.1:11300  users:(("apache2",pid=706566,fd=19))
1798sec  |  ESTAB  0  0  127.0.0.1:51738  127.0.0.1:11300  users:(("apache2",pid=706569,fd=19))
1798sec  |  ESTAB  0  0  127.0.0.1:51754  127.0.0.1:11300  users:(("apache2",pid=706572,fd=19))
1798sec  |  ESTAB  0  0  127.0.0.1:51756  127.0.0.1:11300  users:(("apache2",pid=706560,fd=19))
1799sec  |  ESTAB  0  0  127.0.0.1:38222  127.0.0.1:11300  users:(("apache2",pid=706527,fd=19))
1799sec  |  ESTAB  0  0  127.0.0.1:49714  127.0.0.1:11300  users:(("apache2",pid=706521,fd=19))
1799sec  |  ESTAB  0  0  127.0.0.1:49722  127.0.0.1:11300  users:(("apache2",pid=706522,fd=19))
1799sec  |  ESTAB  0  0  127.0.0.1:49736  127.0.0.1:11300  users:(("apache2",pid=706523,fd=19))
1799sec  |  ESTAB  0  0  127.0.0.1:49742  127.0.0.1:11300  users:(("apache2",pid=706525,fd=19))
1799sec  |  ESTAB  0  0  127.0.0.1:49756  127.0.0.1:11300  users:(("apache2",pid=706526,fd=19))
1799sec  |  ESTAB  0  0  127.0.0.1:49758  127.0.0.1:11300  users:(("apache2",pid=706524,fd=19))
1799sec  |  ESTAB  0  0  127.0.0.1:49764  127.0.0.1:11300  users:(("apache2",pid=706528,fd=19))
1800sec  |  ESTAB  0  0  127.0.0.1:49684  127.0.0.1:11300  users:(("apache2",pid=706508,fd=19))
1800sec  |  ESTAB  0  0  127.0.0.1:49690  127.0.0.1:11300  users:(("apache2",pid=706509,fd=19))
1800sec  |  ESTAB  0  0  127.0.0.1:49700  127.0.0.1:11300  users:(("apache2",pid=706510,fd=19))
1801sec  |  ESTAB  0  0  127.0.0.1:49648  127.0.0.1:11300  users:(("apache2",pid=706480,fd=19))
1801sec  |  ESTAB  0  0  127.0.0.1:49652  127.0.0.1:11300  users:(("apache2",pid=706481,fd=19))
1801sec  |  ESTAB  0  0  127.0.0.1:49654  127.0.0.1:11300  users:(("apache2",pid=706482,fd=19))
1801sec  |  ESTAB  0  0  127.0.0.1:49656  127.0.0.1:11300  users:(("apache2",pid=706492,fd=19))
1801sec  |  ESTAB  0  0  127.0.0.1:49664  127.0.0.1:11300  users:(("apache2",pid=706497,fd=19))
1802sec  |  ESTAB  0  0  127.0.0.1:49630  127.0.0.1:11300  users:(("apache2",pid=706478,fd=19))
1802sec  |  ESTAB  0  0  127.0.0.1:49634  127.0.0.1:11300  users:(("apache2",pid=706479,fd=19))
1802sec  |  ESTAB  0  0  127.0.0.1:50318  127.0.0.1:11300  users:(("apache2",pid=706475,fd=19))
1802sec  |  ESTAB  0  0  127.0.0.1:58706  127.0.0.1:11300  users:(("apache2",pid=706477,fd=19))

[tjbck]

I highly suspect #13364 played a role somehow, could anyone confirm?

[tjbck]

Also just pushed 9657474 to dev, unlikely to have played a role but trying a process of elimination. Testing wanted here.

[Ithanil]

@tjbck I could try this out tonight, so in about 12 hours.

I think the effect began immediately after the updated containers were started, with a ramp up of about 10 minutes (which would correspond to some timeout lengths):

And this is how it looks over the day:

The number of open connections doesn't really correlate to actual usage of the platform.

[tjbck]

@Ithanil So basically you just start the container and even without any users active, there's an uptick in terms of nginx connections?

[wolfaba]

total established connections (not only openwebui) in our environment. People are using it until 18:00, watchtower updates the containers at 20:00 and just after update, with a few users and monitoring, the connection count rises extremly.

[tjbck]

symptoms are very clear, unsure what the root cause is

[tjbck]

#12932 the culprit should be here somewhere, yet to find any.

[Classic298]

@Ithanil excuse my ignorance but what do these two "return True"'s do that causes this? 🤔

[Ithanil]

@Classic298 I think removing these lines effectively disables "breaking connection attempt in the connect event handler if there is no auth data" from the PR, so the main feature of it.
The refactoring part could be kept.

[rgaricano]

to those with conn problems after update: do you clean cookies after upgrade?

[Ithanil]

@rgaricano this issue was not about connection problems (per se) but about connections not being closed

[rgaricano]

yep, and related with user auth identification, that is what "problematics returns" do.
I have to revise it too, but first thinking was same as Tim's, and probably most of us: if returns are well placed!

In my work flow with openwebui (cloned, update requirements... & rebuild) sometimes after upgrade i found problems with opened sessions, opened connections and similar, most of its solved just cleaning cookies & cache, this is why i do it by routine.
I haven't a openwebui production one, but it's fully operational local open served and with some concurrents sessions/users, managed remotly too & , as normal, habitual spiders and searched engines visits.
System here is working as normal as expected.

[Tonghaofeng]

yes, i meet the same problem, and seems still not working in 0.6.7, but 0.6.5 is good。

[Hisma]

I use nginx reverse proxy and it still works, even at 0.6.7. Make sure websockets support is enabled. Also, I don't know if it makes a difference or not, but I have "force ssl" turned off.
I use the nginx proxy manager docker container. Here's what I have set -

That's it. As you can see, the only configurable options I have enabled are "block common exploits" and "web sockets support".
I don't have any other options or custom settings. I use cloudeflare as my dns provider.

[athoik]

Hi,

https://github.com/open-webui/open-webui/releases/tag/v0.6.8 is out! Fixed included!