Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kong startup logs [error] still in larger deployments #14340

Open
1 task done
jeremyjpj0916 opened this issue Mar 4, 2025 · 2 comments
Open
1 task done

Kong startup logs [error] still in larger deployments #14340

jeremyjpj0916 opened this issue Mar 4, 2025 · 2 comments

Comments

@jeremyjpj0916
Copy link
Contributor

jeremyjpj0916 commented Mar 4, 2025

Is there an existing issue for this?

  • I have searched the existing issues

Kong version ($ kong version)

3.9.0

Current Behavior

I have brought this up in the past but still seeing it in our bigger proxy environment(more kong routes/services/plugins etc. Probably 5000+ proxies in total) But startup logs look like this, running Kong in traditional mode:

2025-03-04T00:36:38.376574776-06:00 
2025-03-04T00:36:38.655150112-06:00 nginx: [warn] the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /usr/local/kong/nginx.conf:10
2025-03-04T00:36:38.655581907-06:00 nginx: [warn] load balancing method redefined in /usr/local/kong/nginx.conf:117
2025-03-04T00:36:38.736729604-06:00 2025/03/04 06:36:38 [notice] 26#0: [lua] client.lua:161: new(): [dns_client] supported types: srv ipv4 ipv6 
2025-03-04T00:36:38.814449460-06:00 2025/03/04 06:36:38 [notice] 26#0: [lua] client.lua:161: new(): [dns_client] supported types: srv ipv4 
2025-03-04T00:36:40.124236219-06:00 2025/03/04 06:36:38 [notice] 26#0: [lua] init.lua:794: init(): [request-debug] token for request debugging: 60a4ff7d-cdb8-4f9e-8722-d38743a910a4
2025-03-04T00:36:40.269840926-06:00 2025/03/04 06:36:38 [notice] 26#0: using the "epoll" event method
2025-03-04T00:36:40.269865693-06:00 2025/03/04 06:36:38 [notice] 26#0: openresty/1.25.3.2
2025-03-04T00:36:40.269868859-06:00 2025/03/04 06:36:38 [notice] 26#0: OS: Linux 5.14.0-427.50.1.el9_4.x86_64
2025-03-04T00:36:40.269872185-06:00 2025/03/04 06:36:38 [notice] 26#0: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2025-03-04T00:36:40.269891932-06:00 2025/03/04 06:36:38 [notice] 26#0: start worker processes
2025-03-04T00:36:40.273296529-06:00 2025/03/04 06:36:38 [notice] 26#0: start worker process 39
2025-03-04T00:36:40.276214517-06:00 2025/03/04 06:36:38 [notice] 26#0: start worker process 40
2025-03-04T00:36:40.279107768-06:00 2025/03/04 06:36:38 [notice] 26#0: start worker process 41
2025-03-04T00:36:40.280811604-06:00 2025/03/04 06:36:38 [notice] 26#0: start worker process 42
2025-03-04T00:36:40.297028766-06:00 2025/03/04 06:36:40 [notice] 39#0: *1 [lua] broker.lua:215: init(): event broker is ready to accept connections on worker #0, context: init_worker_by_lua*
2025-03-04T00:36:40.361148511-06:00 2025/03/04 06:36:40 [notice] 39#0: *1 [lua] warmup.lua:113: single_dao(): Preloading 'services' into the core_cache..., context: init_worker_by_lua*
2025-03-04T00:36:40.375630437-06:00 2025/03/04 06:36:40 [notice] 41#0: *3 [lua] globalpatches.lua:75: sleep(): executing a blocking 'sleep' (0.001 seconds), context: init_worker_by_lua*
2025-03-04T00:36:40.376148034-06:00 2025/03/04 06:36:40 [notice] 40#0: *2 [lua] globalpatches.lua:75: sleep(): executing a blocking 'sleep' (0.001 seconds), context: init_worker_by_lua*
2025-03-04T00:36:40.376593635-06:00 2025/03/04 06:36:40 [notice] 41#0: *3 [lua] globalpatches.lua:75: sleep(): executing a blocking 'sleep' (0.002 seconds), context: init_worker_by_lua*
2025-03-04T00:36:40.377299598-06:00 2025/03/04 06:36:40 [notice] 40#0: *2 [lua] globalpatches.lua:75: sleep(): executing a blocking 'sleep' (0.002 seconds), context: init_worker_by_lua*
2025-03-04T00:36:40.378805580-06:00 2025/03/04 06:36:40 [notice] 41#0: *3 [lua] globalpatches.lua:75: sleep(): executing a blocking 'sleep' (0.004 seconds), context: init_worker_by_lua*
2025-03-04T00:36:40.379555716-06:00 2025/03/04 06:36:40 [notice] 40#0: *2 [lua] globalpatches.lua:75: sleep(): executing a blocking 'sleep' (0.004 seconds), context: init_worker_by_lua*
2025-03-04T00:36:40.383917560-06:00 2025/03/04 06:36:40 [notice] 41#0: *3 [lua] globalpatches.lua:75: sleep(): executing a blocking 'sleep' (0.008 seconds), context: init_worker_by_lua*
2025-03-04T00:36:40.383932949-06:00 2025/03/04 06:36:40 [notice] 40#0: *2 [lua] globalpatches.lua:75: sleep(): executing a blocking 'sleep' (0.008 seconds), context: init_worker_by_lua*
2025-03-04T00:36:40.391123262-06:00 2025/03/04 06:36:40 [notice] 41#0: *3 [lua] globalpatches.lua:75: sleep(): executing a blocking 'sleep' (0.016 seconds), context: init_worker_by_lua*
2025-03-04T00:36:40.391830116-06:00 2025/03/04 06:36:40 [notice] 40#0: *2 [lua] globalpatches.lua:75: sleep(): executing a blocking 'sleep' (0.016 seconds), context: init_worker_by_lua*
2025-03-04T00:36:40.407349180-06:00 2025/03/04 06:36:40 [notice] 41#0: *3 [lua] globalpatches.lua:75: sleep(): executing a blocking 'sleep' (0.032 seconds), context: init_worker_by_lua*
2025-03-04T00:36:40.408077044-06:00 2025/03/04 06:36:40 [notice] 40#0: *2 [lua] globalpatches.lua:75: sleep(): executing a blocking 'sleep' (0.032 seconds), context: init_worker_by_lua*
2025-03-04T00:36:40.605634649-06:00 2025/03/04 06:36:40 [notice] 39#0: *1 [lua] warmup.lua:162: single_dao(): finished preloading 'services' into the core_cache (in 245ms), context: init_worker_by_lua*
2025-03-04T00:36:40.605649497-06:00 2025/03/04 06:36:40 [notice] 39#0: *1 [lua] warmup.lua:113: single_dao(): Preloading 'consumers' into the cache..., context: init_worker_by_lua*
2025-03-04T00:36:40.715027742-06:00 2025/03/04 06:36:40 [notice] 39#0: *1 [lua] warmup.lua:162: single_dao(): finished preloading 'consumers' into the cache (in 0ms), context: init_worker_by_lua*
2025-03-04T00:36:40.715044845-06:00 2025/03/04 06:36:40 [notice] 39#0: *1 [lua] warmup.lua:113: single_dao(): Preloading 'acls' into the cache..., context: init_worker_by_lua*
2025-03-04T00:36:45.615918505-06:00 2025/03/04 06:36:45 [error] 42#0: *6 [lua] worker.lua:291: communicate(): failed to connect: failed to receive response header: timeout, context: ngx.timer
2025-03-04T00:36:45.624461612-06:00 2025/03/04 06:36:45 [error] 40#0: *651 [lua] worker.lua:291: communicate(): failed to connect: failed to receive response header: timeout, context: ngx.timer
2025-03-04T00:36:45.626305162-06:00 2025/03/04 06:36:45 [error] 41#0: *679 [lua] worker.lua:291: communicate(): failed to connect: failed to receive response header: timeout, context: ngx.timer
2025-03-04T00:36:50.631674221-06:00 2025/03/04 06:36:50 [error] 42#0: *1936 [lua] worker.lua:291: communicate(): failed to connect: failed to receive response header: timeout, context: ngx.timer
2025-03-04T00:36:50.650011916-06:00 2025/03/04 06:36:50 [error] 41#0: *1938 [lua] worker.lua:291: communicate(): failed to connect: failed to receive response header: timeout, context: ngx.timer
2025-03-04T00:36:50.662620436-06:00 2025/03/04 06:36:50 [error] 40#0: *1940 [lua] worker.lua:291: communicate(): failed to connect: failed to receive response header: timeout, context: ngx.timer
2025-03-04T00:36:55.670112252-06:00 2025/03/04 06:36:55 [error] 42#0: *1942 [lua] worker.lua:291: communicate(): failed to connect: failed to receive response header: timeout, context: ngx.timer
2025-03-04T00:36:55.683588029-06:00 2025/03/04 06:36:55 [error] 40#0: *1944 [lua] worker.lua:291: communicate(): failed to connect: failed to receive response header: timeout, context: ngx.timer
2025-03-04T00:36:55.689511410-06:00 2025/03/04 06:36:55 [error] 41#0: *1946 [lua] worker.lua:291: communicate(): failed to connect: failed to receive response header: timeout, context: ngx.timer
2025-03-04T00:37:00.693422577-06:00 2025/03/04 06:37:00 [error] 42#0: *1948 [lua] worker.lua:291: communicate(): failed to connect: failed to receive response header: timeout, context: ngx.timer
2025-03-04T00:37:00.717759015-06:00 2025/03/04 06:37:00 [error] 40#0: *1950 [lua] worker.lua:291: communicate(): failed to connect: failed to receive response header: timeout, context: ngx.timer
2025-03-04T00:37:00.729000676-06:00 2025/03/04 06:37:00 [error] 41#0: *1952 [lua] worker.lua:291: communicate(): failed to connect: failed to receive response header: timeout, context: ngx.timer
2025-03-04T00:37:05.711611345-06:00 2025/03/04 06:37:05 [error] 42#0: *1954 [lua] worker.lua:291: communicate(): failed to connect: failed to receive response header: timeout, context: ngx.timer
2025-03-04T00:37:05.757056078-06:00 2025/03/04 06:37:05 [error] 40#0: *1956 [lua] worker.lua:291: communicate(): failed to connect: failed to receive response header: timeout, context: ngx.timer
2025-03-04T00:37:05.766073179-06:00 2025/03/04 06:37:05 [error] 41#0: *1958 [lua] worker.lua:291: communicate(): failed to connect: failed to receive response header: timeout, context: ngx.timer
2025-03-04T00:37:10.730712565-06:00 2025/03/04 06:37:10 [error] 42#0: *1960 [lua] worker.lua:291: communicate(): failed to connect: failed to receive response header: timeout, context: ngx.timer
2025-03-04T00:37:10.791566825-06:00 2025/03/04 06:37:10 [error] 41#0: *1962 [lua] worker.lua:291: communicate(): failed to connect: failed to receive response header: timeout, context: ngx.timer
2025-03-04T00:37:10.802934583-06:00 2025/03/04 06:37:10 [error] 40#0: *1964 [lua] worker.lua:291: communicate(): failed to connect: failed to receive response header: timeout, context: ngx.timer
2025-03-04T00:37:15.761436284-06:00 2025/03/04 06:37:15 [error] 42#0: *1966 [lua] worker.lua:291: communicate(): failed to connect: failed to receive response header: timeout, context: ngx.timer
2025-03-04T00:37:15.825371009-06:00 2025/03/04 06:37:15 [error] 40#0: *1968 [lua] worker.lua:291: communicate(): failed to connect: failed to receive response header: timeout, context: ngx.timer
2025-03-04T00:37:15.834548031-06:00 2025/03/04 06:37:15 [error] 41#0: *1970 [lua] worker.lua:291: communicate(): failed to connect: failed to receive response header: timeout, context: ngx.timer
2025-03-04T00:37:20.775059714-06:00 2025/03/04 06:37:20 [error] 42#0: *2025 [lua] worker.lua:291: communicate(): failed to connect: failed to receive response header: timeout, context: ngx.timer
2025-03-04T00:37:20.838767170-06:00 2025/03/04 06:37:20 [error] 40#0: *2030 [lua] worker.lua:291: communicate(): failed to connect: failed to receive response header: timeout, context: ngx.timer
2025-03-04T00:37:20.846771519-06:00 2025/03/04 06:37:20 [error] 41#0: *2032 [lua] worker.lua:291: communicate(): failed to connect: failed to receive response header: timeout, context: ngx.timer
2025-03-04T00:37:25.799809406-06:00 2025/03/04 06:37:25 [error] 42#0: *2091 [lua] worker.lua:291: communicate(): failed to connect: failed to receive response header: timeout, context: ngx.timer
2025-03-04T00:37:25.866307890-06:00 2025/03/04 06:37:25 [error] 40#0: *2093 [lua] worker.lua:291: communicate(): failed to connect: failed to receive response header: timeout, context: ngx.timer
2025-03-04T00:37:25.871060402-06:00 2025/03/04 06:37:25 [error] 41#0: *2095 [lua] worker.lua:291: communicate(): failed to connect: failed to receive response header: timeout, context: ngx.timer
2025-03-04T00:37:30.817564245-06:00 2025/03/04 06:37:30 [error] 42#0: *2120 [lua] worker.lua:291: communicate(): failed to connect: failed to receive response header: timeout, context: ngx.timer
2025-03-04T00:37:30.882516140-06:00 2025/03/04 06:37:30 [error] 41#0: *2122 [lua] worker.lua:291: communicate(): failed to connect: failed to receive response header: timeout, context: ngx.timer
2025-03-04T00:37:30.884853111-06:00 2025/03/04 06:37:30 [error] 40#0: *2124 [lua] worker.lua:291: communicate(): failed to connect: failed to receive response header: timeout, context: ngx.timer
2025-03-04T00:37:35.860038926-06:00 2025/03/04 06:37:35 [error] 42#0: *2149 [lua] worker.lua:291: communicate(): failed to connect: failed to receive response header: timeout, context: ngx.timer
2025-03-04T00:37:35.898458067-06:00 2025/03/04 06:37:35 [error] 40#0: *2151 [lua] worker.lua:291: communicate(): failed to connect: failed to receive response header: timeout, context: ngx.timer
2025-03-04T00:37:35.930674479-06:00 2025/03/04 06:37:35 [error] 41#0: *2153 [lua] worker.lua:291: communicate(): failed to connect: failed to receive response header: timeout, context: ngx.timer
2025-03-04T00:37:40.910393871-06:00 2025/03/04 06:37:40 [error] 42#0: *2161 [lua] worker.lua:291: communicate(): failed to connect: failed to receive response header: timeout, context: ngx.timer
2025-03-04T00:37:40.947125236-06:00 2025/03/04 06:37:40 [error] 41#0: *2163 [lua] worker.lua:291: communicate(): failed to connect: failed to receive response header: timeout, context: ngx.timer
2025-03-04T00:37:40.947849503-06:00 2025/03/04 06:37:40 [error] 40#0: *2165 [lua] worker.lua:291: communicate(): failed to connect: failed to receive response header: timeout, context: ngx.timer
2025-03-04T00:37:45.928647812-06:00 2025/03/04 06:37:45 [error] 42#0: *2191 [lua] worker.lua:291: communicate(): failed to connect: failed to receive response header: timeout, context: ngx.timer
2025-03-04T00:37:45.975420752-06:00 2025/03/04 06:37:45 [error] 41#0: *2193 [lua] worker.lua:291: communicate(): failed to connect: failed to receive response header: timeout, context: ngx.timer
2025-03-04T00:37:45.985366646-06:00 2025/03/04 06:37:45 [error] 40#0: *2195 [lua] worker.lua:291: communicate(): failed to connect: failed to receive response header: timeout, context: ngx.timer
2025-03-04T00:37:50.943635818-06:00 2025/03/04 06:37:50 [error] 42#0: *2214 [lua] worker.lua:291: communicate(): failed to connect: failed to receive response header: timeout, context: ngx.timer
2025-03-04T00:37:51.004468848-06:00 2025/03/04 06:37:51 [error] 40#0: *2216 [lua] worker.lua:291: communicate(): failed to connect: failed to receive response header: timeout, context: ngx.timer
2025-03-04T00:37:51.016171939-06:00 2025/03/04 06:37:51 [error] 41#0: *2218 [lua] worker.lua:291: communicate(): failed to connect: failed to receive response header: timeout, context: ngx.timer
2025-03-04T00:37:52.410209201-06:00 2025/03/04 06:36:40 [notice] 39#0: *1 [lua] warmup.lua:162: single_dao(): finished preloading 'acls' into the cache (in 0ms), context: init_worker_by_lua*
2025-03-04T00:37:52.410275847-06:00 2025/03/04 06:36:40 [notice] 39#0: *1 [lua] warmup.lua:113: single_dao(): Preloading 'certificates' into the core_cache..., context: init_worker_by_lua*
2025-03-04T00:37:52.466804161-06:00 2025/03/04 06:36:40 [notice] 39#0: *1 [lua] warmup.lua:162: single_dao(): finished preloading 'certificates' into the core_cache (in 0ms), context: init_worker_by_lua*
2025-03-04T00:37:52.466841071-06:00 2025/03/04 06:36:40 [notice] 39#0: *1 [lua] warmup.lua:113: single_dao(): Preloading 'jwt_secrets' into the cache..., context: init_worker_by_lua*
2025-03-04T00:37:52.556407248-06:00 2025/03/04 06:36:40 [notice] 39#0: *1 [lua] warmup.lua:162: single_dao(): finished preloading 'jwt_secrets' into the cache (in 0ms), context: init_worker_by_lua*
2025-03-04T00:37:52.556446222-06:00 2025/03/04 06:36:40 [notice] 39#0: *1 [lua] warmup.lua:113: single_dao(): Preloading 'oauth2_credentials' into the cache..., context: init_worker_by_lua*
2025-03-04T00:37:52.658270899-06:00 2025/03/04 06:36:40 [notice] 39#0: *1 [lua] warmup.lua:162: single_dao(): finished preloading 'oauth2_credentials' into the cache (in 0ms), context: init_worker_by_lua*
2025-03-04T00:37:53.020146846-06:00 2025/03/04 06:37:53 [notice] 39#0: *2424 [lua] warmup.lua:46: warming up DNS entries ..., context: ngx.timer
2025-03-04T00:37:53.025000898-06:00 2025/03/04 06:37:53 [error] 39#0: *2333 [lua] broker.lua:235: run(): failed to init socket: failed to flush response header: nginx output filter error, client: unix:, server: kong_worker_events, request: "GET / HTTP/1.1", host: "localhost"
2025-03-04T00:37:53.025033590-06:00 2025/03/04 06:37:53 [error] 39#0: *2334 [lua] broker.lua:235: run(): failed to init socket: failed to flush response header: nginx output filter error, client: unix:, server: kong_worker_events, request: "GET / HTTP/1.1", host: "localhost"
2025-03-04T00:37:53.025042346-06:00 2025/03/04 06:37:53 [error] 39#0: *2335 [lua] broker.lua:235: run(): failed to init socket: failed to flush response header: nginx output filter error, client: unix:, server: kong_worker_events, request: "GET / HTTP/1.1", host: "localhost"
2025-03-04T00:37:53.025109634-06:00 2025/03/04 06:37:53 [error] 39#0: *2336 [lua] broker.lua:235: run(): failed to init socket: failed to flush response header: nginx output filter error, client: unix:, server: kong_worker_events, request: "GET / HTTP/1.1", host: "localhost"
2025-03-04T00:37:53.025204573-06:00 2025/03/04 06:37:53 [error] 39#0: *2337 [lua] broker.lua:235: run(): failed to init socket: failed to flush response header: nginx output filter error, client: unix:, server: kong_worker_events, request: "GET / HTTP/1.1", host: "localhost"
2025-03-04T00:37:53.025216365-06:00 2025/03/04 06:37:53 [error] 39#0: *2338 [lua] broker.lua:235: run(): failed to init socket: failed to flush response header: nginx output filter error, client: unix:, server: kong_worker_events, request: "GET / HTTP/1.1", host: "localhost"
2025-03-04T00:37:53.025305573-06:00 2025/03/04 06:37:53 [error] 39#0: *2339 [lua] broker.lua:235: run(): failed to init socket: failed to flush response header: nginx output filter error, client: unix:, server: kong_worker_events, request: "GET / HTTP/1.1", host: "localhost"
2025-03-04T00:37:53.025434727-06:00 2025/03/04 06:37:53 [error] 39#0: *2340 [lua] broker.lua:235: run(): failed to init socket: failed to flush response header: nginx output filter error, client: unix:, server: kong_worker_events, request: "GET / HTTP/1.1", host: "localhost"
2025-03-04T00:37:53.025442883-06:00 2025/03/04 06:37:53 [error] 39#0: *2341 [lua] broker.lua:235: run(): failed to init socket: failed to flush response header: nginx output filter error, client: unix:, server: kong_worker_events, request: "GET / HTTP/1.1", host: "localhost"
2025-03-04T00:37:53.025581494-06:00 2025/03/04 06:37:53 [error] 39#0: *2342 [lua] broker.lua:235: run(): failed to init socket: failed to flush response header: nginx output filter error, client: unix:, server: kong_worker_events, request: "GET / HTTP/1.1", host: "localhost"
2025-03-04T00:37:53.025604367-06:00 2025/03/04 06:37:53 [error] 39#0: *2343 [lua] broker.lua:235: run(): failed to init socket: failed to flush response header: nginx output filter error, client: unix:, server: kong_worker_events, request: "GET / HTTP/1.1", host: "localhost"
2025-03-04T00:37:53.025653280-06:00 2025/03/04 06:37:53 [error] 39#0: *2344 [lua] broker.lua:235: run(): failed to init socket: failed to flush response header: nginx output filter error, client: unix:, server: kong_worker_events, request: "GET / HTTP/1.1", host: "localhost"
2025-03-04T00:37:53.025662297-06:00 2025/03/04 06:37:53 [error] 39#0: *2345 [lua] broker.lua:235: run(): failed to init socket: failed to flush response header: nginx output filter error, client: unix:, server: kong_worker_events, request: "GET / HTTP/1.1", host: "localhost"
2025-03-04T00:37:53.025668999-06:00 2025/03/04 06:37:53 [error] 39#0: *2346 [lua] broker.lua:235: run(): failed to init socket: failed to flush response header: nginx output filter error, client: unix:, server: kong_worker_events, request: "GET / HTTP/1.1", host: "localhost"
2025-03-04T00:37:53.025680912-06:00 2025/03/04 06:37:53 [error] 39#0: *2347 [lua] broker.lua:235: run(): failed to init socket: failed to flush response header: nginx output filter error, client: unix:, server: kong_worker_events, request: "GET / HTTP/1.1", host: "localhost"
2025-03-04T00:37:53.025725676-06:00 2025/03/04 06:37:53 [error] 39#0: *2348 [lua] broker.lua:235: run(): failed to init socket: failed to flush response header: nginx output filter error, client: unix:, server: kong_worker_events, request: "GET / HTTP/1.1", host: "localhost"
2025-03-04T00:37:53.025745374-06:00 2025/03/04 06:37:53 [error] 39#0: *2349 [lua] broker.lua:235: run(): failed to init socket: failed to flush response header: nginx output filter error, client: unix:, server: kong_worker_events, request: "GET / HTTP/1.1", host: "localhost"
2025-03-04T00:37:53.025753689-06:00 2025/03/04 06:37:53 [error] 39#0: *2350 [lua] broker.lua:235: run(): failed to init socket: failed to flush response header: nginx output filter error, client: unix:, server: kong_worker_events, request: "GET / HTTP/1.1", host: "localhost"
2025-03-04T00:37:53.025756845-06:00 2025/03/04 06:37:53 [error] 39#0: *2351 [lua] broker.lua:235: run(): failed to init socket: failed to flush response header: nginx output filter error, client: unix:, server: kong_worker_events, request: "GET / HTTP/1.1", host: "localhost"
2025-03-04T00:37:53.025759811-06:00 2025/03/04 06:37:53 [error] 39#0: *2352 [lua] broker.lua:235: run(): failed to init socket: failed to flush response header: nginx output filter error, client: unix:, server: kong_worker_events, request: "GET / HTTP/1.1", host: "localhost"
2025-03-04T00:37:53.025765291-06:00 2025/03/04 06:37:53 [error] 39#0: *2353 [lua] broker.lua:235: run(): failed to init socket: failed to flush response header: nginx output filter error, client: unix:, server: kong_worker_events, request: "GET / HTTP/1.1", host: "localhost"
2025-03-04T00:37:53.025786401-06:00 2025/03/04 06:37:53 [error] 39#0: *2354 [lua] broker.lua:235: run(): failed to init socket: failed to flush response header: nginx output filter error, client: unix:, server: kong_worker_events, request: "GET / HTTP/1.1", host: "localhost"
2025-03-04T00:37:53.025795228-06:00 2025/03/04 06:37:53 [error] 39#0: *2355 [lua] broker.lua:235: run(): failed to init socket: failed to flush response header: nginx output filter error, client: unix:, server: kong_worker_events, request: "GET / HTTP/1.1", host: "localhost"
2025-03-04T00:37:53.025822740-06:00 2025/03/04 06:37:53 [error] 39#0: *2356 [lua] broker.lua:235: run(): failed to init socket: failed to flush response header: nginx output filter error, client: unix:, server: kong_worker_events, request: "GET / HTTP/1.1", host: "localhost"
2025-03-04T00:37:53.025893954-06:00 2025/03/04 06:37:53 [error] 39#0: *2357 [lua] broker.lua:235: run(): failed to init socket: failed to flush response header: nginx output filter error, client: unix:, server: kong_worker_events, request: "GET / HTTP/1.1", host: "localhost"
2025-03-04T00:37:53.025900045-06:00 2025/03/04 06:37:53 [error] 39#0: *2358 [lua] broker.lua:235: run(): failed to init socket: failed to flush response header: nginx output filter error, client: unix:, server: kong_worker_events, request: "GET / HTTP/1.1", host: "localhost"
2025-03-04T00:37:53.025914944-06:00 2025/03/04 06:37:53 [error] 39#0: *2359 [lua] broker.lua:235: run(): failed to init socket: failed to flush response header: nginx output filter error, client: unix:, server: kong_worker_events, request: "GET / HTTP/1.1", host: "localhost"
2025-03-04T00:37:53.025930002-06:00 2025/03/04 06:37:53 [error] 39#0: *2360 [lua] broker.lua:235: run(): failed to init socket: failed to flush response header: nginx output filter error, client: unix:, server: kong_worker_events, request: "GET / HTTP/1.1", host: "localhost"
2025-03-04T00:37:53.025953697-06:00 2025/03/04 06:37:53 [error] 39#0: *2361 [lua] broker.lua:235: run(): failed to init socket: failed to flush response header: nginx output filter error, client: unix:, server: kong_worker_events, request: "GET / HTTP/1.1", host: "localhost"
2025-03-04T00:37:53.025982872-06:00 2025/03/04 06:37:53 [error] 39#0: *2362 [lua] broker.lua:235: run(): failed to init socket: failed to flush response header: nginx output filter error, client: unix:, server: kong_worker_events, request: "GET / HTTP/1.1", host: "localhost"
2025-03-04T00:37:53.025989785-06:00 2025/03/04 06:37:53 [error] 39#0: *2363 [lua] broker.lua:235: run(): failed to init socket: failed to flush response header: nginx output filter error, client: unix:, server: kong_worker_events, request: "GET / HTTP/1.1", host: "localhost"
2025-03-04T00:37:53.025995896-06:00 2025/03/04 06:37:53 [error] 39#0: *2364 [lua] broker.lua:235: run(): failed to init socket: failed to flush response header: nginx output filter error, client: unix:, server: kong_worker_events, request: "GET / HTTP/1.1", host: "localhost"
2025-03-04T00:37:53.025998962-06:00 2025/03/04 06:37:53 [error] 39#0: *2365 [lua] broker.lua:235: run(): failed to init socket: failed to flush response header: nginx output filter error, client: unix:, server: kong_worker_events, request: "GET / HTTP/1.1", host: "localhost"
2025-03-04T00:37:53.026079945-06:00 2025/03/04 06:37:53 [error] 39#0: *2366 [lua] broker.lua:235: run(): failed to init socket: failed to flush response header: nginx output filter error, client: unix:, server: kong_worker_events, request: "GET / HTTP/1.1", host: "localhost"
2025-03-04T00:37:53.026084774-06:00 2025/03/04 06:37:53 [error] 39#0: *2367 [lua] broker.lua:235: run(): failed to init socket: failed to flush response header: nginx output filter error, client: unix:, server: kong_worker_events, request: "GET / HTTP/1.1", host: "localhost"
2025-03-04T00:37:53.026106375-06:00 2025/03/04 06:37:53 [error] 39#0: *2368 [lua] broker.lua:235: run(): failed to init socket: failed to flush response header: nginx output filter error, client: unix:, server: kong_worker_events, request: "GET / HTTP/1.1", host: "localhost"
2025-03-04T00:37:53.026117646-06:00 2025/03/04 06:37:53 [error] 39#0: *2369 [lua] broker.lua:235: run(): failed to init socket: failed to flush response header: nginx output filter error, client: unix:, server: kong_worker_events, request: "GET / HTTP/1.1", host: "localhost"
2025-03-04T00:37:53.026126022-06:00 2025/03/04 06:37:53 [error] 39#0: *2370 [lua] broker.lua:235: run(): failed to init socket: failed to flush response header: nginx output filter error, client: unix:, server: kong_worker_events, request: "GET / HTTP/1.1", host: "localhost"
2025-03-04T00:37:53.026217645-06:00 2025/03/04 06:37:53 [error] 39#0: *2371 [lua] broker.lua:235: run(): failed to init socket: failed to flush response header: nginx output filter error, client: unix:, server: kong_worker_events, request: "GET / HTTP/1.1", host: "localhost"
2025-03-04T00:37:53.026224628-06:00 2025/03/04 06:37:53 [error] 39#0: *2372 [lua] broker.lua:235: run(): failed to init socket: failed to flush response header: nginx output filter error, client: unix:, server: kong_worker_events, request: "GET / HTTP/1.1", host: "localhost"
2025-03-04T00:37:53.026227554-06:00 2025/03/04 06:37:53 [error] 39#0: *2373 [lua] broker.lua:235: run(): failed to init socket: failed to flush response header: nginx output filter error, client: unix:, server: kong_worker_events, request: "GET / HTTP/1.1", host: "localhost"
2025-03-04T00:37:53.026230419-06:00 2025/03/04 06:37:53 [error] 39#0: *2374 [lua] broker.lua:235: run(): failed to init socket: failed to flush response header: nginx output filter error, client: unix:, server: kong_worker_events, request: "GET / HTTP/1.1", host: "localhost"
2025-03-04T00:37:53.026357910-06:00 2025/03/04 06:37:53 [notice] 40#0: *2324 [lua] worker.lua:304: communicate(): worker #1 is ready to accept events from unix:/usr/local/kong/sockets/we, context: ngx.timer
2025-03-04T00:37:53.026509005-06:00 2025/03/04 06:37:53 [notice] 41#0: *2326 [lua] worker.lua:304: communicate(): worker #2 is ready to accept events from unix:/usr/local/kong/sockets/we, context: ngx.timer
2025-03-04T00:37:53.026590809-06:00 2025/03/04 06:37:53 [notice] 42#0: *2322 [lua] worker.lua:304: communicate(): worker #3 is ready to accept events from unix:/usr/local/kong/sockets/we, context: ngx.timer
2025-03-04T00:37:53.071008644-06:00 2025/03/04 06:37:53 [notice] 39#0: *2375 [lua] broker.lua:263: run(): worker #3 connected to events broker (worker pid: 42), client: unix:, server: kong_worker_events, request: "GET / HTTP/1.1", host: "localhost"
2025-03-04T00:37:53.071135113-06:00 2025/03/04 06:37:53 [notice] 39#0: *2376 [lua] broker.lua:263: run(): worker #1 connected to events broker (worker pid: 40), client: unix:, server: kong_worker_events, request: "GET / HTTP/1.1", host: "localhost"
2025-03-04T00:37:53.071399762-06:00 2025/03/04 06:37:53 [notice] 39#0: *2377 [lua] broker.lua:263: run(): worker #2 connected to events broker (worker pid: 41), client: unix:, server: kong_worker_events, request: "GET / HTTP/1.1", host: "localhost"
2025-03-04T00:37:53.075160120-06:00 2025/03/04 06:37:53 [notice] 39#0: *2421 [lua] worker.lua:304: communicate(): worker #0 is ready to accept events from unix:/usr/local/kong/sockets/we, context: ngx.timer
2025-03-04T00:37:53.085835611-06:00 2025/03/04 06:37:53 [notice] 39#0: *3063 [lua] broker.lua:263: run(): worker #0 connected to events broker (worker pid: 39), client: unix:, server: kong_worker_events, request: "GET / HTTP/1.1", host: "localhost"
2025-03-04T00:38:01.632549402-06:00 2025/03/04 06:38:01 [notice] 39#0: *2424 [lua] warmup.lua:80: finished warming up DNS entries' into the cache (in 8612ms), context: ngx.timer

After these logs Kong is healthy and then starts regularly handling traffic. Still no idea why all the [error] logs in the startup phase of a larger Kong instance though?

Expected Behavior

No response

Steps To Reproduce

No response

Anything else?

No response

@jeremyjpj0916
Copy link
Contributor Author

jeremyjpj0916 commented Mar 4, 2025

Straight up [error] logs throwing from 06:36:45 to 06:37:53 in those logs, little past a full minute.

Related issue from prior: #13250

@jeremyjpj0916
Copy link
Contributor Author

Is this like the sweet sauce that lets us know the errors are resolved at this point?

2025-03-04T00:37:53.026357910-06:00 2025/03/04 06:37:53 [notice] 40#0: *2324 [lua] worker.lua:304: communicate(): worker #1 is ready to accept events from unix:/usr/local/kong/sockets/we, context: ngx.timer
2025-03-04T00:37:53.026509005-06:00 2025/03/04 06:37:53 [notice] 41#0: *2326 [lua] worker.lua:304: communicate(): worker #2 is ready to accept events from unix:/usr/local/kong/sockets/we, context: ngx.timer
2025-03-04T00:37:53.026590809-06:00 2025/03/04 06:37:53 [notice] 42#0: *2322 [lua] worker.lua:304: communicate(): worker #3 is ready to accept events from unix:/usr/local/kong/sockets/we, context: ngx.timer
2025-03-04T00:37:53.071008644-06:00 2025/03/04 06:37:53 [notice] 39#0: *2375 [lua] broker.lua:263: run(): worker #3 connected to events broker (worker pid: 42), client: unix:, server: kong_worker_events, request: "GET / HTTP/1.1", host: "localhost"
2025-03-04T00:37:53.071135113-06:00 2025/03/04 06:37:53 [notice] 39#0: *2376 [lua] broker.lua:263: run(): worker #1 connected to events broker (worker pid: 40), client: unix:, server: kong_worker_events, request: "GET / HTTP/1.1", host: "localhost"
2025-03-04T00:37:53.071399762-06:00 2025/03/04 06:37:53 [notice] 39#0: *2377 [lua] broker.lua:263: run(): worker #2 connected to events broker (worker pid: 41), client: unix:, server: kong_worker_events, request: "GET / HTTP/1.1", host: "localhost"
2025-03-04T00:37:53.075160120-06:00 2025/03/04 06:37:53 [notice] 39#0: *2421 [lua] worker.lua:304: communicate(): worker #0 is ready to accept events from unix:/usr/local/kong/sockets/we, context: ngx.timer
2025-03-04T00:37:53.085835611-06:00 2025/03/04 06:37:53 [notice] 39#0: *3063 [lua] broker.lua:263: run(): worker #0 connected to events broker (worker pid: 39), client: unix:, server: kong_worker_events, request: "GET / HTTP/1.1", host: "localhost"

Where finally we see "ready to accept events" ? I guess the question is why the tons of errors prior and the full minute of it I suppose. Slow startup procedure stuff for larger Kong runtimes leading to early runtime initial error logs?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant