-
Notifications
You must be signed in to change notification settings - Fork 4.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
nginx worker stuck on shutdown after reload #6904
Comments
This is not happening with kong==2.2.2 |
Thank you for notifying us about this, we'll give it a look as soon as we can. |
@diefans in order to replicate this accurately, it would be useful to know wether you are using a database or a dbless environment. Could you tell us that? A list of installed plugins might also be useful. |
I use postgres. Here is an excerpt of my kong_db:
image: postgres:9.6
networks:
- kong
volumes:
- kong_postgres:/var/lib/postgresql
environment:
POSTGRES_USER: kong
POSTGRES_DB: kong
POSTGRES_PASSWORD: kong
kong:
#image: kong:latest
build: kong
depends_on:
- wait
- kong_db
- kong_bootstrap
- jaeger
- go-plugins
- glu
- backend
- center
- ucm
- static
- toolbox
networks:
- kong
environment:
KONG_DATABASE: postgres
KONG_PG_HOST: kong_db
KONG_PG_USER: kong
KONG_PG_PASSWORD: kong
KONG_CASSANDRA_CONTACT_POINTS: kong_db
KONG_LOG_LEVEL: debug
KONG_PROXY_ACCESS_LOG: /dev/stdout
KONG_ADMIN_ACCESS_LOG: /dev/stdout
KONG_PROXY_ERROR_LOG: /dev/stderr
KONG_ADMIN_ERROR_LOG: /dev/stderr
KONG_PROXY_LISTEN: 0.0.0.0:8080, 0.0.0.0:8443 ssl
KONG_ADMIN_LISTEN: 0.0.0.0:8001, 0.0.0.0:8444 ssl
KONG_PORT_MAP: "80:8080,443:8443"
KONG_NGINX_PROXY_LARGE_CLIENT_HEADER_BUFFERS: 16 24k
KONG_NGINX_PROXY_PROXY_BUFFER_SIZE: 128k
KONG_NGINX_PROXY_PROXY_BUFFERS: 4 256k
KONG_NGINX_PROXY_PROXY_BUSY_BUFFERS_SIZE: 256k
KONG_NGINX_PROXY_INCLUDE: /etc/kong/nginx-session.conf
KONG_NGINX_HTTP_INCLUDE: /etc/kong/extra_http.conf
KONG_LUA_PACKAGE_PATH: /kong-bm-auth/?.lua;/kong-bm-auth/?/handler.lua;;
KONG_GO_PLUGINS_DIR: /go-plugins
KONG_GO_PLUGINSERVER_EXE: /usr/local/bin/go-pluginserver
KONG_PLUGINSERVER_NAMES: go,python
KONG_PLUGINSERVER_GO_SOCKET: /usr/local/kong/go_pluginserver.sock
KONG_PLUGINSERVER_GO_START_CMD: >
/usr/local/bin/go-pluginserver
-kong-prefix /usr/local/kong/
-plugins-directory /go-plugins
KONG_PLUGINSERVER_GO_QUERY_CMD: >
/usr/local/bin/go-pluginserver
-dump-all-plugins
-plugins-directory
/go-plugins
KONG_PLUGINS: ${KONG_PLUGINS-bundled, bm-auth, cors, bm-authx}
volumes:
- ${BM_AUTH_PATH}:/kong-bm-auth
- ./kong/nginx-session.conf:/etc/kong/nginx-session.conf
- ./kong/extra_http.conf:/etc/kong/extra_http.conf
- go_plugins:/go-plugins
extra_hosts: *bm_dns
ports:
- 80:8080
- 443:8443
- 8001:8001 |
If I comment out Interpreting the strace, I would guess something was terminated before something else was able to properly finish - thus waiting for something... |
I created a minimal docker-compose setup to reproduce this behavior: https://github.com/diefans/kong-issue-6904 |
I don't know if this is used somewhere in production, but it would finally eat up resources and lead to some kind of DOS:
|
Summary
On every reload there is one nginx worker stuck with
Steps To Reproduce
docker-compose start kong
docker-compose exec kong kong reload
Additional Details & Logs
Kong version (2.3.3)
Kong error logs
kong.log
Operating system: archlinux
The text was updated successfully, but these errors were encountered: