kongfig fails when triyng to apply changes from a file

[dgonzalezc]

Version of kong: 0.13

step to reproduce:
1.- kong admin running on docker toolbox, IP 192.168.99.100:8001
2.- file dump in the local directory
3.- Executing kongfig apply --path file.yml --host 192.168.99.100:8001

Error:
POST http://192.168.99.100:8001/consumers
{ username: 'si_re_td' }
201 Created { created_at: 1583255363000,
username: 'si_re_td',
id: '7b5b6f71-0166-42b8-ba6a-ed714ec8d777' }
(node:9272) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): Error: http://192.168.99.100:8001/consumers/31b12b2c-160f-419c-a561-0b1cf2e20b28/hmac-auth: 500 Internal Server Error
(node:9272) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
(node:9272) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 2): Error: http://192.168.99.100:8001/consumers/a6c1d1a6-2038-41ce-95dd-ee842bd49187/oauth2: 500 Internal Server Error
Error: http://192.168.99.100:8001/consumers/a6c1d1a6-2038-41ce-95dd-ee842bd49187/acls: 500 Internal Server Error
Error: http://192.168.99.100:8001/consumers/a6c1d1a6-2038-41ce-95dd-ee842bd49187/acls: 500 Internal Server Error
at C:\Users\dgonzalezc\AppData\Roaming\npm\node_modules\kongfig\lib\adminApi.js:139:25
at
at process._tickCallback (internal/process/next_tick.js:188:7)

Testing ErrorCall with Curl:
Request:
curl http://192.168.99.100:8001/consumers/a6c1d1a6-2038-41ce-95dd-ee842bd49187/acls

Response:
{"total":1,"data":[{"group":"mc_cl_groupwhite_cau-api","created_at":1582580072000,"id":"13b57875-5a08-4ad9-a20d-dad6991afbb7","consumer_id":"a6c1d1a6-2038-41ce-95dd-ee842bd49187"}]}

[janv8000]

What's the error logged in the Kong logs, eg. /usr/local/kong/logs/error.log ?

[dgonzalezc]

Kong logs inside the container:
total 0
-rw-r--r-- 1 root root 0 Feb 21 13:56 error.log
-rw-r--r-- 1 root root 0 Feb 21 13:56 admin_access.log
-rw-r--r-- 1 root root 0 Feb 21 13:56 access.log
/usr/local/kong/logs # tail error.log -f

Kong container logs:
$ docker logs 002092d1107a -f --since 1s | grep error
2020/03/11 19:29:37 [error] 19#0: *23270 [lua] responses.lua:121: handler(): FATAL: sorry, too many clients already, client: 192.168.99.1, server: kong_admin, request: "G
ET /consumers/5f4bb953-f365-4f6e-bb6f-fea64ab710b0/basic-auth HTTP/1.1", host: "192.168.99.100:8001"
2020/03/11 19:29:37 [error] 19#0: *23281 [lua] responses.lua:121: handler(): FATAL: sorry, too many clients already, client: 192.168.99.1, server: kong_admin, request: "G
ET /consumers/ce2630fe-23d1-4d36-8b54-5ee4f89aa9fc/jwt HTTP/1.1", host: "192.168.99.100:8001"
2020/03/11 19:29:37 [error] 19#0: *23267 [lua] responses.lua:121: handler(): FATAL: sorry, too many clients already, client: 192.168.99.1, server: kong_admin, request: "G
ET /consumers/5f4bb953-f365-4f6e-bb6f-fea64ab710b0/oauth2 HTTP/1.1", host: "192.168.99.100:8001"
2020/03/11 19:29:37 [error] 19#0: *23286 [lua] responses.lua:121: handler(): FATAL: sorry, too many clients already, client: 192.168.99.1, server: kong_admin, request: "G
ET /consumers/c9001250-b4a8-47a8-a61f-5145606cca8a/key-auth HTTP/1.1", host: "192.168.99.100:8001"
2020/03/11 19:29:37 [error] 19#0: *23246 [lua] responses.lua:121: handler(): FATAL: sorry, too many clients already, client: 192.168.99.1, server: kong_admin, request: "G
ET /consumers/84aaf553-d3c8-4dd9-9e91-969304de9714/basic-auth HTTP/1.1", host: "192.168.99.100:8001"
2020/03/11 19:29:37 [error] 19#0: *23278 [lua] responses.lua:121: handler(): FATAL: sorry, too many clients already, client: 192.168.99.1, server: kong_admin, request: "G
ET /consumers/66b0440f-4d66-49e8-8347-6f6159980d12/acls HTTP/1.1", host: "192.168.99.100:8001"
2020/03/11 19:29:37 [error] 19#0: *23275 [lua] responses.lua:121: handler(): FATAL: sorry, too many clients already, client: 192.168.99.1, server: kong_admin, request: "G
ET /consumers/66b0440f-4d66-49e8-8347-6f6159980d12/jwt HTTP/1.1", host: "192.168.99.100:8001"
2020/03/11 19:29:37 [error] 19#0: *23256 [lua] responses.lua:121: handler(): FATAL: sorry, too many clients already, client: 192.168.99.1, server: kong_admin, request: "G
ET /consumers/115a901d-d6e2-4694-b738-5fd2f5de298c/key-auth HTTP/1.1", host: "192.168.99.100:8001"
2020/03/11 19:29:37 [error] 19#0: *23291 [lua] responses.lua:121: handler(): FATAL: sorry, too many clients already, client: 192.168.99.1, server: kong_admin, request: "G
ET /consumers/689c23d2-95ab-4f96-876a-d09f7d297603/oauth2 HTTP/1.1", host: "192.168.99.100:8001"
2020/03/11 19:29:37 [error] 19#0: *23254 [lua] responses.lua:121: handler(): FATAL: sorry, too many clients already, client: 192.168.99.1, server: kong_admin, request: "G
ET /consumers/122b6a76-04f0-49f3-ab6a-e874dd164d82/acls HTTP/1.1", host: "192.168.99.100:8001"
2020/03/11 19:29:37 [error] 19#0: *23318 [lua] responses.lua:121: handler(): FATAL: sorry, too many clients already, client: 192.168.99.1, server: kong_admin, request: "G
ET /consumers/8d5e0ab2-4665-4618-ab10-31e9ccc5f64c/basic-auth HTTP/1.1", host: "192.168.99.100:8001"
2020/03/11 19:29:37 [error] 19#0: *23317 [lua] responses.lua:121: handler(): FATAL: sorry, too many clients already, client: 192.168.99.1, server: kong_admin, request: "G
ET /consumers/8d5e0ab2-4665-4618-ab10-31e9ccc5f64c/jwt HTTP/1.1", host: "192.168.99.100:8001"
2020/03/11 19:29:37 [error] 19#0: *23320 [lua] responses.lua:121: handler(): FATAL: sorry, too many clients already, client: 192.168.99.1, server: kong_admin, request: "G
ET /consumers/8d5e0ab2-4665-4618-ab10-31e9ccc5f64c/acls HTTP/1.1", host: "192.168.99.100:8001"
2020/03/11 19:29:37 [error] 19#0: *23319 [lua] responses.lua:121: handler(): FATAL: sorry, too many clients already, client: 192.168.99.1, server: kong_admin, request: "G
ET /consumers/8d5e0ab2-4665-4618-ab10-31e9ccc5f64c/hmac-auth HTTP/1.1", host: "192.168.99.100:8001"
2020/03/11 19:29:37 [error] 19#0: *23323 [lua] responses.lua:121: handler(): FATAL: sorry, too many clients already, client: 192.168.99.1, server: kong_admin, request: "G
ET /consumers/aefbc8ae-e15f-416a-9f90-1455096b1e4b/jwt HTTP/1.1", host: "192.168.99.100:8001"
2020/03/11 19:29:37 [error] 19#0: *23322 [lua] responses.lua:121: handler(): FATAL: sorry, too many clients already, client: 192.168.99.1, server: kong_admin, request: "G
ET /consumers/aefbc8ae-e15f-416a-9f90-1455096b1e4b/key-auth HTTP/1.1", host: "192.168.99.100:8001"
2020/03/11 19:29:38 [error] 19#0: *23325 [lua] responses.lua:121: handler(): FATAL: sorry, too many clients already, client: 192.168.99.1, server: kong_admin, request: "G
ET /consumers/aefbc8ae-e15f-416a-9f90-1455096b1e4b/hmac-auth HTTP/1.1", host: "192.168.99.100:8001"
2020/03/11 19:29:38 [error] 19#0: *23326 [lua] responses.lua:121: handler(): FATAL: sorry, too many clients already, client: 192.168.99.1, server: kong_admin, request: "G
ET /consumers/aefbc8ae-e15f-416a-9f90-1455096b1e4b/acls HTTP/1.1", host: "192.168.99.100:8001"

[dgonzalezc]

it could be a postgres max_connections config issue?

kong=# show max_connections

max_connections

100
(1 row)

kongfig apply command is sending several request to the datastore over the api of kong(in my case postgresql 9.6)

That could explain the "FATAL: sorry, too many clients already" message

[janv8000]

I had the exact same problem yesterday. We’re still using a pre 1.0 version of Kong and I believe they fixed a connection keep-alive issue after 1.0, possibly related: Kong/kong#3423. My resolution was to set the limit to 1000. That’s an unreasonable high number according to PG best practices but we only use PG for Kong and are in progress of upgrading to 1.0+… After which I hope to reduce that number again or switch to using Kong without a data store …

[dgonzalezc]

i tested kong migrations up for update the datastore and run a new worker with kong 0.14.1 but does not work either, kong guys mention that a improvement was made to kong for scripting request over the admin api

Kong/kong#3440

but it doesn't work

[dgonzalezc]

Changing the max_connections to 1000 throw me the following errors:

$ docker logs 2d0b2b7ce625 -f --since 10s | grep err
2020/03/12 17:43:20 [error] 47#0: *214325 recv() failed (104: Connection reset by peer), client: 192.168.99.1, server: kong_admin, request: "GET /consumers/e18178ec-f30d-
46ee-a23b-ffb2b7b6c278/key-auth HTTP/1.1", host: "192.168.99.100:8001"
2020/03/12 17:43:20 [error] 47#0: *214325 [lua] responses.lua:121: handler(): receive_message: failed to get type: connection reset by peer, client: 192.168.99.1, server:
kong_admin, request: "GET /consumers/e18178ec-f30d-46ee-a23b-ffb2b7b6c278/key-auth HTTP/1.1", host: "192.168.99.100:8001"
2020/03/12 17:43:20 [error] 47#0: *214324 recv() failed (104: Connection reset by peer), client: 192.168.99.1, server: kong_admin, request: "GET /consumers/e18178ec-f30d-
46ee-a23b-ffb2b7b6c278/oauth2 HTTP/1.1", host: "192.168.99.100:8001"
2020/03/12 17:43:20 [error] 47#0: *214324 [lua] responses.lua:121: handler(): receive_message: failed to get type: connection reset by peer, client: 192.168.99.1, server:
kong_admin, request: "GET /consumers/e18178ec-f30d-46ee-a23b-ffb2b7b6c278/oauth2 HTTP/1.1", host: "192.168.99.100:8001"
2020/03/12 17:43:20 [error] 47#0: *214323 recv() failed (104: Connection reset by peer), client: 192.168.99.1, server: kong_admin, request: "GET /consumers/87894c42-802c-
4c45-bc74-a791cde51975/acls HTTP/1.1", host: "192.168.99.100:8001"
2020/03/12 17:43:20 [error] 47#0: *214323 [lua] responses.lua:121: handler(): receive_message: failed to get type: connection reset by peer, client: 192.168.99.1, server:
kong_admin, request: "GET /consumers/87894c42-802c-4c45-bc74-a791cde51975/acls HTTP/1.1", host: "192.168.99.100:8001"
2020/03/12 17:43:20 [error] 47#0: *214319 recv() failed (104: Connection reset by peer), client: 192.168.99.1, server: kong_admin, request: "GET /consumers/87894c42-802c-
4c45-bc74-a791cde51975/key-auth HTTP/1.1", host: "192.168.99.100:8001"
2020/03/12 17:43:20 [error] 47#0: *214319 [lua] responses.lua:121: handler(): receive_message: failed to get type: connection reset by peer, client: 192.168.99.1, server:
kong_admin, request: "GET /consumers/87894c42-802c-4c45-bc74-a791cde51975/key-auth HTTP/1.1", host: "192.168.99.100:8001"
2020/03/12 17:43:20 [error] 47#0: *214318 recv() failed (104: Connection reset by peer), client: 192.168.99.1, server: kong_admin, request: "GET /consumers/87894c42-802c-
4c45-bc74-a791cde51975/oauth2 HTTP/1.1", host: "192.168.99.100:8001"
2020/03/12 17:43:20 [error] 47#0: *214318 [lua] responses.lua:121: handler(): receive_message: failed to get type: connection reset by peer, client: 192.168.99.1, server:
kong_admin, request: "GET /consumers/87894c42-802c-4c45-bc74-a791cde51975/oauth2 HTTP/1.1", host: "192.168.99.100:8001"
2020/03/12 17:43:20 [error] 47#0: *214317 recv() failed (104: Connection reset by peer), client: 192.168.99.1, server: kong_admin, request: "GET /consumers/0f5a9b8a-86cd-
4aab-8f38-48c59812c857/acls HTTP/1.1", host: "192.168.99.100:8001"

[janv8000]

What's the difference between those two error logs?

[dgonzalezc]

setup:

• docker toolbox
• kongfig 1.5.3
• dump file generated

step first error:

• postgres 9.6 with max_connections=100(default)
• kong 0.13.1

steps second error:

• postgres 9.6 with max_connections=1000
• kong 0.14.1

next steps:

• test kongfig dump/apply with kong 0.15.0
• default postgres 9.6 max_connections config