OAuth2 Plugin - Token generation returns incorrect access_token on concurrent requests with DB transaction timeout

[alisonbudag]

Is there an existing issue for this?

• I have searched the existing issues

Kong version ($ kong version)

2.8.5

Current Behavior

When multiple concurrent requests are made to the OAuth2 token generation endpoint in Kong 2.8.5, if a database transaction timeout occurs, sometimes the response contains an access token associated with a different authenticated_userid.

This issue suggests that there may be a concurrency problem where the plugin incorrectly returns a previously issued token instead of handling the failure properly.

Observed behavior in stress testing:

• Some responses contain an access token belonging to another user instead of returning an error.
• This occurs intermittently when database locks or timeouts happen while handling concurrent token requests.

Expected Behavior

When a database timeout or transaction failure occurs, Kong should not return a token for the wrong user. Instead, it should respond with an error.

Steps To Reproduce

1. Run a high-load test sending multiple simultaneous requests to the /oauth2/token endpoint and validate the returned access_token.
2. While the test is running, lock the oauth2_tokens table for a few seconds and then release it.
3. Repeat step 2 several times while observing the responses.

Test setup

• Load Testing Tool: Artillery.io
• Configuration:

local-kong.yaml

config:
  target: "https://127.0.0.1:8443"
  phases:
    - duration: 60
      arrivalRate: 20
      name: Stress
  processor: "./local-kong-processor.js"
scenarios:
  - name: "Test"
    flow:
      - function: generateAuthenticatedUserid
      - post:
          url: "/auth/oauth2/token"
          form:
            provision_key: '<omitted>'
            client_id: '<omitted>'
            client_secret: '<omitted>'
            grant_type: 'password'
            authenticated_userid: '{{ generatedUserId }}'
          capture:
            - json: "$"
              as: generateTokenResponse
      - function: processGenerateToken
      - get:
          url: "http://localhost:8001/oauth2_tokens/{{ accessToken }}"
          capture:
            - json: "$"
              as: getTokenResponse
      - function: "check"

local-kong-processor.js

async function generateAuthenticatedUserid(context, events) {
    const randomNum = Math.floor(Math.random() * 1000000) + 1;
    context.vars.generatedUserId = `{"username":"load-test.${randomNum}"}`;
}

async function processGenerateToken(context, events) {
    const respObj = context.vars.generateTokenResponse;
    context.vars['accessToken'] = respObj.access_token;
}

async function check(context, events) {
    const respObj = context.vars.getTokenResponse;
    const userId = respObj.authenticated_userid;
    const expectedUserId = context.vars.generatedUserId;
    const accessToken = context.vars.accessToken;

    if (userId !== expectedUserId) {
        console.error(">>>> ERROR: Response User ID:", userId, " Generated User ID:", expectedUserId, " Token:", accessToken);
    }
}

module.exports = {
    generateAuthenticatedUserid,
    processGenerateToken,
    check
};

Results:

Anything else?

• Kong Version: 2.8.5
• Database: PostgreSQL
• Kong Docker config:

docker run -d --name kong \
     --network=kong-net \
     -e KONG_DATABASE=postgres \
     -e KONG_PG_HOST=kong-database \
     -e KONG_PG_USER=kong \
     -e KONG_PG_PASSWORD=kongpass \
     -e KONG_TRUSTED_IPS=0.0.0.0/0 \
     -e KONG_CASSANDRA_CONTACT_POINTS=kong-database \
     -e KONG_PROXY_ACCESS_LOG=/dev/stdout \
     -e KONG_ADMIN_ACCESS_LOG=/dev/stdout \
     -e KONG_PROXY_ERROR_LOG=/dev/stderr \
     -e KONG_ADMIN_ERROR_LOG=/dev/stderr \
     -e "KONG_ADMIN_LISTEN=0.0.0.0:8001,0.0.0.0:8444 ssl" \
     -e "KONG_PROXY_LISTEN=0.0.0.0:8000,0.0.0.0:8443 ssl" \
     -e KONG_OAUTH2_AUTHORIZATION_CODE_LOGIN_TO_HTTP_IF_BEHIND_PROXY=true \
     -p 8000:8000 \
     -p 8443:8443 \
     -p 127.0.0.1:8001:8001 \
     -p 127.0.0.1:8444:8444 \
     kong:2.8.5

I also tested this in Kong 3.9.0.1, and the issue does not occur. In this version, Kong correctly returns an error when the database transaction fails, rather than returning an incorrect access token.

Let me know if more details are needed.

[ProBrian]

Hi @alisonbudag , Thank you for your effort, but 2.8.5 is an old version and we only have LTS on 2.8 for Enterprise Edition. We encourage you to upgrade to newer version like 3.9, which doesn't have this issue as you tested.

[robertoneto-senior]

Hi @ProBrian .

We noted that this issue only occurs in scenarios of high Postgres database load. Our current kong database is sharing a PostgreSQL instance with other applications. When this instance gets in high CPU or disk I/O usage, this bug occurs.

So, my question here ir about tunning to prevent this.

Configurations like pg_timeout, pg_max_concurrent_queries, pg_semaphore_timeout and others (https://docs.konghq.com/gateway/2.8.x/reference/configuration/), can help me with this? Can you provide some tips about them? Or refecente some documentation or case how to tune them?