New DNS not updating records

[ATL-Flaneur]

The new DNS client pins a DNS record received with a TTL of 0, which can happen when querying a caching resolver.

RFC 8767 states:

TTL: a 32-bit unsigned integer number of seconds that specifies the duration that the resource record MAY be cached before the source of the information MUST again be consulted. Zero values are interpreted to mean that the RR can only be used for the transaction in progress, and should not be cached.

We started getting 504s in API tests running in our AWS VPC after updating to 3.9.3 (current master) and setting KONG_NEW_DNS_CLIENT = on. The reason was Kong not updating the ALB IPs associated with the DNS name. These can change for various reasons, and Kong was black-holing traffic to an IP that no longer existed. Similarly, it wasn't sending traffic to new IPs for that DNS name.

The pin occurs because kong/dns/client.lua computes a TTL with ttl = math_min(ttl, answer.ttl) and passes it to mlcache without checking for zero client.lua:309. Then:

    if ttl == 0 then
        -- indefinite ttl for lua-resty-lrucache is 'nil'
        ttl = nil
    end
    
    self.lru:set(key, value, ttl)

The old client doesn’t have this failure mode as it adds staleTtl and explicitly checks lru_ttl <= 0:

  if lru_ttl <= 0 then
    -- item is already expired, so we do not add it
    dnscache:delete(key)
    --[[
    log(DEBUG, PREFIX, "cache set (delete on expired): ", key, " ", frecord(entry))
    --]]
    return
  end

[identity-wael]

There is a configuration workaround worth testing while the zero-TTL handling is fixed: give the new resolver a small positive override, for example:

KONG_RESOLVER_VALID_TTL=5

The relevant setting is resolver_valid_ttl, not the legacy client's dns_valid_ttl. In the exact revision you linked, a configured self.valid_ttl initializes the result TTL and overwrites each answer's TTL; the math_min(..., answer.ttl) branch runs only when no override is configured. A positive value therefore avoids passing zero along that path.

This is a mitigation, not correct handling of a DNS TTL of zero: it deliberately caches answers for the chosen interval, including answers that requested no caching. Pick an interval suitable for the resolver load and acceptable staleness in your environment. Don't use 0 as a “disable cache” workaround here.

After applying the configuration, make sure the affected workers/caches are recreated, then test a DNS answer change to confirm the ALB addresses refresh. I checked this against the source, but have not reproduced your AWS setup. Reverting new_dns_client to off is the other immediate comparison available in the version you describe.

[ATL-Flaneur]

We tried this, and it worked, but it's a workaround to a bug in the new DNS client and needing to overwrite incoming TTLs didn't inspire the confidence to run it in prod. We switched back to the legacy DNS client that's been working fine and not giving us surprises.

[jmrplens]

Confirmed on master today, and the consequence is worse than "not updating": a TTL of zero does not mean "cache briefly", it means "cache forever", and the record outlives the worker.

The path is resolve_all, which calls self.cache:get(key, nil, resolve_callback, ...) (kong/dns/client.lua L611). The callback returns the answers plus the TTL as its third value (return answers, nil, answers.ttl, L463 and L468), and mlcache takes that third value as the TTL for this entry (kong/resty/mlcache/init.lua, the override ttl / neg_ttl block around L720). The value itself comes from process_answers, which does ttl = math_min(ttl, answer.ttl) (L308) and stores it at L333 and L334 with no zero check, exactly as you found.

Zero is a sentinel in mlcache, in both layers. The L1 path is if ttl == 0 then ttl = nil end with the comment "indefinite ttl for lua-resty-lrucache is 'nil'" (L343 to L345 of the vendored copy), and the L2 path marks the entry with NO_TTL_FLAG and calls dict:set(shm_key, shm_value, 0, flags) (L387 to L389), read back at L495 to L497. L2 is the kong_dns_cache shared dictionary, so the pin is not per worker: every worker re-promotes it into its own LRU on each hit, and it survives kong reload. Only evicting the shm or restarting the node clears it.

The legacy client does not hit this because it never hands a bare zero to a cache that reads zero as forever: it inserts with lru_ttl = ttl + staleTtl (kong/resty/dns/client.lua, around L252 to L279, with staleTtl from dns_stale_ttl), so a zero-TTL record lands already stale and gets refreshed instead of frozen.

I could not find an issue or PR for this in the new client. For what it is worth, the same sentinel has bitten the library before: thibaultcha/lua-resty-mlcache#58, "Fix expiration when the remaining TTL is exactly 0".

On the workaround you tried: resolver_valid_ttl does avoid the pin, but read what it does first. In the same function, if self.valid_ttl then answer.ttl = self.valid_ttl end (L305 to L306) replaces the TTL of every answer, so a value of 5 makes Kong re-resolve every name every five seconds no matter what the zone publishes, which is a fair reason not to want it in production. And do not set it to 0, because that goes straight back through L300 and recreates the pin for everything.

For anyone arriving from a default install: new_dns_client is still off in kong.conf.default, so this only affects those who turned it on, but it will matter more when the new client becomes the default.

[ATL-Flaneur]

I opened #15021 and pointed to this thread.