dns_resolver kernel module cache requests with no TTL ?

[osixia]

Issue Report

Bug

Container Linux Version

$ cat /etc/os-release
NAME="Container Linux by CoreOS"
ID=coreos
VERSION=1353.7.0
VERSION_ID=1353.7.0
BUILD_ID=2017-04-26-2154
PRETTY_NAME="Container Linux by CoreOS 1353.7.0 (Ladybug)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://issues.coreos.com"

Environment

Baremetal

Expected Behavior

kernel dns resolver module seems to cache dns result for an infinite time, a TTL set to dns TTL is expected.

Actual Behavior

I mount cephfs volume with ceph kernel module that use the dns resolver kernel module.

ceph kernel client:
https://github.com/ceph/ceph-client/blob/master/net/ceph/Kconfig#L31
https://github.com/ceph/ceph-client/blob/master/Documentation/networking/dns_resolver.txt
the ceph kernel driver uses CONFIG_DNS_RESOLVER as a default value for CEPH_LIB_USE_DNS_RESOLVER

My ceph monitors are behind a round robin dns:

dig ceph-mon.cluster.local
ceph-mon.cluster.local. 30 IN A 10.244.10.11
ceph-mon.cluster.local. 30 IN A 10.244.20.6
ceph-mon.cluster.local. 30 IN A 10.244.30.13

but ceph still use an ip previously returned by the dns:

libceph: resolve 'ceph-mon.cluster.local' (ret=0): 10.244.10.6:0

the monitor with the ip 10.244.10.6 is now down so i can't mount any volume :(

Other Information

following kernel dns_resolver doc:
https://www.kernel.org/doc/Documentation/networking/dns_resolver.txt

The upcall handler program should set the expiry time on the key to that of the
lowest TTL of all the records it has extracted a result from.  This means that
the key will be discarded and recreated when the data it holds has expired.

as i understand the upcall program (set in /etc/request-key.conf ) for dns_resolver is : /sbin/key.dns_resolver

how can i test if a ttl is actually set to the cached key ?
is there a way to clear the cache ?

[osixia]

Just found that keys are listed in /proc/keys
(1) (2) (3) (4) (5) (6) (7) (8) (9)
02258a45 I--Q--- 1 perm 3b010000 0 0 dns_resol ceph-mon.cluster.local: 12

ID (1) Flags (2) Usage (3) Timeout (4) Permissions (5) UID (6) GID (7) Type (8) Description (9)

Timeout (4):
The amount of time until the key will expire, expressed
in human-readable form (weeks, days, hours, minutes,
and seconds). The string perm here means that the key
is permanent (no timeout). The string expd means that
the key has already expired, but has not yet been
garbage collected.

so has perm show i guess the cache for dns query made by the kernel module dns_resolver never expire :(

[osixia]

Ok i get it, this issue comes from key.dns_resolver not implementing TTL for A and AAAA records.
https://git.kernel.org/pub/scm/linux/kernel/git/dhowells/keyutils.git/tree/key.dns_resolver.c

If it can be useful here my command to clear the kernel dns resolver cache:
sudo keyctl clear $((16#$(sudo cat /proc/keys | grep .dns_resolver | awk '{print $1;}')))

Clothing this as it as nothing to do with coreos :)

Sorry.