Mutex contention with DNSSECKeeper::getFromMeta #578

Closed
Habbie opened this Issue Apr 26, 2013 · 1 comment

Projects

None yet

1 participant

@Habbie
Member
Habbie commented Apr 26, 2013

Powerdns 3.1 (from the svn tag):

When doing lots of queries which result in unique SOA records being returned (ie create database of 1m zones and then query an unknown subdomain on each) we get very poor performance due to the mutex lock in DNSSECKeeper::getFromMeta() (called from packethandler.cc editSOA())

Our 16-way test server is producing about 30kqps with this line under the conditions described above (and 40% cpu idle due to threads waiting). Commenting out this line we get 62kqps with 0% idle (all caches disabled, with various other mutex patches installed)

As reported by a mutrace run:

Mutex #120319 (0x0x9c7200) first referenced by:
/usr/lib/mutrace/libmutrace.so(pthread_mutex_lock+0x49) [0x7f90243416b9]
pdns/pdns_server(_ZN4LockC1EP15pthread_mutex_t+0x44) [0x5c4008]
pdns/pdns_server(_ZN12DNSSECKeeper11getFromMetaERKSsS1_RSs+0xa2) [0x6d6a4c]
pdns/pdns_server(_Z7editSOAR12DNSSECKeeperRKSsP9DNSPacket+0x1fa) [0x702a06]
pdns/pdns_server(_ZN13PacketHandler17questionOrRecurseEP9DNSPacketPb+0x1d7d) [0x5d61f3]
pdns/pdns_server(_ZN13PacketHandler8questionEP9DNSPacket+0x2b) [0x5d2737]
pdns/pdns_server(_ZN11DistributorI9DNSPacketS0_13PacketHandlerE8questionEPS0_PFvRKNS2_10AnswerDataEE+0x15a) [0x67141c]
pdns/pdns_server(_Z7qthreadPv+0xf93) [0x66e633]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7f902340de9a]

Mutex # Locked Changed Cont. tot.Time[ms] avg.Time[ms] max.Time[ms] Flags
120319 275562 196965 94067 5451.767 0.020 0.758 M-.--.

Thanks,

Mark Zealey

@Habbie Habbie was assigned Apr 26, 2013
@Habbie Habbie closed this Apr 26, 2013
@Habbie
Member
Habbie commented Apr 26, 2013

Author: ahu
Closed by commit 2768. Please test!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment