LMDB backend can crash after reloading data #1954

Closed
cmouse opened this Issue Dec 14, 2014 · 1 comment

Projects

None yet

2 participants

@cmouse
Contributor
cmouse commented Dec 14, 2014

Found during travis test:

[2014-12-14 18:43:33] escaped-txt: 
[2014-12-14 18:43:33] 4 TXT records with 0 to 3 backslashes before a semicolon.
[2014-12-14 18:43:33] Dec 14 18:43:33 Loading LMDB database ./
[2014-12-14 18:43:33] Dec 14 18:43:33 Loading LMDB database ./
[2014-12-14 18:43:33] pdns_server: mdb.c:4656: mdb_page_get: Assertion `p != ((void *)0)' failed.
[2014-12-14 18:43:33] Dec 14 18:43:33 Got a signal 6, attempting to print trace: 
[2014-12-14 18:43:33] Dec 14 18:43:33 ../pdns/pdns_server() [0x581b0d]
[2014-12-14 18:43:33] Dec 14 18:43:33 /lib/x86_64-linux-gnu/libc.so.6(+0x364c0) [0x7f5a13a7e4c0]
[2014-12-14 18:43:33] Dec 14 18:43:33 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7f5a13a7e445]
[2014-12-14 18:43:33] Dec 14 18:43:33 /lib/x86_64-linux-gnu/libc.so.6(abort+0x17b) [0x7f5a13a81bab]
[2014-12-14 18:43:33] Dec 14 18:43:33 /lib/x86_64-linux-gnu/libc.so.6(+0x2f10e) [0x7f5a13a7710e]
[2014-12-14 18:43:33] Dec 14 18:43:33 /lib/x86_64-linux-gnu/libc.so.6(+0x2f1b2) [0x7f5a13a771b2]
[2014-12-14 18:43:33] Dec 14 18:43:33 /usr/lib/x86_64-linux-gnu/liblmdb.so.0(+0x2a11) [0x7f5a0ea62a11]
[2014-12-14 18:43:33] Dec 14 18:43:33 /usr/lib/x86_64-linux-gnu/liblmdb.so.0(+0x63a6) [0x7f5a0ea663a6]
[2014-12-14 18:43:33] Dec 14 18:43:33 /usr/lib/x86_64-linux-gnu/liblmdb.so.0(+0x667b) [0x7f5a0ea6667b]
[2014-12-14 18:43:33] Dec 14 18:43:33 /usr/lib/x86_64-linux-gnu/liblmdb.so.0(+0x6e6d) [0x7f5a0ea66e6d]
[2014-12-14 18:43:33] Dec 14 18:43:33 /usr/lib/x86_64-linux-gnu/liblmdb.so.0(mdb_cursor_get+0x2fb) [0x7f5a0ea65abb]
[2014-12-14 18:43:33] Dec 14 18:43:33 ./modules/liblmdbbackend.so(_ZN11LMDBBackend3getER17DNSResourceRecord+0x44f) [0x7f5a0ec84f7f]
[2014-12-14 18:43:33] Dec 14 18:43:33 ../pdns/pdns_server(_ZN12UeberBackend6handle3getER17DNSResourceRecord+0x29) [0x5e9ee9]
[2014-12-14 18:43:33] Dec 14 18:43:33 ../pdns/pdns_server(_ZN12UeberBackend3getER17DNSResourceRecord+0xa7) [0x5e9dd7]
[2014-12-14 18:43:33] Dec 14 18:43:33 ../pdns/pdns_server(_ZN13PacketHandler17getBestReferralNSEP9DNSPacketR7SOADataRKSs+0x103) [0x55d523]
[2014-12-14 18:43:33] Dec 14 18:43:33 ../pdns/pdns_server(_ZN13PacketHandler11tryReferralEP9DNSPacketS1_R7SOADataRKSsb+0x35) [0x56c3e5]
[2014-12-14 18:43:33] Dec 14 18:43:33 ../pdns/pdns_server(_ZN13PacketHandler17questionOrRecurseEP9DNSPacketPb+0x1133) [0x566eb3]
[2014-12-14 18:43:33] Dec 14 18:43:33 ../pdns/pdns_server(_ZN13PacketHandler8questionEP9DNSPacket+0xc9) [0x565cb9]
[2014-12-14 18:43:33] Dec 14 18:43:33 ../pdns/pdns_server(_ZN22MultiThreadDistributorI9DNSPacketS0_13PacketHandlerE10makeThreadEPv+0x340) [0x4bdf00]
[2014-12-14 18:43:33] Dec 14 18:43:33 /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7f5a13e0fe9a]

The bit about loading LMDB database at ./ looks very suspicious.

@cmouse cmouse referenced this issue Jan 9, 2015
Closed

Gsql prepared statements #1783

3 of 8 tasks complete
@cmouse
Contributor
cmouse commented Feb 19, 2015

Found the root cause. During database reload, there is a moment when the database is closed and not yet opened, during which we accept queries, as lookups and others are not blocked during database reload.

@Habbie Habbie pushed a commit that closed this issue Feb 24, 2015
@cmouse cmouse Properly lock lmdb database, fixes #1954
The LMDB database needs to be reloaded without allowing requests, so
we use readwrite lock to ensure that this cannot happen.
08631ed
@Habbie Habbie closed this in 08631ed Feb 24, 2015
@cmouse cmouse added a commit to cmouse/pdns that referenced this issue Mar 5, 2015
@cmouse cmouse Properly lock lmdb database, fixes #1954
The LMDB database needs to be reloaded without allowing requests, so
we use readwrite lock to ensure that this cannot happen.
c40307b
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment