pdns_server 3.4.1 Ubuntu abort using chroot: libgcc_s.so.1 must be installed #1907

Closed
lifeforms opened this Issue Dec 6, 2014 · 2 comments

Projects

None yet

3 participants

@lifeforms

When trying PowerDNS 3.4.1 on Ubuntu 12.04, we get a signal 6 (abort) if the chroot feature is used.

Using the static deb file from http://downloads.powerdns.com/releases/deb/pdns-static_3.4.1-1_amd64.deb.

Interesting part in the stack trace is #7 0x00007f97b1cb211e in __GI___libc_fatal (message=0x7f97b250ede8 "libgcc_s.so.1 must be installed for pthread_cancel to work\n")

The problem goes away when libgcc_s.so.1 is copied into the chroot, e.g. /var/spool/powerdns/lib/x86_64-linux-gnu/libgcc_s.so.1. Then pdns appears to be completely stable.

Downgrading back to version 3.3 also resolves the problem.

Not much is logged to syslog.

pdns.conf is pretty trivial (also attempted with numeric uid/gids; loglevel 9 does not give more interesting info):

# This file is managed by Puppet.
# Any manual edits will be overwritten. Please contact support for changes.

daemon=yes
guardian=yes
launch=gmysql
gmysql-host=127.0.0.1
gmysql-dbname=powerdns
gmysql-user=powerdns
gmysql-password=XXX
local-ipv6=::  # for IPv6
setuid=pdns
setgid=pdns
chroot=/var/spool/powerdns
disable-axfr=yes
slave=yes
loglevel=3
log-dns-details=no
version-string=anonymous
webserver=no

The chroot directory is empty.

Syslog output:

pdns[2999]: Guardian is launching an instance
pdns[2999]: Reading random entropy from '/dev/urandom'
pdns[2999]: This is a guarded instance of pdns
pdns[2999]: UDP server bound to 0.0.0.0:53
pdns[2999]: UDPv6 server bound to [::]:53
pdns[2999]: TCP server bound to 0.0.0.0:53
pdns[2999]: TCPv6 server bound to [::]:53
pdns[2999]: PowerDNS Authoritative Server 3.4.1 (jenkins@autotest.powerdns.com) (C) 2001-2014 PowerDNS.COM BV
pdns[2999]: Using 64-bits mode. Built on 20141030110058 by root@autotest.powerdns.com, gcc 4.7.2.
pdns[2999]: PowerDNS comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it according to the terms of the GPL version 2.
pdns[2999]: Polled security status of version 3.4.1 at startup, no known issues reported: OK
pdns[2999]: Set effective group id to 222
pdns[2999]: Chrooted to '/var/spool/powerdns'
pdns[2999]: Set effective user id to 222
pdns[2999]: Creating backend connection for TCP
pdns[2999]: Got a signal 6, attempting to print trace:
pdns[2978]: Our pdns instance (2999) exited after signal 6
pdns[2978]: Dumped core
pdns[2978]: Respawning

Stack traces:

Thread 6 (Thread 0x7f97b0fdd700 (LWP 4121)):
#0  clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:84
#1  0x00007f97b2504dc0 in ?? () at pthread_create.c:172 from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007f97b0fdd700 in ?? ()
#3  0x0000000000000000 in ?? ()

Thread 5 (Thread 0x7f97b07dc700 (LWP 4122)):
#0  clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:84
#1  0x00007f97b2504dc0 in ?? () at pthread_create.c:172 from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007f97b07dc700 in ?? ()
#3  0x0000000000000000 in ?? ()

Thread 4 (Thread 0x7f97affdb700 (LWP 4123)):
#0  clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:84
#1  0x00007f97b2504dc0 in ?? () at pthread_create.c:172 from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007f97affdb700 in ?? ()
#3  0x0000000000000000 in ?? ()

Thread 3 (Thread 0x7f97af7da700 (LWP 4124)):
#0  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
#1  0x00000000006a0e72 in Semaphore::wait() ()
#2  0x000000000068d92a in CommunicatorClass::retrievalLoopThread() ()
#3  0x000000000068f769 in CommunicatorClass::retrieveLaunchhelper(void*) ()
#4  0x00007f97b2504e9a in start_thread (arg=0x7f97af7da700) at pthread_create.c:308
#5  0x00007f97b1d322ed in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 2 (Thread 0x7f97b2930700 (LWP 4120)):
#0  0x00007f97b2506148 in pthread_join (threadid=140289452644096, thread_return=0x0) at pthread_join.c:89
#1  0x000000000077a52a in my_thread_global_init ()
#2  0x0000000000776708 in my_init ()
#3  0x0000000000759776 in mysql_server_init ()
#4  0x000000000076160f in mysql_init ()
#5  0x000000000057088f in SMySQL::SMySQL(std::string const&, std::string const&, unsigned short, std::string const&, std::string const&, std::string const&, std::string const&, bool) ()
#6  0x0000000000568d18 in gMySQLBackend::gMySQLBackend(std::string const&, std::string const&) ()
#7  0x0000000000570215 in gMySQLFactory::make(std::string const&) ()
#8  0x000000000062b59a in BackendMakerClass::all(bool) ()
#9  0x000000000065bef0 in UeberBackend::UeberBackend(std::string const&) ()
#10 0x0000000000600295 in PacketHandler::PacketHandler() ()
#11 0x000000000060b7cd in TCPNameserver::go() ()
#12 0x000000000069bbee in mainthread() ()
#13 0x0000000000546885 in main ()

Thread 1 (LWP 4125):
#0  0x00007f97b1c740d5 in __GI_raise (sig=<optimized out>) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1  0x00007f97b1c7783b in __GI_abort () at abort.c:91
#2  0x0000000000659c89 in ?? ()
#3  <signal handler called>
#4  0x00007f97b1c740d5 in __GI_raise (sig=<optimized out>) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#5  0x00007f97b1c7783b in __GI_abort () at abort.c:91
#6  0x00007f97b1cb204e in __libc_message (do_abort=1, fmt=0x7f97b1db9ff6 "%s") at ../sysdeps/unix/sysv/linux/libc_fatal.c:201
#7  0x00007f97b1cb211e in __GI___libc_fatal (message=0x7f97b250ede8 "libgcc_s.so.1 must be installed for pthread_cancel to work\n")
#8  0x00007f97b250d491 in pthread_cancel_init () at ../nptl/sysdeps/pthread/unwind-forcedunwind.c:65
#9  0x00007f97b250d57c in _Unwind_ForcedUnwind (exc=<optimized out>, stop=<optimized out>, stop_argument=<optimized out>)
    at ../nptl/sysdeps/pthread/unwind-forcedunwind.c:130
#10 0x00007f97b250b6a0 in __GI___pthread_unwind (buf=<optimized out>) at unwind.c:130
#11 0x00007f97b2505ff5 in __do_cancel () at pthreadP.h:265
#12 __pthread_exit (value=<optimized out>) at pthread_exit.c:30
#13 0x000000000077979b in ?? ()
#14 0x00007f97b2504e9a in start_thread (arg=0x7f97aefd9700) at pthread_create.c:308
#15 0x00007f97b1d322ed in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#16 0x0000000000000000 in ?? ()
@Habbie Habbie added defect auth labels Dec 6, 2014
@ahupowerdns ahupowerdns added a commit that referenced this issue Dec 6, 2014
@ahupowerdns ahupowerdns Potential fix for #1907, we now try to trigger libgcc_s.so.1 to load …
…before we chroot. I can't reproduce the bug on my local system,

but this "should" help. Seriously.
2250baf
@ahupowerdns
Member

Thank you for the very thorough report. Apache has also run into this issue,
see http://core.segfault.pl/~hobbit/mod_chroot/caveats.html . I think later
versions of glibc no longer attempt late loading of libgcc_s.so, so I can't
reproduce your issue here. I have committed a potential fix though, which is
now building, and which you could test using the latest debs soon available
on https://autotest.powerdns.com/

On Sat, Dec 06, 2014 at 08:26:55AM -0800, Walter Hop wrote:

When trying PowerDNS 3.4.1 on Ubuntu 12.04, we get a signal 6 (abort) if the chroot feature is used.

Using the static deb file from http://downloads.powerdns.com/releases/deb/pdns-static_3.4.1-1_amd64.deb.

Interesting part in the stack trace is #7 0x00007f97b1cb211e in __GI___libc_fatal (message=0x7f97b250ede8 "libgcc_s.so.1 must be installed for pthread_cancel to work\n")

The problem goes away when libgcc_s.so.1 is copied into the chroot, e.g. /var/spool/powerdns/lib/x86_64-linux-gnu/libgcc_s.so.1. Then pdns appears to be completely stable.

Downgrading back to version 3.3 also resolves the problem.

Not much is logged to syslog.

pdns.conf is pretty trivial (also attempted with numeric uid/gids; loglevel 9 does not give more interesting info):

# This file is managed by Puppet.
# Any manual edits will be overwritten. Please contact support for changes.

daemon=yes
guardian=yes
launch=gmysql
gmysql-host=127.0.0.1
gmysql-dbname=powerdns
gmysql-user=powerdns
gmysql-password=XXX
local-ipv6=::  # for IPv6
setuid=pdns
setgid=pdns
chroot=/var/spool/powerdns
disable-axfr=yes
slave=yes
loglevel=3
log-dns-details=no
version-string=anonymous
webserver=no

The chroot directory is empty.

Syslog output:

pdns[2999]: Guardian is launching an instance
pdns[2999]: Reading random entropy from '/dev/urandom'
pdns[2999]: This is a guarded instance of pdns
pdns[2999]: UDP server bound to 0.0.0.0:53
pdns[2999]: UDPv6 server bound to [::]:53
pdns[2999]: TCP server bound to 0.0.0.0:53
pdns[2999]: TCPv6 server bound to [::]:53
pdns[2999]: PowerDNS Authoritative Server 3.4.1 (jenkins@autotest.powerdns.com) (C) 2001-2014 PowerDNS.COM BV
pdns[2999]: Using 64-bits mode. Built on 20141030110058 by root@autotest.powerdns.com, gcc 4.7.2.
pdns[2999]: PowerDNS comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it according to the terms of the GPL version 2.
pdns[2999]: Polled security status of version 3.4.1 at startup, no known issues reported: OK
pdns[2999]: Set effective group id to 222
pdns[2999]: Chrooted to '/var/spool/powerdns'
pdns[2999]: Set effective user id to 222
pdns[2999]: Creating backend connection for TCP
pdns[2999]: Got a signal 6, attempting to print trace:
pdns[2978]: Our pdns instance (2999) exited after signal 6
pdns[2978]: Dumped core
pdns[2978]: Respawning

Stack traces:

Thread 6 (Thread 0x7f97b0fdd700 (LWP 4121)):
#0 clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:84
#1 0x00007f97b2504dc0 in ?? () at pthread_create.c:172 from /lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007f97b0fdd700 in ?? ()
#3 0x0000000000000000 in ?? ()

Thread 5 (Thread 0x7f97b07dc700 (LWP 4122)):
#0 clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:84
#1 0x00007f97b2504dc0 in ?? () at pthread_create.c:172 from /lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007f97b07dc700 in ?? ()
#3 0x0000000000000000 in ?? ()

Thread 4 (Thread 0x7f97affdb700 (LWP 4123)):
#0 clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:84
#1 0x00007f97b2504dc0 in ?? () at pthread_create.c:172 from /lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007f97affdb700 in ?? ()
#3 0x0000000000000000 in ?? ()

Thread 3 (Thread 0x7f97af7da700 (LWP 4124)):
#0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
#1 0x00000000006a0e72 in Semaphore::wait() ()
#2 0x000000000068d92a in CommunicatorClass::retrievalLoopThread() ()
#3 0x000000000068f769 in CommunicatorClass::retrieveLaunchhelper(void*) ()
#4 0x00007f97b2504e9a in start_thread (arg=0x7f97af7da700) at pthread_create.c:308
#5 0x00007f97b1d322ed in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6 0x0000000000000000 in ?? ()

Thread 2 (Thread 0x7f97b2930700 (LWP 4120)):
#0 0x00007f97b2506148 in pthread_join (threadid=140289452644096, thread_return=0x0) at pthread_join.c:89
#1 0x000000000077a52a in my_thread_global_init ()
#2 0x0000000000776708 in my_init ()
#3 0x0000000000759776 in mysql_server_init ()
#4 0x000000000076160f in mysql_init ()
#5 0x000000000057088f in SMySQL::SMySQL(std::string const&, std::string const&, unsigned short, std::string const&, std::string const&, std::string const&, std::string const&, bool) ()
#6 0x0000000000568d18 in gMySQLBackend::gMySQLBackend(std::string const&, std::string const&) ()
#7 0x0000000000570215 in gMySQLFactory::make(std::string const&) ()
#8 0x000000000062b59a in BackendMakerClass::all(bool) ()
#9 0x000000000065bef0 in UeberBackend::UeberBackend(std::string const&) ()
#10 0x0000000000600295 in PacketHandler::PacketHandler() ()
#11 0x000000000060b7cd in TCPNameserver::go() ()
#12 0x000000000069bbee in mainthread() ()
#13 0x0000000000546885 in main ()

Thread 1 (LWP 4125):
#0 0x00007f97b1c740d5 in __GI_raise (sig=) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1 0x00007f97b1c7783b in __GI_abort () at abort.c:91
#2 0x0000000000659c89 in ?? ()
#3
#4 0x00007f97b1c740d5 in __GI_raise (sig=) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#5 0x00007f97b1c7783b in __GI_abort () at abort.c:91
#6 0x00007f97b1cb204e in __libc_message (do_abort=1, fmt=0x7f97b1db9ff6 "%s") at ../sysdeps/unix/sysv/linux/libc_fatal.c:201
#7 0x00007f97b1cb211e in __GI___libc_fatal (message=0x7f97b250ede8 "libgcc_s.so.1 must be installed for pthread_cancel to work\n")
#8 0x00007f97b250d491 in pthread_cancel_init () at ../nptl/sysdeps/pthread/unwind-forcedunwind.c:65
#9 0x00007f97b250d57c in _Unwind_ForcedUnwind (exc=, stop=, stop_argument=)
at ../nptl/sysdeps/pthread/unwind-forcedunwind.c:130
#10 0x00007f97b250b6a0 in __GI___pthread_unwind (buf=) at unwind.c:130
#11 0x00007f97b2505ff5 in __do_cancel () at pthreadP.h:265
#12 __pthread_exit (value=) at pthread_exit.c:30
#13 0x000000000077979b in ?? ()
#14 0x00007f97b2504e9a in start_thread (arg=0x7f97aefd9700) at pthread_create.c:308
#15 0x00007f97b1d322ed in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#16 0x0000000000000000 in ?? ()


Reply to this email directly or view it on GitHub:
#1907

@lifeforms

The proposed change fixes the issue! GREAT! :)

@lifeforms lifeforms closed this Dec 6, 2014
@mind04 mind04 added a commit to mind04/pdns that referenced this issue Dec 22, 2014
@ahupowerdns @mind04 ahupowerdns + mind04 Potential fix for #1907, we now try to trigger libgcc_s.so.1 to load …
…before we chroot. I can't reproduce the bug on my local system,

but this "should" help. Seriously.
1c57e1d
@Lance0312 Lance0312 added a commit to Lance0312/pdns that referenced this issue Apr 5, 2016
@Lance0312 Lance0312 Do triggerLoadOfLibraries() only in glibc environment
The workaround of preloading libgcc_s.so.1 to memory by creating a dummy
thread from #1907 makes pdns fail to run in musl libc environment with
`setuid` and `chroot` enabled at the same time. musl libc [implements
set\*id calls AS-safe](http://git.musl-libc.org/cgit/musl/commit/?id=78a8ef47c4d92b7680c52a85f80a81e29da86bb9)
by reading `/proc/self/task` to get information of all threads. And
since most users `chroot` to an empty directory for security reason,
`setuid` cannot find `/proc/self/task` in the new root.

I've started [a discussion thread on musl's mailing list](http://www.openwall.com/lists/musl/2016/04/01/15),
however, until the Linux kernel provides truly atomic set\*id calls,
which is a really tough task according to Rich Felker, there seems no
easy way to keep set\*id AS-safe without using `/proc/self/task`.

Since #1907 is an issue of glibc, I think it is rather safe to do the
workaround only in glibc environment.
f64ffb5
@Lance0312 Lance0312 added a commit to Lance0312/pdns that referenced this issue Apr 5, 2016
@Lance0312 Lance0312 Do triggerLoadOfLibraries() only in glibc environment
The workaround of preloading libgcc_s.so.1 to memory by creating a dummy
thread from #1907 makes pdns fail to run in musl libc environment with
`setuid` and `chroot` enabled at the same time. musl libc [implements
set\*id calls AS-safe](http://git.musl-libc.org/cgit/musl/commit/?id=78a8ef47c4d92b7680c52a85f80a81e29da86bb9)
by reading `/proc/self/task` to get information of all threads. And
since most users `chroot` to an empty directory for security reason,
`setuid` cannot find `/proc/self/task` in the new root.

I've started [a discussion thread on musl's mailing list](http://www.openwall.com/lists/musl/2016/04/01/15),
however, until the Linux kernel provides truly atomic set\*id calls,
which is a really tough task according to Rich Felker, there seems no
easy way to keep set\*id AS-safe without using `/proc/self/task`.

Since #1907 is an issue of glibc, I think it is rather safe to do the
fix only in glibc environment, and as a consequence mitigate the
combinational use of `pthread`, `chroot`, and `setuid` in musl libc
environment.
f7c6353
@Lance0312 Lance0312 added a commit to Lance0312/pdns that referenced this issue Apr 5, 2016
@Lance0312 Lance0312 Do triggerLoadOfLibraries() only in glibc environment
The workaround of preloading libgcc_s.so.1 to memory by creating a dummy
thread from #1907 makes pdns fail to run in musl libc environment with
`setuid` and `chroot` enabled at the same time. musl libc [implements
set\*id calls AS-safe][1]
by reading `/proc/self/task` to get information of all threads. And
since most users `chroot` to an empty directory for security reason,
`setuid` cannot find `/proc/self/task` in the new root.

I've started [a discussion thread on musl's mailing list](http://www.openwall.com/lists/musl/2016/04/01/15),
however, until the Linux kernel provides truly atomic set\*id calls,
which is a really tough task according to Rich Felker, there seems no
easy way to keep set\*id AS-safe without using `/proc/self/task`.

Since #1907 is an issue of glibc, I think it is rather safe to do the
fix only in glibc environment, and as a consequence mitigate the
combinational use of `pthread`, `chroot`, and `setuid` in musl libc
environment.

[1][http://git.musl-libc.org/cgit/musl/commit/?id=78a8ef47c4d92b7680c52a85f80a81e29da86bb9]
9e95b6e
@Lance0312 Lance0312 added a commit to Lance0312/pdns that referenced this issue Apr 5, 2016
@Lance0312 Lance0312 Do triggerLoadOfLibraries() only in glibc environment
The workaround of preloading libgcc_s.so.1 to memory by creating a dummy
thread from #1907 makes pdns fail to run in musl libc environment with
`setuid` and `chroot` enabled at the same time. musl libc [implements
set\*id calls AS-safe][1]
by reading `/proc/self/task` to get information of all threads. And
since most users `chroot` to an empty directory for security reason,
`setuid` cannot find `/proc/self/task` in the new root.

I've started [a discussion thread on musl's mailing list](http://www.openwall.com/lists/musl/2016/04/01/15),
however, until the Linux kernel provides truly atomic set\*id calls,
which is a really tough task according to Rich Felker, there seems no
easy way to keep set\*id AS-safe without using `/proc/self/task`.

Since #1907 is an issue of glibc, I think it is rather safe to do the
fix only in glibc environment, and as a consequence mitigate the
combinational use of `pthread`, `chroot`, and `setuid` in musl libc
environment.

[1]: http://git.musl-libc.org/cgit/musl/commit/?id=78a8ef47c4d92b7680c52a85f80a81e29da86bb9
5af253b
@Lance0312 Lance0312 added a commit to Lance0312/pdns that referenced this issue Apr 5, 2016
@Lance0312 Lance0312 Do triggerLoadOfLibraries() only in glibc environment
The workaround of preloading libgcc_s.so.1 to memory by creating a dummy
thread from #1907 makes pdns fail to run in musl libc environment with
`setuid` and `chroot` enabled at the same time. musl libc [implements
set\*id calls AS-safe](http://git.musl-libc.org/cgit/musl/commit/?id=78a8ef47c4d92b7680c52a85f80a81e29da86bb9)
by reading `/proc/self/task` to get information of all threads. And
since most users `chroot` to an empty directory for security reason,
`setuid` cannot find `/proc/self/task` in the new root.

I've started [a discussion thread on musl's mailing list](http://www.openwall.com/lists/musl/2016/04/01/15),
however, until the Linux kernel provides truly atomic set\*id calls,
which is a really tough task according to Rich Felker, there seems no
easy way to keep set\*id AS-safe without using `/proc/self/task`.

Since #1907 is an issue of glibc, I think it is rather safe to do the
fix only in glibc environment, and as a consequence mitigate the
combinational use of `pthread`, `chroot`, and `setuid` in musl libc
environment.
fe040e1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment