Fatal error: glibc detected an invalid stdio handle

[teto]

Happening with the clang branch on ubuntu 16.10
Running the tests generates:
"Fatal error: glibc detected an invalid stdio handle\n"
while they used to work

Description of the problem

./waf --cwd=/home/teto/dce --run test-runner --command-template="gdb -ex 'run --suite=dce-process-manager --verbose --tempdir=/tmp/tmp7xpcdjr5 ' --args %s "

[Switching to Thread 0x7ffff7f80700 (LWP 5419)]
__GI_raise (Reading in symbols for abort.c...done.
sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:58
58	../sysdeps/unix/sysv/linux/raise.c: Aucun fichier ou dossier de ce type.
Current language:  auto
The current source language is "auto; currently c".
(gdb) bt
Reading in symbols for ../sysdeps/unix/sysv/linux/libc_fatal.c...done.
Reading in symbols for vtables.c...done.
Reading in symbols for ioseekoff.c...done.
Reading in symbols for fseek.c...done.
Reading in symbols for /home/teto/dce_clang/model/dce-stdio.cc...done.
Reading in symbols for /home/teto/dce_clang/model/dce-global-variables.cc...done.
Reading in symbols for ../model/libc-setup.cc...done.
Reading in symbols for ../model/libc.cc...done.
Reading in symbols for /home/teto/dce_clang/model/dce-manager.cc...done.
Reading in symbols for /home/teto/dce_clang/model/task-manager.cc...done.
Reading in symbols for /home/teto/dce_clang/model/pthread-fiber-manager.cc...done.
Reading in symbols for pthread_create.c...done.
Reading in symbols for ../sysdeps/unix/sysv/linux/x86_64/clone.S...done.
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:58
#1  0x00007fffed2d63ea in __GI_abort () at abort.c:89
#2  0x00007fffed3180d0 in __libc_message (do_abort=do_abort@entry=1, fmt=fmt@entry=0x7fffed42b716 "%s")
    at ../sysdeps/posix/libc_fatal.c:175
#3  0x00007fffed318112 in __GI___libc_fatal (
    message=message@entry=0x7fffed42d2a8 "Fatal error: glibc detected an invalid stdio handle\n") at ../sysdeps/posix/libc_fatal.c:185
#4  0x00007fffed318916 in _IO_vtable_check () at vtables.c:71
#5  0x00007fffed30fc59 in IO_validate_vtable (vtable=0x7ffff7971370 <dce_fdopen::vtable>) at ../libio/libioP.h:942
#6  _IO_seekoff_unlocked (fp=0x7fffe0002310, offset=-1, dir=<optimized out>, mode=<optimized out>) at ioseekoff.c:65
#7  0x00007fffed316869 in __GI_fseek (fp=0x7fffe0002310, offset=<optimized out>, whence=<optimized out>) at fseek.c:36
#8  0x00007ffff763297f in dce_fseek (stream=0x7fffe0002310, offset=-1, whence=0) at ../model/dce-stdio.cc:586
#9  0x00007ffff76324e9 in dce_fdopen (fildes=0, mode=0x7ffff76fec6f "r") at ../model/dce-stdio.cc:214
#10 0x00007ffff765f299 in dce_global_variables_setup (variables=0x7ffff7f7ef40) at ../model/dce-global-variables.cc:22
#11 0x00007fffe6b65e2f in setup_global_variables () at ../model/libc-setup.cc:43
#12 0x00007fffe6b65d7f in libc_setup (fn=0x7fffe00008c0) at ../model/libc.cc:352
#13 0x00007ffff75bbcb9 in ns3::DceManager::LoadMain (ld=Reading in symbols for /home/teto/dce_clang/model/loader-factory.cc...done.
0x61f590, Reading in symbols for ../src/core/model/time.cc...done.
filename="/home/teto/dce_clang/build/bin_dce/test-empty", proc=0x6e09a0, err=@0x7ffff7f7fb84: 0) at ../model/dce-manager.cc:1237
#14 0x00007ffff75bb689 in ns3::DceManager::PrepareDoStartProcess (current=0x6bf540) at ../model/dce-manager.cc:267
#15 0x00007ffff75bc8cd in ns3::DceManager::DoStartProcess (context=0x6bf540) at ../model/dce-manager.cc:292
#16 0x00007ffff766ef13 in ns3::TaskManager::Trampoline (context=0x6bf710) at ../model/task-manager.cc:275
#17 0x00007ffff7669a88 in ns3::PthreadFiberManager::Run (arg=0x64d740) at ../model/pthread-fiber-manager.cc:402
#18 0x00007fffed66d70a in start_thread (arg=0x7ffff7f80700) at pthread_create.c:333
#19 0x00007fffed3a70af in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105

There is a similar issue here:
https://bugzilla.redhat.com/show_bug.cgi?id=1361037
with an interesting comment:

One _should_ be using '#include <gnu/lib-names.h>' to get LIBC_SO and then dlopen that, it's the only supported solution,
I'll propose a patch to do that.

[teto]

Some interesting info here
https://sourceware.org/glibc/wiki/LibioVtables

Patch in question proposed in https://www.sourceware.org/ml/libc-alpha/2016-08/msg00320.html

[teto]

So I went on #glibc to ask how to deal with it but clearly DCE should not tamper with the vtable as it was the goal of the hardening in the first place. So we have to find another way.
So I had a look at https://github.com/direct-code-execution/ns-3-dce/blob/master/model/dce-stdio.cc#L196
and apparently overriding the virtual table just helps setting "errno".
If I take "my_close" as an example, it calls dce_close which sets possibly sets "current->err = EBADF;"
my_close then checks if there was an error and sets "errno = Current ()->err;". Is there a case where this could not be atomic ?
My current idea would be to have dce_close do "errno = current->err = EBADF;" ?

[thehajime]

it looks like not related to clang support itself, i removed the project label.

[thehajime]

We need to have our own vtable to handle our private syscalls (my_read etc).
I'm going to investigate this.

[mickeyze]

Dear Hajime,
I'm trying to install NS-3 DCE on Ubuntu 18.04 LTS but we have the same issue.

./waf --run dce-zebra-simple
Waf: Entering directory `/home/mickeyze/ns3/dce/dce/source/ns-3-dce/build'

--------------------------------------------------------------------
 Python bindings compilation 
--------------------------------------------------------------------
[ 11/348] Creating build/lib/pkgconfig/libns3-dev-netlink-debug.pc
[112/348] Creating build/lib/pkgconfig/libns3-dev-dce-debug.pc
[115/348] Creating build/myscripts/ns-3-dce-quagga/lib/pkgconfig/libns3-dev-dce-quagga-debug.pc
Waf: Leaving directory `/home/mickeyze/ns3/dce/dce/source/ns-3-dce/build'
Build commands will be stored in build/compile_commands.json
'build' finished successfully (0.261s)
Fatal error: glibc detected an invalid stdio handle

Please, can you provide some instructions how to overcome this issue?
Thank you!

I installed DCE using following:

#bake.py configure -e dce-ns3-dev -e dce-quagga-dev
#bake.py download

 >> Searching for system dependency libsysfs-dev - OK
 >> Downloading iperf - (Nothing to do, source directory already exists) - OK
 >> Searching for system dependency libexpat-dev - OK
 >> Searching for system dependency libpcap-dev - OK
 >> Searching for system dependency gawk - OK
 >> Searching for system dependency indent - OK
 >> Downloading bash - (Nothing to do, source directory already exists) - OK
 >> Downloading thttpd - (Nothing to do, source directory already exists) - OK
 >> Downloading wget - (Nothing to do, source directory already exists) - OK
 >> Searching for system dependency libgoocanvas2 - OK
 >> Searching for system dependency gi-cairo - OK
 >> Searching for system dependency pygobject - OK
 >> Searching for system dependency pygoocanvas - (Nothing to do, source directory already exists) - OK
 >> Searching for system dependency pygraphviz - OK
 >> Searching for system dependency python-dev - OK
 >> Searching for system dependency llvm-dev - OK
 >> Searching for system dependency clang-dev - OK
 >> Searching for system dependency qt4 - OK
 >> Searching for system dependency qt - OK
 >> Downloading elf-loader-patch (target directory:elf-loader-patch) - (Nothing to do, source directory already exists) - OK
 >> Searching for system dependency libc - OK
 >> Searching for system dependency g++ - OK
 >> Searching for system dependency cxxfilt - OK
 >> Searching for system dependency setuptools - OK
 >> Downloading libaspect - (Nothing to do, source directory already exists) - OK
 >> Searching for system dependency cmake - OK
 >> Downloading ccnx - (Nothing to do, source directory already exists) - OK
 >> Downloading quagga - (Nothing to do, source directory already exists) - OK
 >> Downloading iputils - (Nothing to do, source directory already exists) - OK
 >> Downloading castxml - (Nothing to do, source directory already exists) - OK
 >> Searching for system dependency libc-debug - (Nothing to do, source directory already exists) - OK
 >> Downloading pybindgen - (Nothing to do, source directory already exists) - OK
 >> Downloading netanim-3.107 - (Nothing to do, source directory already exists) - OK
 >> Downloading netanim - (Nothing to do, source directory already exists) - OK
 >> Downloading pygccxml - (Nothing to do, source directory already exists) - OK
 >> Downloading elf-loader - (Nothing to do, source directory already exists) - OK
 >> Downloading ns-3-dev - (Nothing to do, source directory already exists) - OK
 >> Downloading dce-meta-dev (target directory:ns-3-dce) - (Nothing to do, source directory already exists) - OK
 >> Downloading pybindgen-0.17.0.post57+nga6376f2 (target directory:pybindgen) - (Nothing to do, source directory already exists) - OK
 >> Downloading dce-ns3-dev (target directory:ns-3-dce) - (Nothing to do, source directory already exists) - OK
 >> Downloading dce-quagga-dev (target directory:ns-3-dce/myscripts/ns-3-dce-quagga/) - (Nothing to do, source directory already exists) - OK
 >> Downloading ns-3.26 - (Nothing to do, source directory already exists) - OK
 >> Downloading dce-meta-1.9 (target directory:ns-3-dce) - (Nothing to do, source directory already exists) - OK
 >> Downloading dce-quagga-1.9 (target directory:ns-3-dce/myscripts/ns-3-dce-quagga/) - (Nothing to do, source directory already exists) - OK
 >> Downloading dce-ns3-1.9 (target directory:ns-3-dce) - (Nothing to do, source directory already exists) - OK

#bake.py build -vvv
.....
Build commands will be stored in build/compile_commands.json
'install' finished successfully (0.336s)
()
 >> Built dce-ns3-1.9 - OK

#gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/4.9/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 4.9.3-13ubuntu2' --with-bugurl=file:///usr/share/doc/gcc-4.9/README.Bugs --enable-languages=c,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.9 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.9 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-gnu-unique-object --disable-vtable-verify --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-4.9-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-4.9-amd64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-4.9-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 4.9.3 (Ubuntu 4.9.3-13ubuntu2) 


#g++ -v
Using built-in specs.
COLLECT_GCC=g++
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/4.9/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 4.9.3-13ubuntu2' --with-bugurl=file:///usr/share/doc/gcc-4.9/README.Bugs --enable-languages=c,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.9 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.9 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-gnu-unique-object --disable-vtable-verify --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-4.9-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-4.9-amd64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-4.9-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 4.9.3 (Ubuntu 4.9.3-13ubuntu2) 


#gccxml
GCC-XML version 0.6.0
No arguments given for GCC C++ parser.  Not running it.
Usage:

gccxml [options] <input-file> -fxml=<output-file>

The following options are available for running GCC-XML:
  --copyright              = Print the GCC-XML copyright and exit.
  --debug                  = Print extra debugging information.
  -fxml=<output-file>      = Specify the XML output file.
  -fxml-start=<xxx>[,...]  = Specify a list of starting declarations.
  --gccxml-compiler <xxx>  = Set GCCXML_COMPILER to "xxx".
  --gccxml-cxxflags <xxx>  = Set GCCXML_CXXFLAGS to "xxx".
  --gccxml-executable <xxx>= Set GCCXML_EXECUTABLE to "xxx".
  --gccxml-cpp <xxx>       = Set GCCXML_CPP to "xxx".
  --gccxml-config <xxx>    = Set GCCXML_CONFIG to "xxx".
  --gccxml-root <xxx>      = Set GCCXML_ROOT to "xxx".
  --help                   = Print full help and exit.
  --help-html              = Print full help in HTML format.
  --man                    = Print a UNIX man page and exit.
  --print                  = Print configuration settings and exit.
  --preprocess             = Preprocess the input and exit.
  -E                       = Alias for --preprocess.
  --version                = Show program name/version banner and exit.

Other flags, such as -I and -D, are passed on to the patched GCC C++
parser executable.

[SPYFF]

@mickeyze I have exactly the same issue.

[teto]

Reopening as per @tomhenderson 's mail. This occurs on Ubuntu 18.04.

DCE works on Fedora 26:
glibc-2.25-13.fc26.x86_64

But on Ubuntu 18.04:
2.27-3ubuntu1

[richi235]

Can confirm. I have the same issue with the current ns-3-dce-dev on:

Debian testing (buster) with glibc 2.28-5

Stack trace looks similar as in opening post by @teto :

1   __GI_raise                             raise.c                  50   0x7ffff626685b 
2   __GI_abort                             abort.c                  79   0x7ffff6251535 
3   __libc_message                         libc_fatal.c             181  0x7ffff62a8728 
4   __GI___libc_fatal                      libc_fatal.c             191  0x7ffff62a8752 
5   _IO_vtable_check                       vtables.c                72   0x7ffff62a8fb7 
6   IO_validate_vtable                     libioP.h                 838  0x7ffff62a0db1 
7   _IO_seekoff_unlocked                   ioseekoff.c              59   0x7ffff62a0db1 
8   __GI_fseek                             fseek.c                  36   0x7ffff62a6f29 
9   dce_fseek                              dce-stdio.cc             581  0x7ffff7e8811b 
10  dce_fdopen                             dce-stdio.cc             209  0x7ffff7e83c8e 
11  dce_global_variables_setup             dce-global-variables.cc  22   0x7ffff7ea5208 
12  setup_global_variables                 libc-setup.cc            37   0x7ffff372c53d 
13  libc_setup                             libc.cc                  225  0x7ffff372c49b 
14  ns3::DceManager::LoadMain              dce-manager.cc           1227 0x7ffff7e292a4 
15  ns3::DceManager::PrepareDoStartProcess dce-manager.cc           267  0x7ffff7e22b94 
16  ns3::DceManager::DoStartProcess        dce-manager.cc           292  0x7ffff7e23074 
17  ns3::TaskManager::Trampoline           task-manager.cc          275  0x7ffff7eb28d0 
18  ns3::PthreadFiberManager::Run          pthread-fiber-manager.cc 402  0x7ffff7eae6df 
19  start_thread                           pthread_create.c         486  0x7ffff63f7fa3 
20  clone                                  clone.S                  95   0x7ffff63287ef 

[pablogil16]

I also got the same error on Ubuntu 18.04.

Is there any workaround for this?

[murillo128]

same issue here:

(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x00007fffef904801 in __GI_abort () at abort.c:79
#2  0x00007fffef94d5e5 in __libc_message (action=(do_abort | do_backtrace), fmt=0x7fffefa7a796 "%s", fmt=0x7fffefa7a796 "%s", action=(do_abort | do_backtrace))
    at ../sysdeps/posix/libc_fatal.c:181
#3  0x00007fffef94d92a in __GI___libc_fatal (message=message@entry=0x7fffefa7c3b8 "Fatal error: glibc detected an invalid stdio handle\n") at ../sysdeps/posix/libc_fatal.c:191
#4  0x00007fffef94e1d7 in _IO_vtable_check () at vtables.c:72
#5  0x00007fffef944cb1 in IO_validate_vtable (vtable=0x7ffff7dd29a0 <dce_fdopen::vtable>) at ../libio/libioP.h:876
#6  _IO_seekoff_unlocked (fp=0x5555557c4480, offset=-1, dir=<optimized out>, mode=<optimized out>) at ioseekoff.c:59
#7  0x00007fffef94bcd9 in __GI_fseek (fp=0x5555557c4480, offset=<optimized out>, whence=<optimized out>) at fseek.c:36
#8  0x00007ffff7abcd8f in dce_fseek (stream=0x5555557c4480, offset=-1, whence=0) at ../model/dce-stdio.cc:580
#9  0x00007ffff7ab8553 in dce_fdopen (fildes=0, mode=0x7ffff7b6b97d "r") at ../model/dce-stdio.cc:208
#10 0x00007ffff7adbb9d in dce_global_variables_setup (variables=0x7ffff7f76b70) at ../model/dce-global-variables.cc:22
#11 0x00007fffed114793 in setup_global_variables () at ../model/libc-setup.cc:37
#12 0x00007fffed1146e2 in libc_setup (fn=0x5555557f6300) at ../model/libc.cc:225
#13 0x00007ffff7a57fc9 in ns3::DceManager::LoadMain (ld=0x555555869050, filename="/usr/local/src/dce/build/bin/iperf", proc=0x555555838f60, err=@0x7ffff7f76cf4: 0)
    at ../model/dce-manager.cc:1227
#14 0x00007ffff7a513d1 in ns3::DceManager::PrepareDoStartProcess (current=0x5555557963c0) at ../model/dce-manager.cc:267
#15 0x00007ffff7a518ee in ns3::DceManager::DoStartProcess (context=0x5555557963c0) at ../model/dce-manager.cc:292
#16 0x00007ffff7ae9bc4 in ns3::TaskManager::Trampoline (context=0x5555557f3ad0) at ../model/task-manager.cc:275
#17 0x00007ffff7ae2cef in ns3::UcontextFiberManager::Trampoline (a0=32767, a1=-139551872, a2=21845, a3=1434401488) at ../model/ucontext-fiber-manager.cc:199
#18 0x00007fffef91c6b0 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#19 0x0000000000000000 in ?? ()

[murillo128]

Not sure if useful but according to the libio: Implement vtable verification patch https://patchwork.ozlabs.org/patch/633012/

To enable backwards compatibility, a special flag variable
(_IO_accept_foreign_vtables), protected by the pointer guard, avoids
process termination if libio stream object constructor functions have
been called earlier.  Such constructor functions are called by the GCC
2.95 libstdc++ library, and this mechanism ensures compatibility with
old binaries.  Existing callers inside glibc of these functions are
adjusted to call the original functions, not the wrappers which enable
vtable compatiblity.

[murillo128]

Related: https://dhavalkapil.com/blogs/FILE-Structure-Exploitation/

[tomhenderson]

Please see #110; if no one has any recent updates on the above threads, we may close this issue and pursue it in #110 (where there is a detailed report regarding Ubuntu 20.04).

[glance-]

I'll written some code to get around this, tnat uses fopencookie instead of messing with the internals of FILE pointers. I'll try to get it published.

[tomhenderson]

I suggested to Parth (who is working on this) that we keep this issue open for the general issue of newer glibc's failing with this issue, and a similar issue #110 renamed to focus on Ubuntu 20.04-specific concerns.

I suggested to Parth that we should explore three general solutions to this vtable verification limitation:

1. find out if there is some way to bypass; perhaps asking the glibc maintainers about it. Or, perhaps some ideas earlier in this thread posted by others are possible
2. have bake build a custom glibc and make DCE use that. Parth has been experimenting with this for Ubuntu 20.04 and glibc-2.25. It seems promising, if a bit heavyweight.
3. explore whether another libc such as musl could be used instead of glibc.

[tomhenderson]

I'll written some code to get around this, tnat uses fopencookie instead of messing with the internals of FILE pointers. I'll try to get it published.

@glance- are you able to publish your code? Florian Westphal was recommending this direction to us on the libc-alpha mailing list today.

[tomhenderson]

Please see Parth's recent post about his attempts to use fopencookie (hopefully there will be follow-up here or on libc-alpha):

https://sourceware.org/pipermail/libc-alpha/2021-July/129432.html

[glance-]

I've how dropped my code in a PR. There's some TODO's / FIXME's left but its working good enough for our use case, and I haven't gotten around to fixing all those and cleaning them up enough to get all the tests running as they should, but works.