cxx_restart test segfaults with flex 2.6.2

[heirecka]

Backtrace and log file (github didn't let me attach it) follow below:

(gdb) bt
#0  testFlexLexer::yy_init_buffer (this=this@entry=0x7fffc91999a0, b=0xca5c20, file=...) at cxx_restart.cc:1319
#1  0x00000000004029e1 in testFlexLexer::yyrestart (this=this@entry=0x7fffc91999a0, input_file=...) at cxx_restart.cc:1191
#2  0x00000000004011b8 in testFlexLexer::yyrestart (input_file=0x0, this=0x7fffc91999a0) at cxx_restart.cc:1202
#3  main () at cxx_restart.ll:48

#!/bin/bash -vx
set -euo pipefail
+ set -euo pipefail

# testwrapper.sh: run a flex test, typically called by a Makefile

# Each test will exercise some feature or aspect of flex. Run the test with any input it may need.

INPUT_DIRECTORY=""
+ INPUT_DIRECTORY=
INPUT_NAME=""
+ INPUT_NAME=
INPUT_COUNT=0
+ INPUT_COUNT=0
USE_REDIRECT=0
+ USE_REDIRECT=0
DO_COMPARISON=0
+ DO_COMPARISON=0

while getopts :d:i:rt1 OPTION ; do
    case $OPTION in
        d) INPUT_DIRECTORY=$OPTARG ;;
        i)
            if [ "$INPUT_NAME" == "" ] ; then
                INPUT_NAME="$OPTARG"
            else
                INPUT_NAME="$INPUT_NAME $OPTARG"
            fi
            INPUT_COUNT=$(($INPUT_COUNT+1))
            ;;
        r) USE_REDIRECT=1 ;;
        t) USE_TABLES=1 ;;
        1) DO_COMPARISON=1 ;;
    esac
    done
+ getopts :d:i:rt1 OPTION
+ case $OPTION in
+ INPUT_DIRECTORY=.
+ getopts :d:i:rt1 OPTION
+ case $OPTION in
+ USE_REDIRECT=1
+ getopts :d:i:rt1 OPTION

TESTNAME="${!OPTIND}"
+ TESTNAME=./cxx_restart

INPUT_NAME=${INPUT_NAME:-$INPUT_DIRECTORY/`basename ${TESTNAME%.exe}`.txt}
++ basename ./cxx_restart
+ INPUT_NAME=./cxx_restart.txt

if [ "$DO_COMPARISON" -eq "1" ] ; then
    test `$TESTNAME 1 < $INPUT_NAME` -eq `$TESTNAME < $INPUT_NAME`
    exit $?
    fi
+ '[' 0 -eq 1 ']'

if [ $INPUT_COUNT -gt 1 ] ; then
    $TESTNAME ${USE_TABLES:+${INPUT_DIRECTORY}/${TESTNAME%.exe}.tables} ${INPUT_NAME}
    exit $?
    fi
+ '[' 0 -gt 1 ']'

if [ -f ${INPUT_NAME} ] ; then
    if [ $USE_REDIRECT == 1 ] ; then
        $TESTNAME ${USE_TABLES:+${INPUT_DIRECTORY}/${TESTNAME%.exe}.tables} < $INPUT_NAME
    else
        $TESTNAME ${USE_TABLES:+${INPUT_DIRECTORY}/${TESTNAME%.exe}.tables} $INPUT_NAME
    fi
else
    $TESTNAME
fi
+ '[' -f ./cxx_restart.txt ']'
+ '[' 1 == 1 ']'
+ ./cxx_restart
./testwrapper.sh: line 53: 22774 Speicherzugriffsfehler  (core dumped) $TESTNAME ${USE_TABLES:+${INPUT_DIRECTORY}/${TESTNAME%.exe}.tables} < $INPUT_NAME
FAIL cxx_restart (exit status: 139)

[westes]

I'm not seeing that, so...

What's your platform and compiler? (And clearly that info should be a part of the log that gets generated or something like that.)

[heirecka]

What's your platform and compiler? (And clearly that info should be a part of the log that gets generated or something like that.)

Exherbo Linux (x86_64-pc-linux-gnu), gcc-6.2.0

After writing the above I just tried with gcc-5.4.0 and the test passes when compiled with that gcc version.

[westes]

with gcc6, does it matter which c++ variant you use? It defaults to c++14, iirc.

[atsampson]

I'm seeing this with GCC 6.2.0 on AMD64 as well. The line it crashes on (from flex.skl:2059) makes it pretty clear what the problem is:

b->yy_input_file = (&file == 0) ? NULL : file.rdbuf();          

(where file is a std::istream&). That condition can't ever be true in a legal C++ program -- the only way you could create a reference to NULL is to dereference a null pointer, which has been undefined behaviour at least since C++98. GCC 6 knows this, and optimises out the test entirely, which is why it crashes here.

Where's the code expecting the invalid reference to come from? The test calls f.yyrestart(NULL);, and yyrestart immediately dereferences the pointer it's given, invoking undefined behaviour. This looks like a case where flex really ought to be using pointers internally rather than references (as indeed it did prior to 336a1de -- I guess the author of that patch didn't consider the possibility of yyrestart(NULL)).

[heirecka]

with gcc6, does it matter which c++ variant you use? It defaults to c++14, iirc.

(where file is a std::istream&). That condition can't ever be true in a legal C++ program -- the only way you could create a reference to NULL is to dereference a null pointer, which has been undefined behaviour at least since C++98. GCC 6 knows this, and optimises out the test entirely, which is why it crashes here.

Indeed, the C++ variant has no influence.

[DemiMarie]

Can we provide a dummy stream that is used when yyrestart(NULL) is called? A default-constructed one should do.

[tpgxyz]

Hi,
i got this error but with LLVM/clang 4.0.0 with flex 2.6.3

• ./cxx_restart
  ./testwrapper.sh: line 53: 10444 Segmentation fault (core dumped) $TESTNAME ${USE_TABLES:+${INPUT_DIRECTORY}/${TESTNAME%.exe}.tables} < $INPUT_NAME
  FAIL cxx_restart (exit status: 139)

More logs can be found here:

http://file-store.openmandriva.org/api/v1/file_stores/58cf12d885222d0edf3ce8b10c10bc5f5ed6342f.log?show=true

[hvdijk]

My apologies for the commit spam, I hadn't realised that they'd show up here even before I had created the pull request. Anyway, creating a new dummy stream seems difficult due to lifetime management, but using the existing yyin stream seems simple enough, since yyin won't be used anyway.

[westes]

@hvdijk no worries at all. I'm looking forward to reviewing your pr, so thanks for that.

[davidnich]

flex 2.6.3 make check fails (and produces a segfault with my scanner as well) with the following compilers:

• Fedora 25 x86_64 clang++ 3.9.1:

clang version 3.9.1 (tags/RELEASE_391/final)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

• Darwin MacOS 10.12.4 Sierra clang++

Apple LLVM version 8.1.0 (clang-802.0.41)
Target: x86_64-apple-darwin16.5.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

make check fails in the cxx_restart test

Therefore it seems that this issue is not resolved.

Error in cxx_restart.log:

+ '[' -f ./cxx_restart.txt ']'
+ '[' 1 == 1 ']'
+ ./cxx_restart
./testwrapper.sh: line 53: 15749 Segmentation fault      (core dumped) $TESTNAME ${USE_TABLES:+${INPUT_DIRECTORY}/${TESTNAME%.exe}.tables} < $INPUT_NAME
FAIL cxx_restart (exit status: 139)

from gdb:

(gdb) run < cxx_restart.txt
Starting program: /export/home/dnichols/src/flex-2.6.3/tests/cxx_restart < cxx_restart.txt
Missing separate debuginfos, use: dnf debuginfo-install glibc-2.24-4.fc25.x86_64

Program received signal SIGSEGV, Segmentation fault.
testFlexLexer::yy_init_buffer (this=0x7fffffffd250, b=0x617c20, file=...) at cxx_restart.cc:1325
1325	
Missing separate debuginfos, use: dnf debuginfo-install libgcc-6.3.1-1.fc25.x86_64 libstdc++-6.3.1-1.fc25.x86_64
(gdb) bt
#0  testFlexLexer::yy_init_buffer (this=0x7fffffffd250, b=0x617c20, file=...) at cxx_restart.cc:1325
#1  0x000000000040297d in testFlexLexer::yyrestart (this=0x7fffffffd250, input_file=...) at cxx_restart.cc:1197
#2  0x0000000000403088 in testFlexLexer::yyrestart (this=0x7fffffffd290, input_file=<optimized out>) at cxx_restart.cc:1208
#3  main () at cxx_restart.ll:48

from valgrind:

==17675== Invalid read of size 8
==17675==    at 0x402AAE: testFlexLexer::yy_init_buffer(yy_buffer_state*, std::istream&) (cxx_restart.cc:1325)
==17675==    by 0x40297C: testFlexLexer::yyrestart(std::istream&) (cxx_restart.cc:1197)
==17675==    by 0x403087: yyrestart (cxx_restart.cc:1208)
==17675==    by 0x403087: main (cxx_restart.ll:48)
==17675==  Address 0x0 is not stack'd, malloc'd or (recently) free'd

[hvdijk]

@davidnich The title just means this was reported against flex 2.6.2, it doesn't mean it got fixed in 2.6.3. It can't: 2.6.3 was already released on 30 Dec 2016, this bug was fixed 12 Jan 2017. It should make it into the next release.

[davidnich]

@hvdijk ok thank you for that information, good to hear!

[ryandesign]

Sounds like it's high time for a new release then.