v2.6.2 introduces behavior change that is incompatible with previous versions

[nvinson]

With version 2.6.1 if prefix was set, then all affected functions would have their declaration changed and all references in the lexer input file would be changed as well. With v2.6.2 only the function declarations are changed.

For example, set prefix to AA, reference yyget_extra() in the lexer input file, and build the C file using v2.6.1 and all calls in the C source file will be to AAget_extra(). However, build with v2.6.2 and the resultant C file will still have calls to yyget_extra() resulting in a broken source file.

This change in behavior appears to have been introduced with commit ee87e5d.

[Polynomial-C]

Consequences of this change can be searched and found here: https://bugs.gentoo.org/598186

[fabled]

Flex 2.6.2 also does not properly handle doxygen. The flex generated code fails with:

[ 23%] Building CXX object qtools/CMakeFiles/qtools.dir/qregexp.cpp.o
configimpl.l:692:32: error: stray '' in program
[a-z_A-Z][a-z_A-Z0-9][ \t]"+=" { QCString cmd=yytext;
configimpl.l: In function 'int configimplYYlex()':
configimpl.l:692:1: error: expected primary-expression before '<' token
[a-z_A-Z][a-z_A-Z0-9][ \t]"+=" { QCString cmd=yytext;
^
configimpl.l:692:9: error: 'a' was not declared in this scope
[a-z_A-Z][a-z_A-Z0-9][ \t]"+=" { QCString cmd=yytext;
^
configimpl.l:692:10: error: expected ',' before '-' token
[a-z_A-Z][a-z_A-Z0-9][ \t]"+=" { QCString cmd=yytext;
^
configimpl.l:692:10: error: expected identifier before '-' token
configimpl.l: In lambda function:
configimpl.l:692:17: error: expected '{' before '[' token
[a-z_A-Z][a-z_A-Z0-9][ \t]"+=" { QCString cmd=yytext;
^
configimpl.l: In function 'int configimplYYlex()':
configimpl.l:692:20: error: 'z_A' was not declared in this scope
[a-z_A-Z][a-z_A-Z0-9][ \t]"+=" { QCString cmd=yytext;
^~~
configimpl.l:692:24: error: 'Z0' was not declared in this scope
[a-z_A-Z][a-z_A-Z0-9][ \t]"+=" { QCString cmd=yytext;
^~
configimpl.l:692:33: error: 't' was not declared in this scope
[a-z_A-Z][a-z_A-Z0-9][ \t]"+=" { QCString cmd=yytext;
^
configimpl.l: In lambda function:
configimpl.l:692:35: error: expected '{' before '' token
[a-z_A-Z][a-z_A-Z0-9][ \t]*"+=" { QCString cmd=yytext;
^

The code it generated looks like:

                                    }

[a-z_A-Z][a-z_A-Z0-9][ \t]"+=" { QCString cmd=yytext;
cmd=cmd.left(cmd.length()-2).stripWhiteSpace();
ConfigOption *option = config->get(cmd);
if (option==0) // oops not

Basically it did not handle the pattern match at all.

[ghost]

Just to add another me too. This issue breaks Mesa also: https://bugs.freedesktop.org/show_bug.cgi?id=98473

[tgurr]

It also breaks kde-frameworks/kservice: https://bugs.kde.org/show_bug.cgi?id=371885 and the developers comment was "So why not report this to the flex authors?".

[westes]

@DemiMarie, could you have a look at this?

[nvinson]

I just tested head against nftables-0.6 (using Gentoo's nftables-0.6-r4 ebuild) and the issue is still present. Could you please reopen this bug until the issue is fixed?

edit: head in this case is commit d374b87

[westes]

Certainly. @DemiMarie, what do you think?

[DemiMarie]

I think I know what is going on.

Flex versions <= 2.6.1 expand M4 macros in the source file. This allows a bad input file to execute arbitrary shell commands and results in broken debugging info. It also corrupts string literals that happen to contain the relevant strings.

I "fixed" this behavior. This also required changing the testsuite. In retrospect, I should probably have realized that this would break code.

The solution is to add CPP (not M4) macros that handle this. I will make a patch.

[DemiMarie]

If the problem is in section 3, you can use --unsafe-no-m4-sect3-escape as a temporary workaround.

[DemiMarie]

@nvinson Can you see if #123 fixes this?

[nvinson]

@nvinson Can you see if #123 fixes this?

nftables compiled with that change. I'll do more extensive testing later this week.

[Polynomial-C]

@westes Can we please have a new release once this issue has been confirmed to be fixed?

[MarcinWieczorek]

Hi guys, I'm having trouble with flex.
Here's my build log, read the bottom please:
http://sprunge.us/dTKi
Do you think it's related?