[Bug] State minimizer ignores EOF actions [sf#80]

[lsf37]

Reported by arhawth on 2006-08-14 21:27 UTC
If two states only differ by the action for the EOF
rule, they are minimized to the same state, and JFlex
emits a warning:

Warning : Lexical states <STATE1> and <STATE2> are
equivalent.

Here is a sample scanner definition that emits the warning:

%%
%{
public static final int WORD = 1;
public static final int WS = 2;
public static final int EOF1 = 3;
public static final int EOF2 = 4;
%}

%unicode

%xstate STATE2

WORD=[a-zA-Z]+
WS=[ \t\n\r]+

%%

<YYINITIAL,STATE2>
{
{WORD}
{
yybegin(STATE2);
return WORD;
}

{WS}
{
yybegin(YYINITIAL);
return WS;
}
}

<YYINITIAL>
{
<<EOF>>
{
return EOF1;
}
}

<STATE2>
{
<<EOF>>
{
return EOF2;
}
}

Here is one where the states are the same except for
the actions in a non-EOF rule. The output does not
emit the warning:

%%
%{
public static final int WORD1 = 1;
public static final int WORD2 = 2;
public static final int WS = 3;
public static final int EOF = 4;
%}

%unicode

%xstate STATE2

WORD=[a-zA-Z]+
WS=[ \t\n\r]+

%%

<YYINITIAL,STATE2>
{
{WS}
{
return WS;
}

<<EOF>>
{
return EOF;
}
}

<YYINITIAL>
{
{WORD}
{
yybegin(STATE2);
return WORD1;
}
}

<STATE2>
{
{WORD}
{
yybegin(YYINITIAL);
return WORD2;
}
}

[lsf37]

Commented by lsf37 on 2007-11-03 11:07 UTC
Logged In: YES
user_id=93534
Originator: NO

This is an interesting one. The bug is actually not really in the minimisation, but in the code emitter (which makes the wrong assumption that was mentioned: just because they point to the same DFA states, two lex states are not necessarily the same, because they can be distinguished by EOF actions).

Fixed in svn revision 267.

[lsf37]

Updated by lsf37 on 2007-11-03 11:07 UTC

• milestone: --> jflex bug
• status: open --> open-fixed

[lsf37]

Updated by lsf37 on 2008-05-27 11:46 UTC

• status: open-fixed --> closed

[lsf37]

Commented by lsf37 on 2007-11-03 11:07 UTC
Logged In: YES
user_id=93534
Originator: NO

This is an interesting one. The bug is actually not really in the minimisation, but in the code emitter (which makes the wrong assumption that was mentioned: just because they point to the same DFA states, two lex states are not necessarily the same, because they can be distinguished by EOF actions).

Fixed in svn revision 267.

[lsf37]

Updated by lsf37 on 2007-11-03 11:07 UTC

• milestone: --> jflex bug
• status: open --> open-fixed

[lsf37]

Updated by lsf37 on 2008-05-27 11:46 UTC

• status: open-fixed --> closed