Ways to define tokens seems to have different effects.

[petermlm]

Suppose you have the following tokens defined:

tokens = (
    'THIS',
    'ANYTHING'
)

t_THIS = r'this'
t_ANYTHING = r'[a-z]+'

Any string is tokenized has an "anything" token. At first I thought this was what should be expected, but then I tried doing the following.

tokens = (
    'THIS',
    'ANYTHING'
)

def t_THIS(t):
    r'this'
    return t

def t_ANYTHING(t):
    r'[a-z]+'
    return t

Like this, any "this" string gets tokenized as a "this" token because that definition comes first, while anything else gets tokenized as "anything".

If I switch the two rules around, then nothing gets tokenized as a "this" token again, which seems to make sense and also is what happens when I do the same thing with flex.

Is this what is supposed to be expected and I am missing something, or is it a bug? Even if it not really a bug, should it be working like this?

[helpermethod]

This works as expected and is covered in the documentation

(i.e., functions are matched in order of specification whereas strings are sorted by regular expression length).

This is why in your first example this is matched by t_ANYTHING (its regular is expression is longer than that of t_THIS.

In the second example t_THIS(t) comes before t_ANYTHING(t), so matches first.