This allows users who would like to use python -O to do this if they use TOKEN decorators instead of docstrings.
When using @TOKEN decorator, use another attribute than __doc__ to st…
I think it's an elegant solution to enter the lexer regexes in the docstrings of function, but as stated in the documentation, this has some drawbacks. When using the TOKEN decorator, there should be no reason to set doc over any other atribute (such as e.g. 'regex') on the function. I'll attach a patch to show how it might be solved differently. Setting another attribute wil allow users of ply (me :) to write a lexer that works with "python -O", if they want to.
Thanks for a very nice framework, I'll let you know if I make something cool with it :)
I'm a little curious about the behavior you're seeing with python -O without this patch. In order to use 'python -O' with ply, it's usually always been necessary to first run the code normally (without -O) to get the resulting table file. Then you run with '-O' and it just works. Does @token crash altogether in this case?
I don't see any chrashes. This patch is more of an enhancment. @token without my patch just sets the doc attribute of the function, which is removed with -O. My patch uses another attribute, namely "regex". This means that if you write a lexer without using docstrings, but only @token, it will work with -O the first run.
As a fairly long time PLY user I have always felt storing the regex's (and the grammar for that matter) in the __doc__ string is a little hacky. I understand why you did it that way, but I would welcome Kristoffer's solution as being part of the main tree.
Just as a note: My patch is backwards compatible. It should not break existing uses of the library.
Incorporated into PLY-3.5. I have some ideas for expansion of this idea in later versions.