Parser inheritance

[ponyatov]

I'm trying to use multiple parsers in one program, encapsulated into classes:
https://github.com/ponyatov/DLR/blob/PLY_issue142/inher.py

VM lexer class works ok in both ways: as described in PLY manual, and with my init (commented).

But when I try to use inherited FORTH class (redefine newline rule for example) I got an error:

ERROR: inher.py:22: Rule t_ignore redefined. Previously defined on line 6
SyntaxError: Can't build lexer

Is any way exists to use inheritance?

Lexer defined using closures in compiler() method works fine until I try to use static class member in inherited FORTH:
https://github.com/ponyatov/DLR/blob/PLY_issue142/VM.py

[dabeaz]

Defining parsers as classes was never the primary mechanism of creating a parser in PLY. I've never written any projects that use parser inheritance nor are they any tests for it. It wouldn't surprise if it didn't work at all.

The error above is being caused by two parsers being defined in the same file. It might work if you split things into multiple files. There still might be issues with any kind of inheritance working though.

[ponyatov]

It looks like problem linked with LexerReflect.validate_module()
in case of multiple t_error() functions defined in multiple classes in one file, as you noted.

I've done dirt hack: add .isobject flag =True if module was redefined by lex(object!=Null) parameter. But the problem is deeper: t_error uniqueness should be checked not on module level, but on the level of an outer construct, where t_error was defined:

• module if t_error defined as free function in a module (as it is done now)
• class if t_error defined as method
• function t_error defined as closure

class LexerReflect(object): ##
    def validate_rules(self): ##
            efunc = self.errorf.get(state, None) ##
            if efunc: ##
                f = efunc ##
                module = inspect.getmodule(f) ## should be .getouter()
                self.modules.add(module) ##
        for module in self.modules: ##
            self.validate_module(module) ##
    def validate_module(self, module): ##
    	if self.isobject: return ##+ don't check redefinition for lexer in class
def lex(module=None, object=None, debug=False, optimize=False, lextab='lextab', ##
        reflags=int(re.VERBOSE), nowarn=False, outputdir=None, debuglog=None, errorlog=None): ##
    lexobj = Lexer() ##
    # Get the module dictionary used for the lexer ##
    if object: ##
        module = object ##
        lexobj.isobject = True ##+
    else: ##+
    	lexobj.isobject = False ##+
    linfo = LexerReflect(ldict, log=errorlog, reflags=reflags) ##
    linfo.isobject = lexobj.isobject ##+

[ponyatov]

A more elegant solution doesn't break anything except test: only two line patch making a warning in case of multiple t_error definitions:

class LexerReflect(object):
    def validate_module(self, module):
        self.log.warning('%s:%d: Rule %s redefined. Previously defined on line %d', filename, linen, name, prev)
        # self.error = True

More simple variant can be used: check special ply_class_inher flag exists in module, and exit from validate_module()

[johnyf]

I have used ply to write a number of parsers as classes (the boilerplate can be found in the package astutils, and an example with inheritance is at: https://github.com/johnyf/openpromela/blob/8a74b37b2b481cb9b1914fd9b2b4b38bb8162a73/openpromela/logic.py#L60). I have not encountered issues with inheritance, but I think in all my use cases the parent class was from a different module.

[ponyatov]

Some problem with disabling lexer rules then token list changed in inherited lexer:

class FORTH(VM):
	t_ignore_COMMENT = r'\#.*|\\.*|\(.*?\)'		# comment
	tokens = ['EQ','STRING',\
			'COLON','SEMICOLON','BEGIN','AGAIN',	# : ;
			'ID']
	t_NOP = None ; t_BYE = None
	p_command_NOP = None ; p_command_BYE = None

need some trick to remove elements from inherited parser

[nikitagupta55]

@ponyatov @johnyf please help in resolving this issue : #149

[dabeaz]

I'm closing this without changing behavior. Mainly this is because PLY is really in a maintenance-only mode and I'm afraid to add features that extend it's functionality in new directions--especially with regards to inheritance (which might have far-reaching consequences).