Parser can't differentiate between ReplaceAll and Division (Times [ , Power])

[weakit]

Take a look at this example.

Mathics:

In[1]:= 1/.2
ReplaceAll::reps: 2 is not a valid replacement rule.
Out[1]= 1 /. 2

Wolfram:

Not really a big problem, but is something that needs to be fixed eventually.

[mmatera]

I have tracked this issue to the tokenizer (mathics/mathics/core/parser/tokenizer.py):

in line 125:
('ReplaceAll', r' \/\. '),
defines the rule for ReplaceAll, that is tested before
('Divide', r' \/|\u00f7 '),
in the method next of the Tokenizer class:
if indices: for index in indices: tag, pattern = self.tokens[index] match = pattern.match(self.code, self.pos) if match is not None: break
I tryed to fix this changing ('ReplaceAll', r' /. '),
by ('ReplaceAll', r' /.![0-9] '),

but for some reason, this produces a failure on the "contribute" routine in the mathics.builtin module. If someone can help me in defining a more sensible regular expression, I am trying to fix this in the branch https://github.com/mathics/Mathics/tree/fix-tokenized

[rocky]

It might just be a simple lexing issue, and adjusting your regexp to lookahead and then backup, but...

Robert Jacobson's FoxySheep parser has the same problem. And although in spirit it is the same, it works very differently.

Is it really just ReplaceAll or are there other functions?

From what I understand about the scanner and parser, I would not attempt to fix this without understanding more about how this all works and the subtleties.

Getting more pure parsing tests in place and a framework for adding test case bugs like this would be useful first too.

To get some idea as to the subtleties involved and how this works see this or the the readme in the FoxySheep repostiory.

[mmatera]

Thanks Rocky, I was aware of that. This was a headake for implementing the Information operator ??. My problem this time was more basic: I was writting wrong the regular expression. I think now it will work...