textparser.ParseError From SYM file Mux line.

[guy-radford-sunswap]

When loading a SYM file that uses MUXes, where the Mux value is a hex string starting with an [a-f], the parser thinks it is of type WORD, not HEXNUMBER. Hex values starting with [0-9] parse fine.

FormatVersion=6.0 // Do not edit this line!
Title="Untitled"

{SENDRECEIVE}

[MuxedMessageExample]
ID=123h
Len=8
Mux=Multiplexer_09 0,8 9 // Multiplexer with value 0x09
Var=DataVariable unsigned 8,8


[MuxedMessageExample]
Len=8
Mux=Multiplexer_A0 0,8 A0h // Multiplexer with value 0xA0
Var=DataVariable unsigned 8,8

The above SYM file will fail when loaded using:

import cantools

db = cantools.db.load_file('DBC_generation_failure_example.sym')

Returns the following error:

Traceback (most recent call last):
  File "/Users/guyradford/PycharmProjects/CAN-Symbols/venv/lib/python3.9/site-packages/textparser.py", line 887, in parse
    return Grammar(self.grammar()).parse(tokens, token_tree)
  File "/Users/guyradford/PycharmProjects/CAN-Symbols/venv/lib/python3.9/site-packages/textparser.py", line 684, in parse
    raise GrammarError(tokens.peek_max().offset)
textparser.GrammarError: Invalid syntax at offset 252.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/guyradford/PycharmProjects/CAN-Symbols/temp/export_to_dbc.py", line 5, in <module>
    db = cantools.db.load_file('DBC_generation_failure_example.sym')
  File "/Users/guyradford/PycharmProjects/CAN-Symbols/venv/lib/python3.9/site-packages/cantools/database/__init__.py", line 193, in load_file
    return load(fin,
  File "/Users/guyradford/PycharmProjects/CAN-Symbols/venv/lib/python3.9/site-packages/cantools/database/__init__.py", line 287, in load
    return load_string(fp.read(),
  File "/Users/guyradford/PycharmProjects/CAN-Symbols/venv/lib/python3.9/site-packages/cantools/database/__init__.py", line 384, in load_string
    return load_can_database('sym')
  File "/Users/guyradford/PycharmProjects/CAN-Symbols/venv/lib/python3.9/site-packages/cantools/database/__init__.py", line 357, in load_can_database
    db.add_sym_string(string)
  File "/Users/guyradford/PycharmProjects/CAN-Symbols/venv/lib/python3.9/site-packages/cantools/database/can/database.py", line 292, in add_sym_string
    database = sym.load_string(string, self._strict, sort_signals=self._sort_signals)
  File "/Users/guyradford/PycharmProjects/CAN-Symbols/venv/lib/python3.9/site-packages/cantools/database/can/formats/sym.py", line 994, in load_string
    tokens = Parser60().parse(string)
  File "/Users/guyradford/PycharmProjects/CAN-Symbols/venv/lib/python3.9/site-packages/textparser.py", line 889, in parse
    raise ParseError(text, e.offset)
textparser.ParseError: Invalid syntax at line 15, column 24: "Mux=Multiplexer_A0 0,8 >>!<<A0h // Multiplexer with value 0xA0"

The error is generated when tokenizing the A0h:

Mux=Multiplexer_A0 0,8 A0h // Multiplexer with value 0xA0
                       ^^^

The Parser is returning type WORD, not HEXNUMBER. If the value is changed to 0A0h the value is parsed successfully and the type is set to HEXNUMBER

I think the issue is with the regex:

            ('HEXNUMBER',          r'-?\d+\.?[0-9A-F]*([eE][+-]?\d+)?(h)'),

Unfortunately, I am not a regex expert and I could not locate any tests for the regex, otherwise, I would have attempted a PR. Hopefully, I have provided enougg context to get this fixed.

Thank you

[guy-radford-sunswap]

Ran some tests using https://regex101.com/
As you can see not all the values are highlighted in blue as would be expected.

[swaldhoer]

This is basically the same problem as #458.