how to support unicode/utf-8? like Chinese?

[iamsk]

Sample code like this:

parser = Lark('''start: WORD "," WORD "!"
            %import common.WORD   // imports from terminal library
            %ignore " "           // Disregard spaces in text
         ''', parser='lalr')
print(parser.parse("Hello,世界!"))

I'm already try a long time, anyone can help on this, thanks!

[ray-linn]

the letter of word is defined as "A-Z" or "a-z"

[erezsh]

Right now, WORD is defined for English only. Perhaps that needs to change.
Meanwhile, you can use the following definition:

WORD: /[^\W\d_]+/

[ray-linn]

I could extract Unicode CHS letters by /u"[\u4e00-\u9fa5]"/. however, looks this rule does not work

LCASE_LETTER: "a".."z"
UCASE_LETTER: "A".."Z"
SIMCH_LETTER: /u"[\u4e00-\u9fa5]"/
LETTER: UCASE_LETTER | LCASE_LETTER | SIMCH_LETTER
WORD: LETTER+

I think something in load_grammer.py may need to be changed

[iamsk]

@ray-linn ok, thanks, I think this will help.

[ray-linn]

I find out how to code the rule for Unicode in grammer.g, here is the example

LCASE_LETTER: "a".."z"
UCASE_LETTER: "A".."Z"
CN_ZH_LETTER: /[u"\u4e00-\u9fa5"]/
LETTER: UCASE_LETTER | LCASE_LETTER | CN_ZH_LETTER
WORD: LETTER+

and it outputs:

Tree(start, [Token(WORD, 'Hello'), Token(WORD, '世界')])

[ruiqurm]

I find out how to code the rule for Unicode in grammer.g, here is the example

LCASE_LETTER: "a".."z"
UCASE_LETTER: "A".."Z"
CN_ZH_LETTER: /[u"\u4e00-\u9fa5"]/
LETTER: UCASE_LETTER | LCASE_LETTER | CN_ZH_LETTER
WORD: LETTER+

and it outputs:

Tree(start, [Token(WORD, 'Hello'), Token(WORD, '世界')])

/[u"\u4e00-\u9fa5"]/ will include the quote marks. You can use /[\u4e00-\u9fa5]/ instead