Syntax: support for extended characters

[UWN]

In 6.5 the processor character set (PCS) is defined. It not only includes the characters defined by char (6.5) but also may include additional members, known as extended characters. Note that this is not an extension in the sense of 5.5.1, but is part of the actual implementation defined character set. The standard also mentions examples in NOTE 2:

... Examples of extended small letter char (6.5.2) are small letters with grave or acute accent and Japanese Kanji characters.

Now, certainly, an implementation does not have to support additional characters at all, but it seems to make a lot of sense in the times of UTF-8.

So I would expect that

?- 改善.
2022/04/29 09:27:57 failed to query: unexpected token: <invalid 改> % unexpected

is rather treated like

?- '改善'.
2022/04/29 09:28:03 error(existence_error(procedure, '改善'/0), 'Unknown procedure.') % expected

which also many existing implementations do.

[ichiban]

Since v.0.10.0, unicode.Unified_Ideograph, unicode.Hiragana, and unicode.Katakana are added for extended small letter char.

$ $(go env GOPATH)/bin/1pl
Top level for ichiban/prolog v0.10.0
This is for testing purposes only!
See https://github.com/ichiban/prolog for more details.
Type Ctrl-C or 'halt.' to exit.
?- 改善.
2022/07/16 13:43:08 error(existence_error(procedure,改善/0),root)
?- プロログ.
2022/07/16 13:43:15 error(existence_error(procedure,プロログ/0),root)
?- はい.
2022/07/16 13:44:34 error(existence_error(procedure,はい/0),root)

[UWN]

For Korean letters I get:

?- char_code(Ch, 0x314b).
Ch = 'ㅋ', unexpected. % Expected without quotes
?- Ch = 'ㅋ'.
2022/07/18 08:02:44 failed to query: unexpected token: {invalid 'ㅋ}
?- X='\x628\'.
X = 'ب', unexpected. % Expected without quotes
?- X='\x5d0\'.
X = 'א', unexpected. % Expected without quotes

(Warning, I am really not an expert in Unicode at all). It seems that you can treat all

[UWN]

... It seems that you can treat all letters that are qualified as Lo just as Ll, that is as a lower case letter.

[guregu]

Seems like v0.10.0 broke something with ー (長音符)

?- write('あー').
2022/08/01 05:22:09 failed to query: unexpected token: {invalid 'あー}

Works in v0.9.1.

[ichiban]

Fixed in v0.10.4!

$ $(go env GOPATH)/bin/1pl
Top level for ichiban/prolog v0.10.4
This is for testing purposes only!
See https://github.com/ichiban/prolog for more details.
Type Ctrl-C or 'halt.' to exit.
?- write('あー').
あーtrue.
?- 

[UWN]

Even the following works:

?-  writeq('𱍊').
𱍊true.