Japanese characters support

[khanhhua]

I am writing a Japanese program (simplied as follows):

sov([S,SMarker,O,V]) :- # watashi ha gakusei desu.
  subject(S, SMarker),
  noun(O),
  verb(V).


subject(N, "は") :-
  pronoun(N).

pronoun("私").
pronoun("あなた").

noun("学生").
verb("です").

The output when I do a query ?sov prints out unicode in my iterm2

mg >?sov
sov(["\u{003042}\u{00306a}\u{00305f}", "\u{00306f}", "\u{005b66}\u{00751f}", "\u{003067}\u{003059}"])
sov(["\u{0079c1}", "\u{00306f}", "\u{005b66}\u{00751f}", "\u{003067}\u{003059}"])
Found 2 entries for sov(X0).

... even when I tried the LANG and LC_ALL trick

bash-5.3$ locale
LANG="ja_JP.UTF-8"
LC_COLLATE="C"
LC_CTYPE="C"
LC_MESSAGES="C"
LC_MONETARY="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_ALL="C"

and run go run interpreter/mg/mg.go under the new env variables.

Is this an issue of limited charset support at the current version of mangle or have I setup something wrong?

[burakemir]

This is due to the way the interpreter is implemented. It is a shortcoming, I filed #61 to address your problem.

The interpreter uses the String() method to format a Mangle "constant" (value):

mangle/interpreter/interpreter.go

Line 244 in 081bd25

results = append(results, fact.String())

The String() method is implemented in a way that it will always escape characters (not the Escape call here):

mangle/ast/ast.go

Line 512 in 081bd25

s, err := Escape(c.Symbol, false /* isBytes */)

The interpreter has many shortcomings since it was only added for quick experiments.
This particular choice was done because before there were byte strings, people used strings for binary data and escaping was useful.

The easiest fix to support proper unicode output would be if the interpreter supported an output mode that does not use the ast.Constant String() method, but output strings directly.