On Windows (but not on UNIX) redirecting the stdin of main to a pipe or a file results in wrong decoding of non-ASCII characters

[enzomich]

For a small RAG application I have written a Python wrapper that opens Llama.cpp's main into a subprocess using subprocess.Popen() and communicates with it through two pipes (yes, I'm using the --simple-io option). Everything works fine, with an exception: if the line sent to main's stdin contains non-ASCII characters (e.g., Greek or Cyrillic or even just Latin with accents or other diacritical marks) those characters, and only those, are received as garbled text (and understood by the model with a lot of fantasy). Initially I thought that I was doing something wrong, but then I discovered exactly the same thing happens without my Python wrapper, by launching main at the command line and redirecting its stdin using a "main < file.txt" or "echo input_line | main" command:

C:\Users\enzom\AI\LlamaFeeder>echo Translate "Σήμερον ἐστὶν εὔδια ἡμέρα" | \Users\enzom\AI\llama.cpp\llama-b2391-bin-win-cublas-cu12.2.0-x64\main -m \Users\enzom\AI\llama.cpp\Models\mixtral-8x7b-instruct-v0.1.Q4_K_M.gguf --simple-io --instruct --temp 0.1
Log start
main: build = 2391 (7ab7b733)
[...]
 - If you want to submit another line, end your input with '\'.


>  The English translation of "ΣήμεÏον á¼ÏÏὶν εá½Î´Î¹Î± ἡμέÏα" is "The children are playing in the park."

>
>
>  Trans
>
> late
>
>  "
>
> Î
>
> £
> Î

Please also note the garbage in the following lines until main is killed with a Ctrl-C, as if it hadn't noticed that the pipe was closed at the other side.

On the other hand, if the instruction is entered at the console prompt everything works as expected:

C:\Users\enzom\AI\LlamaFeeder>\Users\enzom\AI\llama.cpp\llama-b2391-bin-win-cublas-cu12.2.0-x64\main -m \Users\enzom\AI\llama.cpp\Models\mixtral-8x7b-instruct-v0.1.Q4_K_M.gguf --simple-io --instruct --temp 0.1 
Log start
[...]
 - If you want to submit another line, end your input with '\'.


> Translate "Σήμερον ἐστὶν εὔδια ἡμέρα"
 The translation of "Σήμερον ἐστὶν εὔδια ἡμέρα" is "Today is a fair day."

>

Any idea about how to fix this?

[Green-Sky]

Are you sure the file encoding is utf8 ?

[enzomich]

Yes I am. In the example above the prompt came from a pipe, but here is one where the command line redirects main's stdin to a UTF-8 encoded text file ..\Texts\TranslateGreek.txt containing Translate "Σήμερον ἐστὶν εὔδια ἡμέρα" (meaning "Today is a fair day"). It was prepared with Notepad, which allows to specify the encoding, and Python agrees that the file is indeed UTF-8 encoded:

C:\Users\enzom\AI\LlamaFeeder>Python
Python 3.12.2 (tags/v3.12.2:6abddd9, Feb  6 2024, 21:26:36) [MSC v.1937 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> with open(r'..\Texts\TranslateGreek.txt', 'r', encoding='utf-8') as file:
...   print(file.read())
...
Translate "Σήμερον ἐστὶν εὔδια ἡμέρα"

>>>

However, when main's stdin is redirected to that file, the result is garbage:

C:\Users\enzom\AI\LlamaFeeder>\Users\enzom\AI\llama.cpp\llama-b2391-bin-win-cublas-cu12.2.0-x64\main -m \Users\enzom\AI\llama.cpp\Models\mixtral-8x7b-instruct-v0.1.Q4_K_M.gguf --simple-io --instruct --temp 0.1 < ..\Texts\TranslateGreek.txt
Log start
main: build = 2391 (7ab7b733)
[...]
 - If you want to submit another line, end your input with '\'.


>  The English translation of "ΣήμεÏον á¼ÏÏὶν εá½Î´Î¹Î± ἡμέÏα" is "The children are playing football."

>
>
>  Trans
> late
>  "
> Î
> £
[...] <-- Killed with Ctrl-C

llama_print_timings:        load time =    2992.67 ms
llama_print_timings:      sample time =      10.92 ms /   109 runs   (    0.10 ms per token,  9978.94 tokens per second)
llama_print_timings: prompt eval time =   11862.09 ms /    77 tokens (  154.05 ms per token,     6.49 tokens per second)
llama_print_timings:        eval time =   23289.80 ms /   109 runs   (  213.67 ms per token,     4.68 tokens per second)
llama_print_timings:       total time =   35433.63 ms /   186 tokens

Instead of "Σήμερον ἐστὶν εὔδια ἡμέρα", main reads "ΣήμεÏον á¼ÏÏὶν εá½Î´Î¹Î± ἡμέÏα". And that's what is read by Python opening the file as if it were ISO-8859-1 encoded:

C:\Users\enzom\AI\LlamaFeeder>Python
Python 3.12.2 (tags/v3.12.2:6abddd9, Feb  6 2024, 21:26:36) [MSC v.1937 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> with open(r'..\Texts\TranslateGreek.txt', 'r', encoding='iso-8859-1') as file:
...   print(file.read())
...
Translate "ΣήμεÏον á¼ÏÏὶν εá½Î´Î¹Î± ἡμέÏα"

>>>

[Green-Sky]

@enzomich looks like we only set the output codepage to utf8, try adding

SetConsoleCP(CP_UTF8);

next to this line:
https://github.com/ggerganov/llama.cpp/blob/076b08649ecc3b0e1c0709c2a086a63eddd1bf32/common/console.cpp#L89

I am not sure how it will affect other parts like special char inputs in non-piped scenarios.

[enzomich]

@Green-Sky that fix didn't work, but this one did: I inserted after line 96 (just before the #else):

		if(simple_io) {
				_setmode(_fileno(stdin), _O_U8TEXT);	
		}

From what I understand (but I may be wrong, non being very familiar with Windows) SetConsoleCP(...) affects the console, but _setmode(stdin, ...) affects the stdin file descriptor also when the input is redirected away from the console to a file or a pipe -- like in this case.

So, for what I'm concerned this issue may be considered closed, if the fix is brought to the code.

[misureaudio]

I can confirm that the bug is present when piping to main, and that the code presented by enzomich solves the issue.

[enzomich]

@Green-Sky , as @misureaudio has confirmed both this issue and my fix, is there any chance of raising the status to "confirmed" and, evenctually, have the fix accepted and merged?

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.

[hasaranga]

why is this still not fixed? @ggerganov

[ggerganov]

Please open a PR with the proposed fix and we'll merge it. I don't have Windows environment to test this