Cannot search for Non-Ascii characters (ex. å)

[Whil-]

Hi!

Not entirely sure if the error is in rg.el or not but thought to report it anyhow to see if it's reproducible or not.

On a PC, Windows 10, searching with rg.el for a string including a non-ascii character doesn't give any results back. The exact same search in terminal works fine.

Example search query:

C:\Some\Path> "rg.exe" --color=always --colors=match:fg:red --colors=path:fg:magenta --colors=line:fg:green --colors=column:none -n --column --type-add="gyp:*.gyp" --type-add="gyp:*.gypi" --heading --no-config --fixed-strings --type=all -e "String with å in it" .

[dajva]

Å, ä and ö works fine on linux although the characters get escaped. Did you try the same with emacs grep? On linux I get the same behavior there. Escaped chars and search hits.
I don't officially support windows since I can't test it. I am happy to accept patches if you find bugs though.

[Whil-]

Hmm, would like to do some debugging but don't know where to start. Any tips to get started? Where is this escaping taking place for example?

[dajva]

The escaping of the search string is triggered by grep-expand-template that is using shell-quote-argument to escape the search string. FWIW, in bash it works without the escaping. I don't know how it works on windows. At least the shell-quote-argument has separate code paths for different platforms.
So I would start with looking into if any escaping is needed on windows and if that is correctly done from within this package.
Since this is reusing code from builtin grep.el you can also try with grep search to see what happens.

[Whil-]

Investigated a bit more and got it to work with default-process-coding-system set to '(latin-1-dos . latin-1-unix).

I tested with a global config:

(setq default-process-coding-system '(latin-1-dos . latin-1-unix))

That made it work for me personally. Not sure if that's the path to take for this package when running on windows. Maybe a slippery slope...!?

[svraka]

I know, it's an old thread but someone might still find useful. The suggestion above was not sufficient for me (I'm on Windows 10 with a Hungarian locale). Although rg can search for non-ASCII characters after setting default-process-coding-system, output will be garbled. This however works:

(prefer-coding-system 'utf-8)
(setq default-process-coding-system '(undecided-dos . windows-1250))

Replace windows-1250 according to your system's locale. undecided-dos is the default on Windows for both reading and writing.