Enconding problems with color.diff.meta

[firesoft-de]

Hi,
I encountered some problems using diff-so-fancy on windows with

git config --global color.diff.meta       "11"

Using this option the console produces the following output

ΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇ
modified: xxx/xxx/xxx.sh
ΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇ
@@ -1,8 +1,12 @@
 #!/bin/sh

instead of

───────────────────────────────────────────────────────────────────────────────
modified: xxx/xxx/xxx.sh
───────────────────────────────────────────────────────────────────────────────
@ xxx/xxx/xxx.sh:4 @
#!/bin/sh

[scottchiefbaker]

What terminal are you using? The dash character it draws is a unicode character which may cause problems with some terminals. You can see it in action here.

[firesoft-de]

I'm using MIGW64 so underneath it should be the default Windows Terminal.

[scottchiefbaker]

There is also this config option.

[firesoft-de]

I got it running for me (with the unicode symbol) by removing the meta option. I didn't tested if a color value given to meta produces valid output. Just for the record: With .meta = 11 every mingw window and also every other instance of cmd or powershell shows broken glyphs. Even when git isn't running. Very strange o.O

[rdnlsmith]

This happened for me as well. I'm using:

• Windows Terminal 0.9.433.0
• PowerShell Core 6.2.4
• Git 2.25.1.windows.1

Setting meta = yellow bold, for instance, works fine, or even a different number like 52. It seems to be specifically the 11 which produces the odd symbols.

[OJFord]

Do you see the same issue with echo -e "\\033[38;5;11m foo"?

[rdnlsmith]

I ran the PowerShell equivalent and got this:

I also ran it in Git Bash (still via Windows Terminal) and got this:

Possibly related, if I attempt to use a number to specify a highlight background color (e.g. oldHighlight = red bold 52 as in the README), I get the same shade of grey no matter what:

This occurs in both PowerShell and Git Bash. I'm not certain whether it's a limitation of Windows Terminal, or if it's a Windows-specific problem in d-s-f. For now, I'm using e.g. red bold reverse instead, which does work:

[rdnlsmith]

I think this confirms that Windows Terminal is capable of displaying both foreground and background colors specified via ANSI escapes:

[scottchiefbaker]

The Perl equivalent would be:

perl -E 'binmode(STDOUT, "encoding(UTF-8)"); $bar = "\x{2500}" x 50; $yellow = "\e[38;5;11m"; print "$yellow$bar\n";'

What output do you get from that? Whatever we come up with, this is going to end up being some weird terminal problem.

[rdnlsmith]

In Git Bash, I got the weird glyphs:

I tried to run it in PowerShell, and got an error. I'm sure I'm probably doing this wrong.

[scottchiefbaker]

PowerShell may not understand the ' vs " (I'm not a PowerShell expert).

Write those commands to a text file and then run perl.exe on that script.

[rdnlsmith]

That executed. I got the glyphs again.

[scottchiefbaker]

I'm at a loss... clearly diff-so-fancy is doing the right thing, it's just something with your terminal that's interpreting the UTF-8 characters incorrectly.

[rdnlsmith]

I'm not sure it's UTF-8, necessarily. If I use \e[38;5;52m instead of \e[38;5;11m, it draws the lines correctly, although the colors are still incorrect.

My thought was perhaps that the escape sequence with 11 is changing the code page? But, I don't if that's even possible. I also wonder if it could have something to do specifically with the perl.exe that ships with Git for Windows. I may just open an issue for Windows Terminal and see what they think.

Thanks for your effort!

[scottchiefbaker]

Definitely a weird issue. Let me know what you find.

[rdnlsmith]

It seems like this issue is a limitation of perl-for-Windows. I had the thought to create a PowerShell-based version, which works nicely—except, perhaps unsurprisingly, less-for-Windows has the same problem, and there doesn't seem to be a suitable Windows-native alternative (you can pipe it through Out-Host -Paging, but it only pages forwards).

[scottchiefbaker]

Does it work if you use cygwin / bash?

[rdnlsmith]

Cygwin exhibits the same problem. I imagine that WSL would work, on account of having native Linux binaries instead of (old-ish) Windows versions. I was just hoping to have a more "native" solution.

Anyways, it seems like this is solidly outside of d-s-f's control, so you may as well close the issue in my opinion. Perhaps you might mention in the README that Windows users without WSL might want to use

  git config --global color.diff-highlight.oldNormal    "red bold"
- git config --global color.diff-highlight.oldHighlight "red bold 52"
+ git config --global color.diff-highlight.oldHighlight "red bold reverse"
  git config --global color.diff-highlight.newNormal    "green bold"
- git config --global color.diff-highlight.newHighlight "green bold 22"
+ git config --global color.diff-highlight.newHighlight "green bold reverse"

- git config --global color.diff.meta       "11"
+ git config --global color.diff.meta       "yellow bold"

instead of the other settings. I also may yet continue my PowerShell effort, in hopes that a better pager might come along.

Thanks again for your time!

[scottchiefbaker]

Glad we sort of got things resolved.