included less can't display utf8 chars

[jankatins]

• I was not able to find an open
  or closed issue
  matching what I'm seeing

Setup

• Which version of Git for Windows are you using? 32-bit or 64-bit? Include the
  output of git version as well.

64bit AFAIK

c:\data\external\R\repr (packageIQR)
λ git version
git version 2.7.2.windows.1

• Which version of Windows are you running? 32-bit or 64-bit?

64bit Windows 7

• What options did you set as part of the installation? Or did you choose the
  defaults?

Defaults (add cmd to path, nothing else added, but cmder adds the unix commands into the path)

• Any other interesting things about your environment that might be related
  to the issue you're seeing?

I use cmder (ConEmu + clink) and usually put chcp 65001

Details

• Which terminal/shell are you running Git from? e.g Bash/CMD/PowerShell/other

cmder with cmd, but this also happens in a normal cmd window

• What commands did you run to trigger this issue? If you can provide a
  Minimal, Complete, and Verifiable example
  this will help us understand the issue.

c:\data\external\R\repr (packageIQR)
λ which less
C:\Program Files\Git\usr\bin\less.exe

c:\data\external\R\repr (packageIQR)
λ chcp 850
Aktive Codepage: 850.

c:\data\external\R\repr (packageIQR)
λ git slog |grep a27c12e
a27c12e don’t double escape latex list names

c:\data\external\R\repr (packageIQR)
λ git slog
[...]
a27c12e don<E2><80><99>t double escape latex list names
[...]

c:\data\external\R\repr (packageIQR)
λ chcp 65001
Aktive Codepage: 65001.

c:\data\external\R\repr (packageIQR)
λ git slog |grep a27c12e
a27c12e don’t double escape latex list names

c:\data\external\R\repr (packageIQR)
λ git slog
[...]
a27c12e don<E2><80><99>t double escape latex list names

• What did you expect to occur after running these commands?

all commands should produces a proper don’t

• What actually happened instead?

UTF8 is mangeled when used with less (which I infered from git slog |grep a27c12e |less producing the same mangeled output

• If the problem was occurring with a specific repository, can you provide the
  URL to that repository to help us with testing?

https://github.com/IRkernel/repr.git

[fourpastmidnight]

@JanSchulz So, I found this SO Q&A that describes this problem (probably on an actual Linux workstation).

I tried running a few commands listed in the accepted answer and I got the same kind of thing you did, just character codes, not the Unicode symbol. Later in the SO exchange, someone mentioned the LESSCHARSET variable.

Now, this may be specific to my installation of Git for Windows, but run the following commands to see if your situation is the same:

$ echo -e '\xe2\x82\xac'
€
$ echo -e '\xe2\x82\xac' | less
        1 ▒<82>▒
$ echo $LESSCHARSET
latin1
$ echo -e '\xe2\x82\xac' | LESSCHARSET=utf-8 less
        1 €

Hope that helps.

[jankatins]

I can reproduce:

λ bash
$ echo -e '\xe2\x82\xac'
€
$ echo -e '\xe2\x82\xac' | less
# bad escaped chars
$ echo $LESSCHARSET
# nothing, not set
$ echo -e '\xe2\x82\xac' | LESSCHARSET=utf-8 less
# Good €

Is there a way that git sets that somewhere so that the used less has it available or do I have to set it?

[kostix]

To make it clear: does less misbehave only under chcp 65001 or in the "normal" case as well (chcp left alone and is hence set to the "OEM" codepage)?

[kostix]

I also wonder if it works if you invoke it with ... | less -R or with LESS=-R in environment.

[fourpastmidnight]

@kostix For me, my LESS variable is set as follows:

$ echo $LESS
-i -N -w -z-4 -g -e -M -X -F -R -P%t?f%f \ :stdin .?pb%pb\%:?lbLine %lb:?bbByte %bb:-...

So, in my case, I have a custom Git prompt that I created. In my bash.rc, I cargo-culted something and have LESSCHARSET=latin1 in there, which explains why I have this issue in my installation. (I guess I'll be removing that variable definition. 😉) However, this doesn't explain why @JanSchulz is having the issue. Per a thread on linuxquestions.org:

less manpage agrees with you:

If neither LESSCHARSET nor LESSCHARDEF is set, but any of the strings "UTF-8", "UTF8", "utf-8" or "utf8" is found in the LC_ALL, LC_CTYPE or LANG environment variables, then the default character set is utf-8.

My LANG environment is set to en_US.UTF-8. So, if I execute the following on my install, I get the correct output:

$ export LESSCHARSET=   # My LESSCHARSET was set to latin1 by my bash.rc :(
$ echo -e '\xe2\x82\xac' | less
        1 €

This is the default "out-of-the-box" behavior, and the expected behavior. I suggest you check your locale settings again and have a look at the linked-to StackOveflow Q&A.

[jankatins]

does less misbehave only under chcp 65001 or in the "normal" case as well (chcp left alone and is hence set to the "OEM" codepage)?

in both cases (see first git slog in the initial message).

when I use bash, I have only LANG=C in env (no LC.*)

As far as I can remember, I have not configured any bash related config file (I usually use cmd in cmder).

Anyway, I think one problem is that even with cp 850, git spits out utf8 (that's at least my interpretation from the above example with cp 850), so my guess would be that less should always assume that it gets UTF8?

[fourpastmidnight]

@JanSchulz Per the blockquote in my comment above, less uses a heuristic to determine what encoding to output.

From what you posted about your setup, this is my current understanding:

1. You run Git from the Windows Command Prompt through a console host called cmder
2. You change the Windows Command Prompt codepage to 65001 (UTF-8)
3. Your LANG environment variable is set to C.

From what I've been able to glean from browsing around about this issue, less determines what character encoding to output data in via the blockquote I posted above earlier (I'll repost it again):

less manpage agrees with you:

If neither LESSCHARSET nor LESSCHARDEF is set, but any of the strings "UTF-8", "UTF8", "utf-8" or "utf8" is found in the LC_ALL, LC_CTYPE or LANG environment variables, then the default character set is utf-8.

So what this tells me is:

1. You've set your command prompt codepage to UTF-8, and so the command prompt is capable of displaying UTF-8 characters (though, this SO Q&A makes it abundantly clear that Windows Command Prompt and UTF-8 (i.e. codepage 65001) really don't work well together and a lot of things tend to break).
2. According to the blockquote above, less didn't find anything to indicate that it should output data in the UTF-8 character encoding:
   • LESSCHARSET= -- it's not defined or null
   • LESSCHARDEF= -- it's not defined or null
   • You stated that you have not set any LC_* variables
   • You have LANG=C

That last bullet point is most likely the culprit here. It appears that less processes the list in the block quote from top-to-bottom (though, someone correct me if I'm wrong). Since in the first three bullets above, the environment variables are undefined or null, less looks to LANG=C, which means that less will use the C "locale" when outputting data. What is the C locale? Basically, it's US English ASCII, or codepage 850. So, you have a mismatch. The terminal can display UTF-8 (because you changed the console to codepage 65001) but less thinks that it should be displaying data it outputs in codepage 850. And codepage 850 doesn't understand UTF-8, and so less outputs "unprintable" ASCII characters in hexadecimal, e.g. <82> in my example earlier.

You have two options:

1. Ensure that LESSCHARSET=UTF-8, though on my install, this is LESSCHARSET=.
2. Choose a UTF-8 locale for LANG, such as us-EN.UTF-8 for example. By default, Git for Windows in the BASH environment sets this variable (see below).

Also, I use Git from the BASH via MinTTY, and I have a %PROGRAMFILES%\Git\etc\profile.d\lang.sh file that comes with Git for Windows and (at least when running via the BASH) this file sets the system-wide language settings that the Git environment will use (including utilities such as less). Basically, it has one line:

test -z "${LC_ALL:-${LC_CTYPE:-$LANG}}" && export LANG=$(/usr/bin/locale -uU)

This is how my LANG variable is set to us-EN.UTF-8 for number 2 above.

HTH.

[jankatins]

Thank you, it finally clicked why I ran into that problem. :-)

Just for the reference: both setting LESSCHARSET=UTF-8 and removing LANG worked... I think I remember that I ran into problems setting LANG=en-EN.UTF-8 because it was missing locale data. So I'm going with setting LESSCHARSET

Thanks again!

[fourpastmidnight]

@JanSchultz I'm glad I could help!