Misaligned register output for Unicode journal using mintty

[kohanyirobert]

I'm using msys2 on Windows 10 with mintty 2.9.0 (x86_64-pc-msys) and I if I have a journal like this

1/1 a
        ábc                                      123
        abc

and I do hledger -f a.journal r this is the output

2019/01/01 a                    ábc                           123           123
                                abc                           -123             0

The same thing in cmd.exe setting the default codepage to Unicode with chcp 65001 works, the output look likes this

2019/01/01 a                    ábc                            123           123
                                abc                           -123             0

(which is the expected).

I've built hledger using these steps.

C:\Users\rkohanyi>hledger --version
hledger 1.12

If I just cat the journal file on mintty it looks okay, as the second output. Same with type (cat equivalent) in cmd.exe.

In mintty I use Lucida Console as my font (tried changing it, but it doesn't matter, shouldn't matter), locale is en_US character set is UTF-8 (Unicode).

The file itself is encoded with UTF-8 (verified in vim).

It seems to me that this behaviour is displayed when hledger executes through mintty.
Any idea on changing/fixing this?
I mainly use mintty and this 1 character misalignment throws me off completely sometimes :D

[simonmichael]

Thanks for the bug report. I don't have a machine to test this on myself. I wonder if it's any relation to #536 or #708 ?

[kohanyirobert]

#708 is not related. The problem described by OP happens when I use cmd.exe and I don't set the terminal's codepage to Unicode (by default it's latin1 or worse, but one can switch it with chcp 65001 and then everything is displayed perfectly).

#536 related I think. With my above example in mintty this

$ hledger -f a.journal b á
--------------------
                   0

doesn't work (although I ran this a few minutes before starting to write my post ... and it worked ... or at least it seemed to work, but now it doesn't :sad:).

But it does work in cmd.exe with the Unicode codepage set.

I've skimmed through the issue and basically it boils down to either it works under Cygwin or Windows, right? Not both? :D Not a big deal. Weird. One solution is to use different binaries in different terminals, maybe? I won't go down that route tho', it's not a blocking problem. Easier to rename accounts not use accented letters.

[simonmichael]

Thanks for the testing of this and the other windows issues. I'll update those pages also, but here's a summary of my current understanding of windows non-ascii issues:

• six windows shell/build environments mentioned so far:
  cmd, powershell, cygwin-mintty, git-mingw64, msys2-mintty, appveyor

• I think git-mingw64 can be considered one of the minttys, appveyor is cmd, and cmd and powershell can be considered similar.

• three issues under discussion:

  • hledger bal Расходы doesn't work under Cygwin #536 non-cygwin hledger build in cygwin-mintty: non-ascii queries fail (due to misparsed journal)
  • hledger-web on Windows can show non-ascii characters as mojibake (garbled) #708 appveyor hledger-web build started from mingw64 or cmd: misrenders characters
  • Misaligned register output for Unicode journal using mintty #961 msys2-mintty hledger build in msys2-mintty: register misaligns with non-ascii characters

• cmd/powershell builds in cmd/powershell: chcp 65001 generally resolves such issues

• cmd/powershell hledger builds in mintty (cygwin or msys2): generally suffer from these issues. Using a mintty build usually resolves them (but not Misaligned register output for Unicode journal using mintty #961).

• mintty hledger builds in cmd/powershell: ? presumably suffer from these issues and using a cmd/powershell build resolves

Back to #961: the á seems to shorten the line by a character. hledger has built-in logic to compensate for double width characters, perhaps it's misjudging á to be one of these ? If you have the time, you could try debugging it (eg make ghci, insert traceShowId or dbg0 "LABEL"'s in hledger/Hledger/Cli/Commands/Register.hs postingsReportItemAsText, :reload, :main r)...

--------------------------------------- 80 -------------------------------------
normal:
2019/01/01 a                    ábc                            123           123
                                abc                           -123             0
--------------------------------------- 80 -------------------------------------
yours:
2019/01/01 a                    ábc                           123           123
                                abc                           -123             0

[simonmichael]

Or as a quick test: what do you get from make ghci, charWidth 'á' ? I get 1. If you get 2, perhaps a fix is needed in hledger-lib/Hledger/Utils/String.hs charWidth. (Forcing it to 2 did replicate the kind of output you're seeing.)

[kohanyirobert]

I did the following in both cmd.exe and mintty: started ghci, then import Hledger.Utils.String, then charWidth 'á'. Both reported 1.

I did not fully comprehend this part:

(eg make ghci, insert traceShowId or dbg0 "LABEL"'s in hledger/Hledger/Cli/Commands/Register.hs postingsReportItemAsText, :reload, :main r)..

I'm okay to do this, but break it down for me a bit :D Do I need to modify the source and recompile to do this? Or can I use ghci to somehow hook into the execution and modify it? I know a bit of Haskell (I get and used traceShowId), but I'm not using it daily so I'm not sure what you mean.

[simonmichael]

Certainly! That is shorthand for a typical debugging workflow. It involves modifying the source a little bit and running it (in interpreted mode) from the ghci prompt. make ghci in the main hledger directory gets a useful ghci prompt. :main [ARGS] runs hledger from the ghci prompt. :reload reloads any source file changes into ghci. The kind of source file change I would typically add is to print out intermediate values. Inserting traceShowId or dbg0 "somelabel" before an expression is a good way to do that. Feel free to join #hledger for more help.

[kohanyirobert]

I cloned the repo (at this commit: c8b0c9a) and did make ghci in the repo's root dir in mintty and cmd.exe both.
I think it downloaded/compiled everything but there's in error just before the ghci prompt starts

C:\Users\rkohanyi\Work\git\hledger>make ghci
which: no gsed in ... [list of places gsed was looked for]
stack exec -- ghci  -rtsopts -Wall -fno-warn-unused-do-bind -fno-warn-name-shadowing -fno-warn-missing-signatures -fno-warn-orphans -fno-warn-type-defaults  -ihledger-lib -ihledger-lib/other/ledger-parse -ihledger -ihledger-ui -ihledger-web -ihledger-web/app -ihledger-api     -DPATCHLEVEL=166 -DDEVELOPMENT -DVERSION="\"1.12.99\"" hledger/Hledger/Cli/Main.hs
GHCi, version 8.6.3: http://www.haskell.org/ghc/  :? for help
Loaded GHCi configuration from C:\Users\rkohanyi\Work\git\hledger\.ghci

hledger\Hledger\Cli\Commands\Roi.hs:207:0: error:
     error: missing binary operator before token "("
     #if MIN_VERSION_math_functions(0,3,0)

    |
207 | #if MIN_VERSION_math_functions(0,3,0)
    | ^
`gcc.exe' failed in phase `C pre-processor'. (Exit code: 1)

Not sure about this. I don't have gnu-sed and I couldn't install it in pacman inside mintty... is it strictly required?

I've checked that I have a recent-ish gcc (7.3.0), not sure if that's a problem.

I didn't have time to dive into this more than this (maybe next week), but I'm still here and listening if you have any input, I'll help get to the bottom of this.