Most Glyphs not showing on macOS - E.g. Menu Glyphs and Character Map

[BDisp]

It's possible to scroll top and see the old screen. It's a weird and ugly situation. Its happens with the Terminal and even ITerm2.

[tig]

The scrolling thing happens on Windows too.

Would you please do me the favor of running the Character Map scenario, selecting Box Drawing & Geometric Shapes, take a screenshot and post here? I don't know how to use a mac.

Also please post what typeface is being used in this GIF.

[BDisp]

Here it is and I also enabled all encodings in the Terminal. iTerm2 is the same. Sorry for the delay.

[tig]

Da fuq? That's a pretty bad/deep bug. I wonder what I did wrong?

As you scroll up/down do ANY glyphs show?

[BDisp]

Here are all the encodings.

[tig]

I don't have time to dive into this today or tomorrow. But it seems there's some silly configuration related bug in ConsoleDriver/CursesDriver when running on Mac based on this.

[tig]

Hey @mklement0 - you seem like you know what you are doing on a Mac. Can you help us with this?

[mklement0]

I also see the symptom, both in Terminal and iTerm2, and it seems that any non-ASCII-range character isn't recognized - even a window title such as "Motörhead" shows the problem, as does the "Se iniciará el análisis" label in the ./Example project.

I know the Mac primarily from a shell perspective and know nothing about ncurses, so in order to be helpful I would need guidance.

Don't know if it's relevant, but skimming the source code I see setlocale(LC_ALL, ""); in binding.cs, which should use the calling process' locale; from man setlocale on a Mac:

an argument of "" will determine the name of the new locale taking into account the environment variables LANG and LC_*

In my case it is en_US.UTF-8.

[BDisp]

Mine was "en_US.UTF-8" too. I've to changed to the "Mac email" and I get the right encoding to my local, but with unicode characters the issue still persist. So the Mac don't obey the setlocale.

[mklement0]

Mac email doesn't look like a valid locale identifier (locale -a shows all defined ones from a shell, and just locale shows wether the one you tried to set took effect; for instance, if you run $env:LANG = 'Mac email' from PowerShell /
export LANG='Mac email' from Bash, locale then shows the fallback locale, "C", in the various LC_* categories).
In what way do you get the right encoding with this?

[BDisp]

I only changed from the preferences of the Terminal and the iTerm2. Terminal only assumes that encoding when I run the GUI and not when I'm typing in the prompt. iTerm2 assumes on both. So I didn't run any command in PowerShell.

[mklement0]

I had some misconceptions:

• It is indeed only the terminal emulator's character-encoding setting that matters for display, for all programs.

• Typically, terminal emulators are configured to reflect the locale + character encoding in the LANG environment variable (which can be overridden via either category-specific individual LC_* environment variables or for all categories with LC_ALL), e.g. en_US.UTF-8.

  • Locale-aware programs do respect these environment variables, so you can do the following to print the current date in French, for instance (using Bash syntax):

    • LC_ALL=fr_FR date -> French-style date

  • And while programs even respect the character-encoding part of the value, this won't work for display in the terminal, because the terminal isn't aware of the requested ad hoc character-encoding change; e.g., on macOS the following correctly translates single byte 0xe8, representing lowercase è in ISO-8859-1 encoding, to single byte 0xc8, its uppercase È equivalent (on Linux, utility tr isn't locale-aware); however, unless the terminal too is set to ISO-8859-1, the result will print as the replacement character:

    • printf '\xe8' | LC_ALL=fr_FR.ISO8859-1 tr '[:lower:]' '[:upper:]' | od -t x1 -> c8; without | od -t x1, in a UTF-8 terminal window, you'd see � or ?, depending on the terminal.

---

Returning to the ncurses library:

On macOS, there's only one binary, /usr/lib/libncurses.dylib, which contains both the narrow and wide-character functions.

The version that ships with macOS 10.15.7 is quite old (ncurses5.4-config --version): 5.7.20081102 vs. 6.2.20200212, the current version.

From man ncurses (emphasis added):

       The  library uses the locale which the calling program has initialized.
       That is normally done with setlocale:

	     setlocale(LC_ALL, "");

       If the locale is not initialized, **the library assumes  that  characters
       are  printable  as in ISO-8859-1,** to work with certain legacy programs.

The mystery is that Terminal.Gui does call setlocale(LC_ALL, ""), yet ncurses on macOS seemingly always uses its built-in default, ISO-8859-1.

One thing worth looking into: Is the setlocale(LC_ALL, "") call truly effective on macOS?

[BDisp]

Thanks. Yes the setlocale (LC_ALL, "") is called. It may not be influential on the macOS.

[tig]

Can we find the source to any console GUI apps that show these glyphs correctly on MacOS? I mean there's got to be other apps that work? Actually, now that I think about it, I'm pretty sure PowerShell with oh-my-posh enabled and using a PowerLine font would likely work on Mac these days or lots of people would be complaining. What else can we look at? (Sorry, but I don't have a Mac so I can't help).

[BDisp]

@tig I don't have it either. I just installed it on VMware.

[mklement0]

The glyphs seem to render fine (please confirm):

If I run the following on macOS from PowerShell (from the project root; the command extracts the Unicode chars. from ConsoleDrive.cs and makes PowerShell print them):

'"{0}"' -f -join ([regex]::Matches((gc -raw ./Terminal.Gui/Core/ConsoleDriver.cs), "\\u....").Value -replace '\\u', '`u{' -replace '$', '}') | iex

I get:

• In iTerm2.app with font 'Monaco regular':

• In Terminal.app with font 'Menlo regular':

[BDisp]

It gives an "iex not found" error and I also can't install Powershell on macOS because it warns that it can't verify the package for the purposes of malicious code.

[mklement0]

What I posted is a PowerShell command, so it only makes sense to run it from inside PowerShell (iex is the built-in alias for the Invoke-Expression cmdlet; the command also uses the gc alias for the Get-Content cmdlet).

How did you try to install PowerShell? The package downloadable from https://github.com/PowerShell/PowerShell#get-powershell should work.

If you have Homebrew installed, you can also try:

brew tap caskroom/cask; brew cask install powershell

[BDisp]

Through this link https://github.com/dotnet/core/blob/master/release-notes/3.0/3.0-supported-os.md#macos.

It give this message on the macOS Catalina:

“powershell-7.0.3-osx-x64.pkg” can’t be opened because Apple cannot check it for malicious software.
This software needs to be updated. Contact the developer for more information.
github-production-release-asset-2e65be.s3.amazonaws.com.

I'll try with brew. Thanks.

[BDisp]

I confirm that it works.

iTerm2-Menlo:

iTerm2-Monaco:

Terminal-Menlo:

Terminal-monaco:

[BDisp]

If ConsoleDriver.cs is rendering well why is the Terminal.Gui app failing? Oh damn.

[mklement0]

ncurses isn't sending UTF-8 to the terminal (it is sending ISO-8859-1, always), so we need to figure out why.

By the way, given that Terminal.Gui can only ever work as expected with UTF-8, the right thing to do would be to refuse startup if the terminal window's character encoding is found to be something other than UTF-8.

[migueldeicaza]

Hello,

The posters are right, this is a problem with ncurses, not really an issue with the terminal.

What might be happening is that ncurses is being initialized implicitly by some code before setlocale is invoked.

Perhaps an additional dependency, or some invocation to the library is taking place that loads and initializes ncurses before we call setlocale. One quick way of testing this theory would be to call setlocale manually in Main():

                        setlocale(LC_ALL, "");

Before anything else - also, not sure if we are now using global constructors (the ones that run on assembly load), that could also be initializing curses before we have a chance to.

[BDisp]

Thanks @migueldeicaza, I tried, but I was unsuccessful. I think the new version of ncurses forces us to save the initial state and restore it on exit, something extra that initscr and endwin did automatically. An alternative that I use to not have to close and reopen the terminal is to press ctrl + c and write reset blindly and now I can enjoy the terminal again. Nothing that a bash doesn't solve, but it would be a hack. It is preferable to let ncurses manage this. I have tried several suggestions but as this is a recent problem with the new version of ncurses I have not been able to.

[migueldeicaza]

I find that surprising, what version of MacOS is doing that?

How do I reproduce that second problem?

[BDisp]

Sorry. This only happens on Linux. The conversation started at #931 and is now here :-)
Start Terminal.Gui on Linux and exit. Type something and you will verify that the console does not respond.

[BDisp]

Can anyone explain these huge differences between ubuntu running on VMware and ubuntu running on WSL? I am not referring to the difference in the font, as WSL uses Windows 10 fonts, but the disconcerted displacement of the characters printed in WSL.

Ubuntu on VMware:

Ubuntu on WSL:

[BDisp]

About most of the glyphs not printing on macOS I tried the following before creating UnixMainLoop in Application.Init and the problem persists:

			if (Driver == null) {
				var p = Environment.OSVersion.Platform;
				if (UseSystemConsole) {
					mainLoopDriver = new NetMainLoop (() => Console.ReadKey (true));
					Driver = new NetDriver ();
				} else if (p == PlatformID.Win32NT || p == PlatformID.Win32S || p == PlatformID.Win32Windows) {
					var windowsDriver = new WindowsDriver ();
					mainLoopDriver = windowsDriver;
					Driver = windowsDriver;
				} else {
					Unix.Terminal.Curses.setlocale (Unix.Terminal.Curses.LC_ALL, "");
					mainLoopDriver = new UnixMainLoop ();
					Driver = new CursesDriver ();
				}
				Driver.Init (TerminalResized);
				MainLoop = new MainLoop (mainLoopDriver);
				SynchronizationContext.SetSynchronizationContext (new MainLoopSyncContext (MainLoop));
			}

[tig]

Can anyone explain these huge differences between ubuntu running on VMware and ubuntu running on WSL? I am not referring to the difference in the font, as WSL uses Windows 10 fonts, but the disconcerted displacement of the characters printed in WSL.

Ubuntu on VMware:

Ubuntu o WSL:

What terminals are you using here?

[BDisp]

What terminals are you using here?

WSL Ubuntu itself on Windows and the Ubuntu Terminal on the virtual machine with Ubuntu installed.
It seems to me that I am coming to some conclusion. Rune.ColumnWidth has a return that is not always what is expected and I will try to submit a PR on NStack to correct how to interpret what returns ustring.ConsoleWidth. There are Unicode characters that return a width equal to 0 and this induces a strange behavior in the printing of the characters because the calculation of the print spacing is poorly calculated. On the Ubuntu system that is installed on the virtual machine they print well because they interpret the characters well but in WSL and Windows Terminal it causes the disproportionate printing of the characters and their misalignment.

[tig]

What terminals are you using here?

WSL Ubuntu itself on Windows and the Ubuntu Terminal on the virtual machine with Ubuntu installed.
It seems to me that I am coming to some conclusion. Rune.ColumnWidth has a return that is not always what is expected and I will try to submit a PR on NStack to correct how to interpret what returns ustring.ConsoleWidth. There are Unicode characters that return a width equal to 0 and this induces a strange behavior in the printing of the characters because the calculation of the print spacing is poorly calculated. On the Ubuntu system that is installed on the virtual machine they print well because they interpret the characters well but in WSL and Windows Terminal it causes the disproportionate printing of the characters and their misalignment.

See #41

I've already been down this path...

[mklement0]

Re macOS: The setlocale method is misdefined as returning int, according to the man page it is char *, so if you define it as IntPtr and convert it to a string with Marshal.PtrToStringAuto(), it shows that the locale is seemingly correctly picked up from the environment.

As an aside: Window.initscr() is reentered via main_window = new Window (methods.initscr ()); - is that expected?

The strange thing is that I eventually did get the glyphs to render correctly, namely by setting LC_CTYPE (define it as 0) instead of LC_ALL, to target the character-encoding category explicitly - even though LC_ALL should set all categories.

Similarly strangely, querying the effective LC_ALL category (by passing null instead of "") yields the correct value even when only LC_CTYPE was set.
Do the other locale categories matter and should they be set explicitly as well?

[BDisp]

As an side: Window.initscr() is reentered via main_window = new Window (methods.initscr ()); - is that expected?

Yes.

I'll investigate. LC_ALL should work unless there is a gap in bitwise that does not cover all others.

[mklement0]

Thanks, @BDisp, but the logic of LC_ALL is not a matter of bit patterns - as man setlocale on macOS states:

LC_ALL Set the entire locale generically.

That is, the LC_* categories are not flags to be bit-ORed - they are distinct values, with LC_ALL by convention overriding all others.

[BDisp]

@ mklement0 thank you very much for your commitment. Your research is of enormous value. I hope you can come up with the solution to this because I'm already burning the fuses :-)

[migueldeicaza]

Hello,

@mklement0 found the culprit, the issue is that the LC_ALL definition is wrong for Mac. It should be 0, according to locale.h:

#define LC_ALL          0
#define LC_COLLATE      1
#define LC_CTYPE        2
#define LC_MONETARY     3
#define LC_NUMERIC      4
#define LC_TIME         5
#define LC_MESSAGES     6

I do not know what the value is on Linux, but if it is not zero, then we will need to check the OS and use 0 on Mac, and 6 o Linux.

[migueldeicaza]

Also, for good measure, we should change setlocale to return an IntPtr, and not an int, in case we ever find a system that uses different calling conventions - it is just good hygiene.

[tig]

Can someone please verify that this is fixed? I don't have/use a mac. Thanks.

[BDisp]

I uninstalled VMware on Windows 10 and now I just use VirtualBox and I can't install macOS.

[mklement0]

It looks like the problem is still present, since the macOS-specific LC_ALL value still isn't being set.

Replacing https://github.com/migueldeicaza/gui.cs/blob/cbbce244dc222eaae6190dc694afe6f1aaedc9e1/Terminal.Gui/ConsoleDrivers/CursesDriver/constants.cs#L168

with the following seems to work:

static public int LC_ALL { get; private set; }
static Curses() {
  LC_ALL = System.Runtime.InteropServices.RuntimeInformation.IsOSPlatform(System.Runtime.InteropServices.OSPlatform.OSX) ? 0 : 6;
}