Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Full unicode support #306

Open
kazimuth opened this issue Jan 13, 2017 · 20 comments

Comments

@kazimuth
Copy link

commented Jan 13, 2017

TL;DR: Supporting unicode is hard, and it might be a good idea to use a library that knows how to do it well. Maybe look into using harfbuzz as well as freetype for font rendering. Unicode-width is good but doesn't do everything.

The problem of translating sequences of unicode codepoints to actual you-can-draw-this-on-screen glyphs, supporting things like character width (#265), ligatures (#50), bidirectional text (like in arabic), and text reordering (!), is called complex text layout. It is, appropriately, complex, and most terminals don't actually do it very well.

Windows and Mac both have systems that perform layout, integrated with their font rasterizers:

  • On windows, there's several different supported libraries, from various periods of windows history: DirectWrite, GDI, and Uniscribe on windows.
  • On OS X, there's Core Text.
  • On linux, there's a stack: freetype for rasterizing, harfbuzz for "text shaping", and pango for full layout. All of these libraries are actually cross-platform, though. See this article for more on the difference between pango and harfbuzz.

There's also ICU, which is cross-platform and supported by IBM (I think?)

Of the available options:

  • OS X Terminal / Iterm2 use Core Text
  • Terminator (and everything else that uses GTK) uses Pango
  • Windows CMD and Powershell use black magic and baby tears
  • Chrome uses harfbuzz
  • Firefox uses pango
  • ...

It seems to me like harfbuzz is the best option in terms of cross-platform support and level of control. You basically hand it a line of text and it tells you all the glyphs to draw in that line. Keep in mind I'm not actually a font rendering person, though, and it's possible I'm missing important details here.

This would probably have a performance cost, but I'm not sure how much of a performance cost. With a clever implementation it might not be too bad, and would be a huge boon to international users.

Other relevant links:

@kazimuth

This comment has been minimized.

Copy link
Author

commented Jan 13, 2017

(The challenge, of course, is that a lot of these features may interfere with the way the terminal is expected to work; and also it may be difficult to implement them in a performant way. If you have to call freetype to reraster things every frame you lose the benefits of GPU rendering. So it might be reasonable to put this off for a while.)

@khaledhosny

This comment has been minimized.

Copy link

commented Jan 15, 2017

You might want to check mlterm which supported bidirectional text for a long time, and it latest versions seems to have added more support for complex text using HarfBuzz as well.

@bjesus

This comment has been minimized.

Copy link

commented Jan 26, 2017

I just want to confirm that it's not a font issue - I'm running Alacritty on Arch Linux and all Hebrew text just shows up as boxes. I'm not talking being displayed right-to-left - as said before, many terminals just keep everything left to right - but no matter what font I choose I can't even see the letters. Is this normal or am I missing something?

screenshot from 2017-01-26 19-16-03

@jwilm

This comment has been minimized.

Copy link
Owner

commented Jan 26, 2017

@bjesus Not sure why you say this isn't a font issue. That's definitely a font issue. The font alacritty is looking in for those glyphs doesn't have them.

@bjesus

This comment has been minimized.

Copy link

commented Jan 26, 2017

Hi! Thanks for Alacritty! I never said it wasn't font issue, I was only asking... and the reason I thought so is that I tried many different fonts that worked just fine on other terminals. Any suggestion for a font that I should try? I've tried monospace, "Droid Sans Mono", "Consolas" and even "Arial". None of them showed the Hebrew characters.

@dywedir

This comment has been minimized.

Copy link
Contributor

commented Jan 26, 2017

@bjesus try Cousine (from ttf-croscore)

@jwilm

This comment has been minimized.

Copy link
Owner

commented Jan 26, 2017

@bjesus the reason those other fonts work is not because they have the glyphs, but because the application you're using supports fallback fonts--something we plan to add to Alacritty.

@bjesus

This comment has been minimized.

Copy link

commented Jan 26, 2017

Thanks @N-006 and @jwilm , it work just fine now. Sorry about the misunderstanding, I didn't think about fallback fonts.

@voxadam

This comment has been minimized.

Copy link

commented Dec 20, 2017

@yoshuawuyts

This comment has been minimized.

Copy link

commented Jan 7, 2018

For people (like me) looking on how to enable unicode on Alacritty. There is now an RFC in #957 for font configuration - including fallback fonts, which would solve most unicode issues.

Tooke me a few clicks to find out about it, figured linking back here would be useful!

@horta

This comment has been minimized.

Copy link

commented Jan 30, 2019

printf '\xF0\x9D\x9C\x99' is supposed to print 𝜙, but on Alacritty it prints an empty space. Is this due to the limitations that this thread refer to? Or is it a bug?

@chrisduerr

This comment has been minimized.

Copy link
Collaborator

commented Jan 30, 2019

tmp

@horta Most likely a font issue.

@horta

This comment has been minimized.

Copy link

commented Jan 30, 2019

I just removed the alacritty configuration file, restart it and tried again. Still prints a space. Interestingly, some unicode characters work:

screenshot 2019-01-30 at 23 18 13

@horta

This comment has been minimized.

Copy link

commented Jan 30, 2019

I'm using macos, so it defaults to Menlo. However, using the same Menlo font on the Terminal.app it prints correctly.

@xarthurx

This comment has been minimized.

Copy link

commented Mar 23, 2019

image

I'm also experiencing issues which I believe is due to unicode width.

I use a "lambda" symbol in my prompt. Two issues noticed:

  1. the cursor is too far away from the real location (in the pic, the curcor is actually right after the letters, no space in between);

  2. when using tmux to do two panes, this issue also cause the separator char mis-aligned.

@chrisduerr

This comment has been minimized.

Copy link
Collaborator

commented Mar 23, 2019

@xarthurx See #1295 for tracking that issue.

@smeikx

This comment has been minimized.

Copy link

commented May 1, 2019

On macOS 10.14.4 using a german keyboard just typing an umlaut (like ä) produces �ä. All following umlauts look fine: �öüä.
The real trouble starts when typing sharp s (ß), that produces: ��[7m<009f>, every time. Similar problem with ohter characters like ellipsis (…), ��[7m<0080>, or em dash (—), ��[7m<0080><0094>.

@chrisduerr

This comment has been minimized.

Copy link
Collaborator

commented May 1, 2019

Do you have the correct locale set?

@smeikx

This comment has been minimized.

Copy link

commented May 1, 2019

No, I didn’t know about that! iTerm seems to infer it from the system settings, as echo $LANG showed; in Alacritty that didn’t print anything.
I added export LANG="de_AT.UTF-8" to my .zshrc to resolve the problem.
Thank you and sorry for polluting the thread.

@tyru

This comment has been minimized.

Copy link

commented Aug 11, 2019

I missed this issue. related?
#1101 (comment)
#1606 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.