Add better support for Arabic language to Text Layer

[rodolforg]

Arabic, Hebrew and other languages that needs text shaping is now finally supported, thanks to FriBiDi and HarfBuzz libraries.
These libraries are already backend for Gtk/Glib, so there is not any change for packaging.

Fix #507
Fix #1621

[rodolforg]

Actually I don't know anything about those languages.
I tested only compared some random web texts appearance with that Synfig now provides.
So, please, test it – if you find any problem, and try to explain me what is wrong.

@Fahad-Alsaidi could you test it?

[rodolforg]

Known issue: TAB is not rendered properly when HarfBuzz is enabled.

[lgtm-com]

This pull request introduces 1 alert and fixes 1 when merging 2089a98 into 223026f - view on LGTM.com

new alerts:

• 1 for FIXME comment

fixed alerts:

• 1 for Multiplication result converted to larger type

[rodolforg]

Comparison: orange/yellow (whatever): Synfig Studio 1.4.0; the green one is current proposal

[lgtm-com]

This pull request introduces 1 alert and fixes 1 when merging c42938b into 223026f - view on LGTM.com

new alerts:

• 1 for FIXME comment

fixed alerts:

• 1 for Multiplication result converted to larger type

[rodolforg]

The last forced push was to amend the following comments, due to @ice0 review:

• Layer_FreeType: separate code for possible folders for fonts (make new method static)
• Layer_FreeType: separate utf32 converter and line parser (remove the useless lambda function that I failed to implement in the next commit for FriBiDi) and solved the BiDi issue another way)
• Layer_FreeType: add support to text shaping via HarfBuzz (mark a construct as explicit and added some comments)

[lgtm-com]

This pull request introduces 1 alert and fixes 1 when merging 693c61f into f32ae7f - view on LGTM.com

new alerts:

• 1 for FIXME comment

fixed alerts:

• 1 for Multiplication result converted to larger type

[rodolforg]

@FirasH could you test it?
Mixing scripts, adding texts with multiple lines, etc.?

I don't know Arabic, Hebrew, etc... So I tried to make according to the manuals, but I had to adapt to Synfig needs.

[FirasH]

@FirasH could you test it?
Mixing scripts, adding texts with multiple lines, etc.?

I don't know Arabic, Hebrew, etc... So I tried to make according to the manuals, but I had to adapt to Synfig needs.

Testing now!

[FirasH]

I cannot start the AppVeyor image on Windows, need to build it.
I'll report ASAP.

[FirasH]

1. Noticed that the word سلام is displayed with a spacing between the first and the second character as displayed here (top one is correct, made with Scribus):
   
   even changing the first letter the issue is still present.
   Keeping in mind Arabic is typed from right to left the problem is related to the part of text in the "middle" which is composed of two letters ل and ا

2. Noticed that the word كَلَام doesn't have the wrong spacing between first and second character, but the "accents" are wrongly placed as displayed here (top one is correct):
   
   If I remove the "accents" characters are split as before:
   

[rodolforg]

Thank you, @FirasH !

I'll fix it. Luckily, this error happened after I tried to fix the case when I include an English word in an Arabic phrase like I show a few comments above. I tested without that fix, and the text shapes like you showed me how it should do! :)

What about multiple lines? If a single phrase extends to next line it shapes differently if the second line would be another phrase starting with the same word? Er... So confusing. Let me show I want to say. Please consider each character-sequence as a valid word, and if a character-sequence repeats, it means they are the same word:

Case 1: one single phrase in two lines (5 words in the first line and the last 3 in the second line)

eeee dddd cccc bbbb aaaaa
.hhhh gggg ffff

Case 2: two phrases, one per line

.dddd cccc bbbb aaaaa
.mmmm jjjj ffff

My question is: does the ffff word could be shaped differently when comparing both cases above? It's the same word using the same characters, but one is in the middle of the sentence, another (case 2) is at the sentence begining.
Sorry for this question, but I don't know nothing about this language.

In English, Portuguese (my mother tongue/native language), French, Spanish, German, I know that "it doesn't matter": the first character needs to be changed to "upper case", but it's another character/byte that user need to type directly, it's not automatic. In Japanese, the position in phrase doesn't matter at all.

Explaining why I'm asking it: if it doesn't matter for Arabic and other languages, I can split the text into lines first and optimize the code a bit. Otherwise I could only do it at each frame rendering (in current text renderer).

[FirasH]

What about multiple lines? If a single phrase extends to next line it shapes differently if the second line would be another phrase starting with the same word? Er... So confusing. Let me show I want to say. Please consider each character-sequence as a valid word, and if a character-sequence repeats, it means they are the same word:

Case 1: one single phrase in two lines (5 words in the first line and the last 3 in the second line)

eeee dddd cccc bbbb aaaaa
.hhhh gggg ffff

Case 2: two phrases, one per line

.dddd cccc bbbb aaaaa
.mmmm jjjj ffff

My question is: does the ffff word could be shaped differently when comparing both cases above? It's the same word using the same characters, but one is in the middle of the sentence, another (case 2) is at the sentence begining.
Sorry for this question, but I don't know nothing about this language.

The shape of ffff is the same after a white space or a new line, Case 1 and Case 2 should look the same.

In English, Portuguese (my mother tongue/native language), French, Spanish, German, I know that "it doesn't matter": the first character needs to be changed to "upper case", but it's another character/byte that user need to type directly, it's not automatic. In Japanese, the position in phrase doesn't matter at all.

Explaining why I'm asking it: if it doesn't matter for Arabic and other languages, I can split the text into lines first and optimize the code a bit. Otherwise I could only do it at each frame rendering (in current text renderer).

In Arabic there is no upper cases, in your example ffff should look the same for both Case 1 and Case 2.

[Fahad-Alsaidi]

Great work I test it, and it works fine except Firas comments

[rodolforg]

I believe I fixed it :) @FirasH @Fahad-Alsaidi

[lgtm-com]

This pull request fixes 1 alert when merging ace87eb into fb80f27 - view on LGTM.com

fixed alerts:

• 1 for Multiplication result converted to larger type

[Fahad-Alsaidi]

I believe I fixed it :) @FirasH @Fahad-Alsaidi

well, there is no difference see

I notice one bug in many fonts in my system, the last letter is converted to rectangular see:
in Dubai font

in Amiri Quran font

here is a test file with fonts
arabic-test-file.zip

[rodolforg]

Oh my. I was so happy before… :)


I'll try to understand what is happening.

[rodolforg]

Hey! I think I found out the problem @Fahad-Alsaidi : synfig doesn't recognize the font file ;)
Try to write the filename (with file extension)! :D

[Fahad-Alsaidi]

I test it again

I got this error:

error: Layer_Freetype: Unable to open font face. (err=1): Dubai_Regular.otf

how can I check that I am using the same code you have??

[lgtm-com]

This pull request fixes 1 alert when merging fd29152 into fcaa6fb - view on LGTM.com

fixed alerts:

• 1 for Multiplication result converted to larger type

[ice0]

Merged. Thank you!

[morevnaproject]

Woohooo! 🎉 🎉 🎉

[morevnaproject]

@rodolforg Now eagerly waiting a PR with optimized (cobra-enabled) text layer. ^__^

[Fahad-Alsaidi]

I would like to thank @rodolforg & @morevnaproject & @ice0 for supporting RTL in synfig. You did a great job, thanks 🌹🌹

[rodolforg]

@rodolforg Now eagerly waiting a PR with optimized (cobra-enabled) text layer. ^__^

@morevnaproject
I'm working to make a clean version of what I wrote before HarfBuzz :) It still works, but I need to restore text alignment ("orientation" parameter).

I would like to thank @rodolforg & @morevnaproject & @ice0 for supporting RTL in synfig. You did a great job, thanks roserose

@Fahad-Alsaidi
Sadly we still don't provide a fallback-font for languages not supported by the chosen font.

[morevnaproject]

Sadly we still don't provide a fallback-font for languages not supported by the chosen font.

I've got an idea how we can handle situations with missing font.
What if in Synfig file the text layer will store not just string, but also a vectorized copy of the text (i.e. text converted to regions). So, if font is not available, Synfig still will be able to display correct text (although it will not be editable).
AFAIK Gimp does something similar for text layers by keeping rasterized copy of the layer in file. ^__^