Default_Ignorable_Code_Point in hyphenation?

[sommerluk]

Some characters in Unicode have the property Default_Ignorable_Code_Point. This includes characters like the ZWNJ zero width non-joiner U+200C. Most format characters have this property. With CTL and OpenType support available in Scribus, the usage of these characters will increase.

These characters have yet a special treatment in spell checking since #215. The question is how to treat these characters in hyphenation. Currently, the automatic hyphenation does not work as expected when characters like ZWNJ are present. Example: The German word “Auflage” should be hyphenated “Auf-la-ge”, and this works correctly in Scribus when no ZWNJ is there. But when there is a ZWNJ between “Auf” and “lage” then the first hyphenation point is not found. It’s only “Aufla-ge”.

Possible solution:

• Do nothing. (At least the current situation does not create much harm.)
• Support Default_Ignorable_Code_Point characters as expected. Might have more overhead (mapping of character positions) but might be the best solution. At scribus/text/specialchars.cpp there is bool specialChars::isIgnorableCodePoint(uint ch) that might be usefull.

[sommerluk]

Might be related to #145