[WIP] BiDi and shaping support for Font::draw/Font::get_string_size

COPYRIGHT.txt

@@ -168,6 +168,26 @@ Comment: glad
 Copyright: 2013-2018, David Herberth
 License: Expat
 
+Files: ./thirdparty/harfbuzz/
+Comment: HarfBuzz
+Copyright: 2010, 2011, 2012, Google, Inc.
+ 2012, Mozilla Foundation
+ 2011, Codethink Limited
+ 2008, 2010, Nokia Corporation and/or its subsidiary(-ies)
+ 2009, Keith Stribley
+ 2009, Martin Hosken and SIL International
+ 2007, Chris Wilson
+ 2006, Behdad Esfahbod
+ 2005, David Turner
+ 2004, 2007, 2008, 2009, 2010, Red Hat, Inc.
+ 1998-2004, David Turner and Werner Lemberg
+License: MIT-HarfBuzz
+
+Files: ./thirdparty/icu4c/
+Comment: International Components for Unicode
+Copyright: 1991-2018 Unicode, Inc.
+License: Unicode
+
 Files: ./thirdparty/jpeg_compressor/
 Comment: jpeg-compressor
 Copyright: 2012, Rich Geldreich
@@ -1630,6 +1650,44 @@ License: LGPL-2.1+SLE (libwebsockets)
  .
  That's all there is to it!
 
+License: MIT-HarfBuzz
+ HarfBuzz is licensed under the so-called "Old MIT" license.  Details follow.
+ For parts of HarfBuzz that are licensed under different licenses see individual
+ files names COPYING in subdirectories where applicable.
+ .
+ Copyright (c) 2010,2011,2012  Google, Inc.
+ Copyright (c) 2012  Mozilla Foundation
+ Copyright (c) 2011  Codethink Limited
+ Copyright (c) 2008,2010  Nokia Corporation and/or its subsidiary(-ies)
+ Copyright (c) 2009  Keith Stribley
+ Copyright (c) 2009  Martin Hosken and SIL International
+ Copyright (c) 2007  Chris Wilson
+ Copyright (c) 2006  Behdad Esfahbod
+ Copyright (c) 2005  David Turner
+ Copyright (c) 2004,2007,2008,2009,2010  Red Hat, Inc.
+ Copyright (c) 1998-2004  David Turner and Werner Lemberg
+ .
+ For full copyright notices consult the individual files in the package.
+ .
+ .
+ Permission is hereby granted, without written agreement and without
+ license or royalty fees, to use, copy, modify, and distribute this
+ software and its documentation for any purpose, provided that the
+ above copyright notice and the following two paragraphs appear in
+ all copies of this software.
+ .
+ IN NO EVENT SHALL THE COPYRIGHT HOLDER BE LIABLE TO ANY PARTY FOR
+ DIRECT, INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES
+ ARISING OUT OF THE USE OF THIS SOFTWARE AND ITS DOCUMENTATION, EVEN
+ IF THE COPYRIGHT HOLDER HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH
+ DAMAGE.
+ .
+ THE COPYRIGHT HOLDER SPECIFICALLY DISCLAIMS ANY WARRANTIES, INCLUDING,
+ BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
+ FITNESS FOR A PARTICULAR PURPOSE.  THE SOFTWARE PROVIDED HEREUNDER IS
+ ON AN "AS IS" BASIS, AND THE COPYRIGHT HOLDER HAS NO OBLIGATION TO
+ PROVIDE MAINTENANCE, SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS.
+
 License: MPL-2.0
  Mozilla Public License Version 2.0
  ==================================
@@ -2106,6 +2164,41 @@ License: RSA-MD
  These notices must be retained in any copies of any part of this
  documentation and/or software.
 
+License: Unicode
+ COPYRIGHT AND PERMISSION NOTICE (ICU 58 and later)
+ .
+ Copyright (C) 1991-2017 Unicode, Inc. All rights reserved.
+ Distributed under the Terms of Use in http://www.unicode.org/copyright.html
+ .
+ Permission is hereby granted, free of charge, to any person obtaining
+ a copy of the Unicode data files and any associated documentation
+ (the "Data Files") or Unicode software and any associated documentation
+ (the "Software") to deal in the Data Files or Software
+ without restriction, including without limitation the rights to use,
+ copy, modify, merge, publish, distribute, and/or sell copies of
+ the Data Files or Software, and to permit persons to whom the Data Files
+ or Software are furnished to do so, provided that either
+ (a) this copyright and permission notice appear with all copies
+ of the Data Files or Software, or
+ (b) this copyright and permission notice appear in associated
+ Documentation.
+ .
+ THE DATA FILES AND SOFTWARE ARE PROVIDED "AS IS", WITHOUT WARRANTY OF
+ ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE
+ WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ NONINFRINGEMENT OF THIRD PARTY RIGHTS.
+ IN NO EVENT SHALL THE COPYRIGHT HOLDER OR HOLDERS INCLUDED IN THIS
+ NOTICE BE LIABLE FOR ANY CLAIM, OR ANY SPECIAL INDIRECT OR CONSEQUENTIAL
+ DAMAGES, OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE,
+ DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER
+ TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
+ PERFORMANCE OF THE DATA FILES OR SOFTWARE.
+ .
+ Except as contained in this notice, the name of a copyright holder
+ shall not be used in advertising or otherwise to promote the sale,
+ use or other dealings in these Data Files or Software without prior
+ written authorization of the copyright holder.
+
 License: Zlib
  This software is provided 'as-is', without any express or implied
  warranty.  In no event will the authors be held liable for any damages

SConstruct

@@ -168,6 +168,8 @@ opts.Add(BoolVariable('builtin_bullet', "Use the built-in Bullet library", True)
 opts.Add(BoolVariable('builtin_certs', "Bundle default SSL certificates to be used if you don't specify an override in the project settings", True))
 opts.Add(BoolVariable('builtin_enet', "Use the built-in ENet library", True))
 opts.Add(BoolVariable('builtin_freetype', "Use the built-in FreeType library", True))
+opts.Add(BoolVariable('builtin_harfbuzz', "Use the built-in HarfBuzz library", True))
+opts.Add(BoolVariable('builtin_icu', "Use the built-in ICU library", True))
 opts.Add(BoolVariable('builtin_libogg', "Use the built-in libogg library", True))
 opts.Add(BoolVariable('builtin_libpng', "Use the built-in libpng library", True))
 opts.Add(BoolVariable('builtin_libtheora', "Use the built-in libtheora library", True))
@@ -186,6 +188,10 @@ opts.Add(BoolVariable('builtin_xatlas', "Use the built-in xatlas library", True)
 opts.Add(BoolVariable('builtin_zlib', "Use the built-in zlib library", True))
 opts.Add(BoolVariable('builtin_zstd', "Use the built-in Zstd library", True))
 
+# ICU/HarfBuzz options
+opts.Add(BoolVariable('use_text_shaping', "Use ICU and HarfBuzz for BiDi and text shaping", True))
+opts.Add(BoolVariable('use_staitc_icu_data', "Use statically-linked ICU break iterator data", False))
+
 # Compilation environment setup
 opts.Add("CXX", "C++ compiler")
 opts.Add("CC", "C compiler")

core/global_constants.cpp

@@ -108,6 +108,21 @@ void register_global_constants() {
 	BIND_GLOBAL_ENUM_CONSTANT(VERTICAL);
 	BIND_GLOBAL_ENUM_CONSTANT(HORIZONTAL);
 
+	BIND_GLOBAL_ENUM_CONSTANT(TEXT_DIRECTION_LTR);
+	BIND_GLOBAL_ENUM_CONSTANT(TEXT_DIRECTION_RTL);
+	BIND_GLOBAL_ENUM_CONSTANT(TEXT_DIRECTION_LOCALE);
+	BIND_GLOBAL_ENUM_CONSTANT(TEXT_DIRECTION_AUTO);
+
+	BIND_GLOBAL_ENUM_CONSTANT(TEXT_JUSTIFICATION_NONE);
+	BIND_GLOBAL_ENUM_CONSTANT(TEXT_JUSTIFICATION_KASHIDA_AND_WHITESPACE);
+	BIND_GLOBAL_ENUM_CONSTANT(TEXT_JUSTIFICATION_KASHIDA_ONLY);
+	BIND_GLOBAL_ENUM_CONSTANT(TEXT_JUSTIFICATION_WHITESPACE_ONLY);
+
+	BIND_GLOBAL_ENUM_CONSTANT(TEXT_BREAK_NONE);
+	BIND_GLOBAL_ENUM_CONSTANT(TEXT_BREAK_MANDATORY);
+	BIND_GLOBAL_ENUM_CONSTANT(TEXT_BREAK_MANDATORY_AND_WORD_BOUND);
+	BIND_GLOBAL_ENUM_CONSTANT(TEXT_BREAK_MANDATORY_AND_ANYWHERE);
+
 	BIND_GLOBAL_ENUM_CONSTANT(HALIGN_LEFT);
 	BIND_GLOBAL_ENUM_CONSTANT(HALIGN_CENTER);
 	BIND_GLOBAL_ENUM_CONSTANT(HALIGN_RIGHT);

core/math/math_defs.h

@@ -71,6 +71,30 @@ enum Orientation {
 	VERTICAL
 };
 
+enum TextDirection {
+
+	TEXT_DIRECTION_LTR,
+	TEXT_DIRECTION_RTL,
+	TEXT_DIRECTION_LOCALE,
+	TEXT_DIRECTION_AUTO
+};
+
+enum TextJustification {
+
+	TEXT_JUSTIFICATION_NONE,
+	TEXT_JUSTIFICATION_KASHIDA_AND_WHITESPACE,
+	TEXT_JUSTIFICATION_KASHIDA_ONLY,
+	TEXT_JUSTIFICATION_WHITESPACE_ONLY
+};
+
+enum TextBreak {
+
+	TEXT_BREAK_NONE,
+	TEXT_BREAK_MANDATORY,
+	TEXT_BREAK_MANDATORY_AND_WORD_BOUND,
+	TEXT_BREAK_MANDATORY_AND_ANYWHERE
+};
+
 enum HAlign {
 
 	HALIGN_LEFT,

core/method_bind.h

@@ -182,6 +182,9 @@ VARIANT_ENUM_CAST(Error);
 VARIANT_ENUM_CAST(Margin);
 VARIANT_ENUM_CAST(Corner);
 VARIANT_ENUM_CAST(Orientation);
+VARIANT_ENUM_CAST(TextDirection);
+VARIANT_ENUM_CAST(TextJustification);
+VARIANT_ENUM_CAST(TextBreak);
 VARIANT_ENUM_CAST(HAlign);
 VARIANT_ENUM_CAST(VAlign);
 VARIANT_ENUM_CAST(PropertyHint);

core/ustring.cpp

@@ -1391,19 +1391,28 @@ bool String::parse_utf8(const char *p_utf8, int p_len) {
 				uint8_t c = *ptrtmp;
 
 				/* Determine the number of characters in sequence */
-				if ((c & 0x80) == 0)
+				if ((c & 0x80) == 0) {
 					skip = 0;
-				else if ((c & 0xE0) == 0xC0)
+				} else if ((c & 0xE0) == 0xC0) {
 					skip = 1;
-				else if ((c & 0xF0) == 0xE0)
+				} else if ((c & 0xF0) == 0xE0) {
 					skip = 2;
-				else if ((c & 0xF8) == 0xF0)
+				} else if ((c & 0xF8) == 0xF0) {
 					skip = 3;
-				else if ((c & 0xFC) == 0xF8)
+					if (sizeof(wchar_t) == 2) {
+						str_size++; //charcode > 0xFFFF, surrogate pair on windows
+					}
+				} else if ((c & 0xFC) == 0xF8) {
 					skip = 4;
-				else if ((c & 0xFE) == 0xFC)
+					if (sizeof(wchar_t) == 2) {
+						str_size++; //charcode > 0xFFFF, surrogate pair on windows
+					}
+				} else if ((c & 0xFE) == 0xFC) {
 					skip = 5;
-				else {
+					if (sizeof(wchar_t) == 2) {
+						str_size++; //charcode > 0xFFFF, surrogate pair on windows
+					}
+				} else {
 					_UNICERROR("invalid skip");
 					return true; //invalid utf8
 				}
@@ -1500,10 +1509,13 @@ bool String::parse_utf8(const char *p_utf8, int p_len) {
 
 		//printf("char %i, len %i\n",unichar,len);
 		if (sizeof(wchar_t) == 2 && unichar > 0xFFFF) {
-			unichar = ' '; //too long for windows
+			//too long for windows - encode as surrogate pairs
+			*(dst++) = ((unichar - 0x10000) >> 10) + 0xD800;
+			*(dst++) = ((unichar - 0x10000) & 0x3FFUL) + 0xDC00;
+		} else {
+			*(dst++) = unichar;
 		}
 
-		*(dst++) = unichar;
 		cstr_size -= len;
 		p_utf8 += len;
 	}

core/ustring.h

@@ -28,8 +28,8 @@
 /* SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.                */
 /*************************************************************************/
 
-#ifndef USTRING_H
-#define USTRING_H
+#ifndef CORE_USTRING_H //USTRING_H conflicts with ICU's <unicode/ustring.h>
+#define CORE_USTRING_H
 
 #include "core/array.h"
 #include "core/cowdata.h"
@@ -412,4 +412,4 @@ String RTR(const String &);
 bool is_symbol(CharType c);
 bool select_word(const String &p_s, int p_col, int &r_beg, int &r_end);
 
-#endif // USTRING_H
+#endif // CORE_USTRING_H

doc/classes/@GlobalScope.xml

@@ -105,6 +105,42 @@
 		<constant name="HORIZONTAL" value="0" enum="Orientation">
 			General horizontal alignment, used usually for [Separator], [ScrollBar], [Slider], etc.
 		</constant>
+		<constant name="TEXT_DIRECTION_LTR" value="0" enum="TextDirection">
+			Left-to-right text writing direction.
+		</constant>
+		<constant name="TEXT_DIRECTION_RTL" value="1" enum="TextDirection">
+			Right-to-left text writing direction.
+		</constant>
+		<constant name="TEXT_DIRECTION_LOCALE" value="2" enum="TextDirection">
+			Text writing direction is derived from the locale's script according to the CLDR metadata.
+		</constant>
+		<constant name="TEXT_DIRECTION_AUTO" value="3" enum="TextDirection">
+			Text writing direction is derived from the first character in the string with BiDi class L, R, or AL or locale's script if text is not strongly directional.
+		</constant>
+		<constant name="TEXT_JUSTIFICATION_NONE" value="0" enum="TextJustification">
+			No text justification.
+		</constant>
+		<constant name="TEXT_JUSTIFICATION_KASHIDA_AND_WHITESPACE" value="1" enum="TextJustification">
+			Use kashida and whitespace elongation to justify text.
+		</constant>
+		<constant name="TEXT_JUSTIFICATION_KASHIDA_ONLY" value="2" enum="TextJustification">
+			Use kashida elongation to justify text.
+		</constant>
+		<constant name="TEXT_JUSTIFICATION_WHITESPACE_ONLY" value="3" enum="TextJustification">
+			Use whitespace elongation to justify text.
+		</constant>
+		<constant name="TEXT_BREAK_NONE" value="0" enum="TextBreak">
+			No line breaking.
+		</constant>
+		<constant name="TEXT_BREAK_MANDATORY" value="1" enum="TextBreak">
+			Break lines only at mandatory break points.
+		</constant>
+		<constant name="TEXT_BREAK_MANDATORY_AND_WORD_BOUND" value="2" enum="TextBreak">
+			Break lines at mandatory break points and word boundaries.
+		</constant>
+		<constant name="TEXT_BREAK_MANDATORY_AND_ANYWHERE" value="3" enum="TextBreak">
+			Break lines at mandatory break points and grapheme cluster boundaries.
+		</constant>
 		<constant name="HALIGN_LEFT" value="0" enum="HAlign">
 			Horizontal left alignment, usually for text-derived classes.
 		</constant>

doc/classes/CanvasItem.xml

@@ -144,6 +144,27 @@
 			<description>
 			</description>
 		</method>
+		<method name="draw_paragraph">
+			<return type="void">
+			</return>
+			<argument index="0" name="font" type="Font">
+			</argument>
+			<argument index="1" name="position" type="Vector2">
+			</argument>
+			<argument index="2" name="text" type="String">
+			</argument>
+			<argument index="3" name="modulate" type="Color" default="Color( 1, 1, 1, 1 )">
+			</argument>
+			<argument index="4" name="clip_w" type="int" default="-1">
+			</argument>
+			<argument index="5" name="break_flags" type="int" enum="TextBreak" default="2">
+			</argument>
+			<argument index="6" name="just_flags" type="int" enum="TextJustification" default="1">
+			</argument>
+			<description>
+				Draws a paragraph of text using a custom font.
+			</description>
+		</method>
 		<method name="draw_polygon">
 			<return type="void">
 			</return>

doc/classes/ICUData.xml

@@ -0,0 +1,16 @@
+<?xml version="1.0" encoding="UTF-8" ?>
+<class name="ICUData" inherits="Resource" category="Core" version="3.1">
+	<brief_description>
+		ICD break iterator data.
+	</brief_description>
+	<description>
+	</description>
+	<tutorials>
+	</tutorials>
+	<demos>
+	</demos>
+	<methods>
+	</methods>
+	<constants>
+	</constants>
+</class>

doc/classes/Label.xml

@@ -23,7 +23,7 @@
 			<return type="int">
 			</return>
 			<description>
-				Returns the font size in pixels.
+				Returns the line height in pixels.
 			</description>
 		</method>
 		<method name="get_total_character_count" qualifiers="const">
@@ -51,18 +51,28 @@
 		<member name="clip_text" type="bool" setter="set_clip_text" getter="is_clipping_text">
 			If [code]true[/code], the Label only shows the text that fits inside its bounding rectangle. It also lets you scale the node down freely.
 		</member>
+		<member name="language" type="String" setter="set_language" getter="get_language">
+			Language code for line-breaking and text shaping algorithms.
+		</member>
 		<member name="lines_skipped" type="int" setter="set_lines_skipped" getter="get_lines_skipped">
 			The node ignores the first [code]lines_skipped[/code] lines before it starts to display text.
 		</member>
 		<member name="max_lines_visible" type="int" setter="set_max_lines_visible" getter="get_max_lines_visible">
 			Limits the lines of text the node shows on screen.
 		</member>
+		<member name="ot_features" type="String" setter="set_ot_features" getter="get_ot_features">
+			Comma separated list of OpenType feature tags.
+			More information can be found at https://docs.microsoft.com/en-us/typography/opentype/spec/featuretags
+		</member>
 		<member name="percent_visible" type="float" setter="set_percent_visible" getter="get_percent_visible">
 			Limits the count of visible characters. If you set [code]percent_visible[/code] to 50, only up to half of the text's characters will display on screen. Useful to animate the text in a dialog box.
 		</member>
 		<member name="text" type="String" setter="set_text" getter="get_text">
 			The text to display on screen.
 		</member>
+		<member name="text_direction" type="int" setter="set_text_direction" getter="get_text_direction" enum="TextDirection">
+			Base text writing direction.
+		</member>
 		<member name="uppercase" type="bool" setter="set_uppercase" getter="is_uppercase">
 			If [code]true[/code], all the text displays as UPPERCASE.
 		</member>