Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cyrillic text issue after updating to haxe 4.0.0-rc.5 (in openfl TextField) #2264

Closed
www0z0k opened this issue Sep 22, 2019 · 32 comments · Fixed by #2300
Closed

Cyrillic text issue after updating to haxe 4.0.0-rc.5 (in openfl TextField) #2264

www0z0k opened this issue Sep 22, 2019 · 32 comments · Fixed by #2300

Comments

@www0z0k
Copy link

@www0z0k www0z0k commented Sep 22, 2019

After upgrading haxe I get correct traces of Cyrillic strings’ lengths and values, but the TextField itself looks like:
Screenshot 2019-09-21 at 20 21 43

However tracing the textField.text gives a correct result in the console.

This is on mac/ios targets. toUpperCase() and length work as expected for Cyrillic strings, but the text looks like:
Screenshot 2019-09-22 at 03 15 47

On neko target toUpperCase() and length don't work correctly, but the text is displayed ok:
Screenshot 2019-09-22 at 03 17 14

Here is the full code to reproduce it:

	var _body:TextField = new TextField();
        addChild(_body);
	_body.text = 'фыва йцук';

Should I downgrade back to 3.4.7 version or there might be another way around?

@MSGhero

This comment has been minimized.

Copy link
Contributor

@MSGhero MSGhero commented Sep 22, 2019

I need to look into it

@MSGhero

This comment has been minimized.

Copy link
Contributor

@MSGhero MSGhero commented Sep 23, 2019

@www0z0k what versions of OpenFL, Lime, and hxcpp are you on?

@www0z0k

This comment has been minimized.

Copy link
Author

@www0z0k www0z0k commented Sep 23, 2019

@MSGhero
OpenFL Command-Line Tools (8.9.5-LoqcFr)
Lime Command-Line Tools (7.6.3)
hxcpp: [4.0.52]

@Txori

This comment has been minimized.

Copy link

@Txori Txori commented Sep 23, 2019

Hi. I think I might have the same issue regarding Japanese characters. It works on Flash but not on Android: http://forum.haxeflixel.com/topic/934/display-japanese-font-on-android-target/

flixel: [4.6.3]
openfl: [8.9.4]
lime: [7.6.2]
hxcpp: [4.0.52]

@www0z0k , type "haxelib list" in cmd

@MSGhero

This comment has been minimized.

Copy link
Contributor

@MSGhero MSGhero commented Sep 23, 2019

I may not have time for a few days to look into this, but can one of you go here (https://github.com/openfl/openfl/blob/develop/src/openfl/_internal/text/UTF8String.hx) and change line 6 to = UnicodeString instead of normal String?

@www0z0k

This comment has been minimized.

Copy link
Author

@www0z0k www0z0k commented Sep 23, 2019

@MSGhero where can I find openfl internals locally?

@Txori

This comment has been minimized.

Copy link

@Txori Txori commented Sep 23, 2019

@www0z0k
c:/HaxeToolKit/haxe/lib/openfl/8,9,4/src/openfl/_internal/text/UTF8String.hx

But I just tried and it's not working.

@www0z0k

This comment has been minimized.

Copy link
Author

@www0z0k www0z0k commented Sep 23, 2019

@Txori I wonder where is it located on osx
Are you sure that it takes the source files each time you run and nothing gets precompiled?

@Txori

This comment has been minimized.

Copy link

@Txori Txori commented Sep 23, 2019

Sorry I know almost nothing about osx... Isn't it a bit like linux? Maybe something like HD/usr/lib/haxe ?

To verify that it takes modification of the source files, I suppressed .\export\android\obj\src\openfl_internal\text folder and it gets regenerated.

@MSGhero

This comment has been minimized.

Copy link
Contributor

@MSGhero MSGhero commented Sep 24, 2019

I'm getting compiler errors from within hxcpp, so I can't even test this. The text looks fine on HTML5 target, and if you're saying it also looks fine on Neko, then this might actually be a hxcpp issue?

I'll try to reinstall haxe and stuff tomorrow and see if that helps. Otherwise, I'm only guessing at what the real problem is here.

@www0z0k

This comment has been minimized.

Copy link
Author

@www0z0k www0z0k commented Sep 24, 2019

@MSGhero
please check the displayed text along with:

trace('йцу'.length);
trace('йцу'.charAt(0));

It shows length 6 for actual 3 letters and charAt ? on platforms which display text correctly and shows correct traces on platforms that display incorrect text, which makes me assume it has something to do with the new full unicode support in haxe 4

@MSGhero

This comment has been minimized.

Copy link
Contributor

@MSGhero MSGhero commented Sep 24, 2019

I cannot compile when my code has non-ASCII characters in it. Like I get compiler errors from hxcpp based on the contents of my strings.

If you can try this on HTML5 and see what the lengths are, that would be helpful.

@www0z0k

This comment has been minimized.

Copy link
Author

@www0z0k www0z0k commented Sep 24, 2019

@MSGhero this is not what I expected, the only platform I tested with 100% correct behaviour:

var _body:TextField = new TextField();
addChild(_body);
_body.width = 500;
_body.text = 'фыва, length = ' + 'фыва'.length + ' at 0 = "' + 'фыва'.charAt(0) + '"';

shows:
Screenshot 2019-09-24 at 16 42 13

Tested on osx if it matters

@Gama11

This comment has been minimized.

Copy link
Member

@Gama11 Gama11 commented Sep 25, 2019

@MSGhero I recall some hxcpp issue where that happens when your Visual Studio version / C++ compiler is too old.

@MSGhero

This comment has been minimized.

Copy link
Contributor

@MSGhero MSGhero commented Sep 25, 2019

Ohhhh interesting. Thanks @Gama11 I’ll try it out

@MSGhero

This comment has been minimized.

Copy link
Contributor

@MSGhero MSGhero commented Sep 27, 2019

@jgranick this issue and #2256 are caused by these lines on rc4 and rc5: https://github.com/openfl/openfl/blob/develop/src/openfl/_internal/text/TextLayout.hx#L143-L148

This is a Lime/Harfbuzz interface, so you probably have a much better understanding of it than I do. Non-UTF8 codepoints look like they're being wrapped or truncated.

@Txori

This comment has been minimized.

Copy link

@Txori Txori commented Oct 2, 2019

When I revert back openfl/src/openfl/_internal/text/TextLayout.hx to b151041, every thing is working perfectly. What caused the roll back?

@filipp8

This comment has been minimized.

Copy link

@filipp8 filipp8 commented Oct 7, 2019

I have the same problem with japanese text on rc3

@Txori

This comment has been minimized.

Copy link

@Txori Txori commented Oct 7, 2019

Just un-comment those lines in openfl/src/openfl/_internal/text/TextLayout.hx

// #if ((haxe_ver < "4.0.0") || neko || mac || linux || hl)
__hbBuffer.addUTF8(text, 0, -1);
// #else
// for (i in 0...text.length)
// __hbBuffer.add(text.charCodeAt(i), i);
// #end

It should work, unless you try a Windows build: 8dcb6ba

@rainyt

This comment has been minimized.

Copy link
Contributor

@rainyt rainyt commented Oct 9, 2019

8dcb6ba#commitcomment-35418723
Can see if this helps.

@MSGhero

This comment has been minimized.

Copy link
Contributor

@MSGhero MSGhero commented Oct 9, 2019

I can try it out tonight, thanks!

@MSGhero

This comment has been minimized.

Copy link
Contributor

@MSGhero MSGhero commented Oct 11, 2019

@rainyt uh yeah it works. That's way simpler than anything I was going to try. I want to run this through the unit tests first to be sure, but wow.

@jgranick

This comment has been minimized.

Copy link
Member

@jgranick jgranick commented Oct 11, 2019

Is there any way to set the type to unicode and then set the text in one call instead of character by character?

@MSGhero

This comment has been minimized.

Copy link
Contributor

@MSGhero MSGhero commented Oct 11, 2019

@jgranick I checked the HB source, and all of the addUTF functions set the content type to UNICODE. So it likely comes down to fitting addUTF16’s API and then making sure that’s appropriate for all character types.

@MSGhero

This comment has been minimized.

Copy link
Contributor

@MSGhero MSGhero commented Oct 16, 2019

I've been too busy to test things, but how about this?
__hbBuffer.addUTF16(DataPointer.fromBytes(Bytes.ofString(text, RawNative)), text.length, 0, -1);

It's basically what Joshua said before, but I didn't see a cleaner way of writing it

@www0z0k

This comment has been minimized.

Copy link
Author

@www0z0k www0z0k commented Oct 23, 2019

any chance of a fix in some nightly build or something?

@MSGhero

This comment has been minimized.

Copy link
Contributor

@MSGhero MSGhero commented Oct 23, 2019

I’m close, I just don’t have much free time for coding like I used to. Sorry about that.

The proposed solution where the add goes into a for loop doesn’t account for a lot of other work the addUTF functions perform, so it’s not a long-term fix that can be committed.

@stnguyen

This comment has been minimized.

Copy link

@stnguyen stnguyen commented Nov 4, 2019

anybody has a fix for it?

@ngrebenshikov

This comment has been minimized.

Copy link
Contributor

@ngrebenshikov ngrebenshikov commented Nov 6, 2019

My env: target iOS, lime (git), hxcpp (git), haxe 4.0.1
It's strange but the code @MSGhero mentioned (__hbBuffer.addUTF16(DataPointer.fromBytes(Bytes.ofString(text, RawNative)), text.length, 0, -1);) doesn't work for me.

But uncommenting previous fix helped me after setting the contentType to the unicode. So my code inside TextLayout.hx

__hbBuffer.contentType = lime.text.harfbuzz.HBBufferContentType.UNICODE;
for (i in 0...text.length) {
    __hbBuffer.add(text.charCodeAt(i), i);
}
@stnguyen

This comment has been minimized.

Copy link

@stnguyen stnguyen commented Nov 6, 2019

@ngrebenshikov @rainyt do you use haxe 4 String, or another string class?
Would you pls post a fully working example, with note about versions of haxe, openfl, lime, and platform?
Thank you so much!

@ngrebenshikov

This comment has been minimized.

Copy link
Contributor

@ngrebenshikov ngrebenshikov commented Nov 7, 2019

@stnguyen I can only share may environment. I don't have a small working example to share.
Haxe - 4.0.1
String - Haxe 4 String
Hxcpp - latest Github version
Lime - latest GitHub version
OpenFL - 8.9.5
Build platform - OSX Catalina
Target platform - iOS

@MSGhero

This comment has been minimized.

Copy link
Contributor

@MSGhero MSGhero commented Dec 3, 2019

Ok. I apologize for the long delay. I'm not very familiar with cpp, I've moved, I'm busy, etc, but I got it. Below is on the latest Haxe with a 2 line modification to TextLayout and DataPointer. Joshua pointed out before that I should test this on several C targets just to be sure nothing breaks across the board.

image
The mods are:

// __hbBuffer.addUTF8(text, 0, -1);
__hbBuffer.addUTF16(untyped __cpp__('(uintptr_t){0}', text.wc_str()), text.length, 0, -1);

and giving DataPointer a from DataPointerType in the abstract declaration.

What I was doing in the past was getting the address of text which is a Haxe object which means nothing to the text shaping engine. wc_str() gives you the UTF16 pointer of the string, which means nothing to Haxe and specifically nothing to the DataPointer class. The above line is a direct cpp call that I found in DataPointer that has to be called from within TextLayout to make use of the UTF16 pointer, which then enables DataPointer to be used as intended.

PR soon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

9 participants
You can’t perform that action at this time.