[Apple Bug] TextKit's calculatedSize is not matching NSAttributedString for CJK languages.

[binchensjtu]

I found that the height of calculatedSize in ASTextNode is incorrect for Chinese and Japanese.
The test code snippet as below:

NSString *text = @"你好";//Some Chinese characters
NSDictionary *attributes = @{NSFontAttributeName: [UIFont systemFontOfSize:17.0]};
ASTextNode *textNode = [[ASTextNode alloc] init];
textNode.attributedString = [[NSAttributedString alloc] initWithString:text attributes:attributes];

CGSize textSize1 = [textNode measure:constrainedSize];
CGSize textSize2 = [text sizeWithAttributes:attributes];

NSLog(@"textSize1: %@", NSStringFromCGSize(textSize1));
NSLog(@"textSize2: %@", NSStringFromCGSize(textSize2));

The output is:

textSize1: {34, 17}
textSize2: {34, 20.287109375}

Pure Chinese characters will lead to the issue. The bounds of text node is tight. The mix of Chinese and alphabets is correct.
The same problem to pure Japanese characters. Haven't test more languages.

The screenshot of pure Chinese character text node and mix character text node.

[appleguy]

@binchensjtu Thank you for reporting this! Support for the languages of the world has been a top motivator for me since the first prototypes of AsyncDisplayKit, because many languages take more computational effort to perform layout as well as rasterization than the English-biased developers frequently building iOS frameworks. A few questions:

• Have you replicated this test with UILabel, UITextField, or UITextView? How about the older NSString-based sizing methods that pass in a font directly?
• Have you tried using CoreText or TextKit directly, which are what ASTextNode is built on?
• Do you have a set of recommendations or test cases for how I could verify what behavior is correct / expected? Do you feel that the sizing behavior of NSAttributedString itself is consistently correct?

@ocrickard - I promise not to invoke you on every text issue we run across (you may be surprised that I've fixed several recently and am becoming increasingly comfortable with the rendering code that was previously opaque to me). However, I have to ask: has anything like this (let's say CJK languages in general) been studied, or issues reported, with ComponentKit?

@rcancro, @nguyenhuy - I know you guys have at least some additional expertise here beyond my own, and you may know native CJK speakers who have used TextKit / Core Text in detail and could help us audit the behavior. Let me know.

[ocrickard]

Hmm, clipping is always an issue sadly. We created a size estimation method for ASDK that I brought over to ComponentKit and I don't think I touched it in the refactor, so it would be interesting to know if this reproed before the refactor was ported.

There will be differences between NSStringDrawing categories and TextKit. NSStringDrawing internally still uses an RTF engine that I believe is fairly different from TextKit, and is substantially slower for complex strings. If we can try TextKit directly as you're suggesting that would be great.

[ocrickard]

As for if we've had issues reported, I think I saw one fly by recently, so we likely have the same woes. We marked it up to an apple bug until we saw evidence to the contrary.

[binchensjtu]

@appleguy Thanks for fast reply in your holiday! I never notice this problem when using basic apple text widget. I test it today. Including CoreText.

  NSString *text = @"你好";
  UIFont *font = [UIFont systemFontOfSize:17.0];
  NSDictionary *attributes = @{NSFontAttributeName: font };

  NSAttributedString *attributedString = [[NSAttributedString alloc] initWithString:text attributes:attributes];

  _label = [[UILabel alloc] init];
  _label.font = font;
  _label.text = text;
  NSLog(@"label plain text size %@", NSStringFromCGSize([_label sizeThatFits:CGSizeMake(FLT_MAX, FLT_MAX)]));

  _textView = [[UITextView alloc] init];
  _textView.font = font;
  _textView.text = text;
  _textView.textContainer.lineFragmentPadding = 0;
  _textView.textContainerInset = UIEdgeInsetsZero;
  NSLog(@"text view plain text size %@", NSStringFromCGSize([_textView sizeThatFits:CGSizeMake(FLT_MAX, FLT_MAX)]));

  _textField = [[UITextField alloc] init];
  _textField.font = font;
  _textField.text = text;
  NSLog(@"text field plain text size %@", NSStringFromCGSize([_textField sizeThatFits:CGSizeMake(FLT_MAX, FLT_MAX)]));

  CTFramesetterRef framesetter = CTFramesetterCreateWithAttributedString((CFAttributedStringRef)attributedString);
  CGSize coreTextSize = CTFramesetterSuggestFrameSizeWithConstraints(framesetter, CFRangeMake(0,0), nil, CGSizeMake(CGFLOAT_MAX, CGFLOAT_MAX), nil);
  NSLog(@"core text size %@", NSStringFromCGSize(coreTextSize));

output is

label plain text size {34, 20.5}
text view plain text size {34, 20.5}
text field plain text size {38, 21}
core text size {34, 17}

I believe some padding in text filed so the width is larger. You're right. The CoreText result is the same as ASTextNode. I didn't deep dive in basic text widgets. Do you know the calculation detail of these basic widgets?

Actually I would like to have stable calculation result while using pure CJK and English-CJK-Mix. Because these two situation are common in non-English-language market. The stable height make UI designer happy 😃 . What I want is the sizing behavior of NSAttributedString. Do you guys have any idea to workaround this or "fix" this?

[ocrickard]

Yeah, this is a long-standing problem. I believe that Apple's text sizing methods in NSStringDrawing categories use RTF string drawing engine I mentioned and/or add padding. It may be possible to come up with a more stable fix for this if someone sat down for a couple days and messed with the sizing routines we have... We built those from trial and error over about two weeks in the development of Paper and we didn't revisit them much since.

[ocrickard]

Alternatively a "fix" for you could be to add padding to your nodes to avoid the clip until we can get this fixed.

[binchensjtu]

Thanks @ocrickard
I believe this problem is some kind of relative to font. Since iOS9, the system default font for Chinese is PingFang. I explicit use [UIFont fontWithName:@"Helvetica" size:17.0]. The calculation of pure CJK and English-CJK-Mix are the same using ASTextNode and CoreText in iOS9.2. Even though it is still different from apple's NSStringDrawing categories.

[ocrickard]

Oh yeah definitely there's a mismatch somewhere between the font glyph tables and the TextKit/CoreText drawing engine, we've seen this in different manifestations for a long time. Here's where we calculate the size of the context, it uses pure TextKit sizing methods, feel free to play with it and see if you can improve it:

https://github.com/facebook/AsyncDisplayKit/blob/master/AsyncDisplayKit/TextKit/ASTextKitRenderer.mm#L95

[ocrickard]

I believe @bcunning recently dealt with clipping in ASTextNode for arabic recently. Do you have any insights or suggestions?

[ocrickard]

Also @appleguy btw while I'm looking at some of this code https://github.com/facebook/AsyncDisplayKit/blob/master/AsyncDisplayKit/ASTextNode.mm#L418 is incorrect. @bcunning and I recently ran into it. It should just be filling the clipping bounds of the CGContext. So:

CGContextFillRect(context, CGContextGetClipBoundingBox(context));

[appleguy]

@ocrickard thanks for bringing in the know-how! :-D I'll fix that rect fill issue.

@binchensjtu Have you tried AsyncDisplayKit's behavior on 1.9.2 as well as the more recent versions? I'm just curious if any of the changes to the text rendering engine that landed in 1.9.3 would affect this, but it does look like the story may end more sadly...if the bug is ultimately Apple's, it will be harder for us to work around in a way that doesn't cause problems if they fix the bug. However, I'm very interested in considering any ideas for a workaround, such as comparing sizes to known-correct ones before adding padding. We'd need to be able to very efficiently detect the cases that need adjustment, though, and I fear that might be impossible.