Workaround (fix?) for #603 encoding issue during line wrap

[johnnyshields]

This PR contains @stayhero's test case which reproduces #603 and a workaround. I'm not sure this is the best way to solve the issue, would be great if someone more familiar with Prawn can help.

This issue is affecting many of my clients as we're generating non-English PDFs.

[practicingruby]

I'm fairly sure that this is a bad behavior rather than a bug. The "built-in" fonts supported by PDF are only going to work for glyphs in the Windows 1252 character set, which is why we recommend against using them for any internationalized text.

If you call font("some.font_ttf") with a font that supports the glyphs you need, everything should work as expected. There should be no need to use fallback fonts at all... they're really only meant to be used for things like CJK character sets where you want one font for your chinese glyphs and one font for your romanized glyphs.

But even if you DO use fallback fonts, you can probably avoid this problem by just using a TTF font rather than the built-in fonts for your default.

I haven't had a chance to confirm that, but it's my suspicion. There is probably something weird going on with encodings for the default fonts too, which would cause this problem.

[practicingruby]

It probably wasn't clear from my response, but to clarify I definitely would like to fix this issue, whether it's a bug or just a poorly defined behavior. I don't think the solution is to to transcode missing glyphs as you're doing in this pull request, because it should not be necessary.

If this fixes the problem, I think it does so at the surface level. I want to either figure out where the root cause of the issue happens, or figure out whether this would actually be a safe way to solve this at the surface level.

[practicingruby]

Here's an example of how to avoid this problem. Fallback fonts can be used if you want to use multiple TTF fonts that each contain different glyphs, but if you're just looking for internationalized text any TTF font that supports the characters you need will work fine:

@pdf = Prawn::Document.new

@pdf.font_families.update("dejavu" => { :normal => "path/to/DejaVuSans.ttf" })
@pdf.font("dejavu")

@pdf.bounding_box([1, 1], :width => 90, :height => 50) do
  broken_text = " Sample Text\nSAMPLE SAMPLE SAMPLEoddělení ZMĚN\nSAMPLE"
  @pdf.text broken_text, :overflow => :shrink_to_fit
end

[johnnyshields]

@sandal to summarize:

1. For international char-based docs, it recommend to set font and to not set fallback_font. Doing so avoids this issue.
2. fallback_font is only intended in cases where you intend to use the PDF built-in fonts as a primary.
3. regardless of 1 and 2 we should still fix this issue.

correct?

[practicingruby]

1. For international char-based docs, it recommend to set font and to not set fallback_font. Doing so avoids this issue.

It's not necessary to use fallback_fonts for internationalized text, you only need it when you want to use multiple different fonts that have different glyphs in each (i.e. Roman glyphs in one font and Chinese glyphs in another). You can use or not use fallback_fonts, as long as you make sure that you're NOT using the PDF built-in fonts, and things will work correctly.

Or to really sum it up: Prawn is meant to be internationalization-friendly by default, you just need to provide UTF-8 aware fonts for it to use. fallback_fonts is an advanced, special-case feature. I have a ticket open to document this more clearly, because it's a common misunderstanding.

2. fallback_font is only intended in cases where you intend to use the PDF built-in fonts as a primary.

fallback_font is probably not safe for use in its current form with PDF built-in fonts, and so no, this is not the recommendation. See above for more detail. But if you do use fallback_fonts, you're using it because you need internationalization, and if you need internationalization, always use TTF files you provide rather than relying on Built-in fonts. Windows-1252 might work, but it's severely limited.

3. regardless of 1 and 2 we should still fix this issue.

Agree! But the fix may not be "make the failing code sample work as-is". The fix will be "define the correct behavior, and make the APIs work consistently based on that definition". We need to find the root of the problem before figuring out the answer to that.

[johnnyshields]

I've verified that pdf.font fixes the issue

[practicingruby]

Closing this PR because I don't think it's going to be directly merged, but #729 references it so that we can come back and read the discussion when we get around to properly defining fallback font behavior.

[johnnyshields]

No issue since I started using pdf.font (thanks for that!) One request--can we make sure that the behavior is documented?

[practicingruby]

@johnnyshields: Defining the behavior implies documenting it at the API doc / manual level, so yeah, it definitely will be.