Convert `uniXXXX` glyph names to proper ones when building the `charCodeToGlyphId` map for TrueType fonts (bug 1132849, issue 6893, issue 6894) #7069

Snuffleupagus · 2016-03-07T21:03:36Z

Some PDF generators, e.g. Scribus PDF, use improper uniXXXX glyph names which breaks the glyph mapping. We can avoid this by converting them to "standard" glyph names instead.

Fixes https://bugzilla.mozilla.org/show_bug.cgi?id=1132849.
Fixes #6893.
Fixes #6894.

/cc @timvandermeij

timvandermeij · 2016-03-07T21:53:50Z

Awesome work! I can confirm that this fixes both issues I filed. The text is also more crisp and more like how other PDF viewers render the files (what I see is best seen in the reference images on Linux from above). The original and the reduced test case from the Bugzilla issue already seem to be fixed for me with the current master (at least on Arch Linux when comparing PDF.js with Okular), but nevertheless it's good to include it as a reduced test case.

@brendandahl Could you please review this?

Snuffleupagus · 2016-03-07T22:13:44Z

The original and the reduced test case from the Bugzilla issue already seem to be fixed for me with the current master (on Arch Linux when comparing PDF.js with Okular), but nevertheless it's good to include it as a reduced test case.

For me there're slight differences with 1132849.pdf on Windows (with HWA on), which is why I included it.

Although there's now a couple of somewhat similar test-cases, which might be unnecessary. The already existing issue6889.pdf could probably be replaced with issue6894.pdf in this patch, since those two appear to be using the exact same font (but the new one covers the font-weight issues as well).

However, I didn't want to remove any existing tests in this patch.

Snuffleupagus · 2016-03-08T11:02:28Z

I've update the patch slightly, to only attempt this conversion for the this.differences case. Since otherwise we just read from one of the standard encodings, which we know don't suffer from this issue.

/botio test

brendandahl · 2016-03-08T18:21:51Z

This is looking good. I was curious how other PDF viewers handle this and found https://pdfium.googlesource.com/pdfium/+/master/core/src/fxge/freetype/fx_freetype.cpp#56 . Seems like it would be a good idea to make a general function for getting the unicode from glyph name. There function handles names like 'u123456' and a few other cases. Now I wonder if we have any broken pdfs with names like the above.

Snuffleupagus · 2016-03-08T19:31:50Z

Seems like it would be a good idea to make a general function for getting the unicode from glyph name.

Sure, that sounds fine! The only one I can remember seeing in practice is uniXXXX, but I suppose it can't hurt to support the uXXXX/uXXXXXX case as well.

@brendandahl How closely do you want that function to mimic what the FreeType code does?
Should we enforce only upper-case hexadecimal chars, to prevent accidentally matching things that are not actually hex numbers?

Snuffleupagus · 2016-03-09T14:04:54Z

I've added a getUnicodeForGlyph helper function, similarly to the FreeType one, which is used in various places to do the glyphName -> Unicode conversion.
I kept the recoverGlyphName, which is using getUnicodeForGlyph internally, since we may need both the standard/non-standard glyph names when creating the charCodeToGlyphId map for TrueType fonts.

@brendandahl Is this what you had in mind?

/botio-linux preview

pdfjsbot · 2016-03-09T14:04:56Z

From: Bot.io (Linux)

Received

Command cmd_preview from @Snuffleupagus received. Current queue size: 0

Live output at: http://107.21.233.14:8877/b0e4ee870623fb9/output.txt

pdfjsbot · 2016-03-09T14:05:55Z

From: Bot.io (Linux)

Success

Full output at http://107.21.233.14:8877/b0e4ee870623fb9/output.txt

Total script time: 0.99 mins

Published

brendandahl · 2016-03-09T18:11:24Z

src/core/unicode.js

+        hexStr = name.substr(3);
+      } else if (nameLen >= 5 && nameLen <= 7) { // 'uXXXX{XX}'
+        hexStr = name.substr(1);
+      }


It's a bit easier to follow if we add an else and return -1 early here.

brendandahl · 2016-03-09T18:14:44Z

@brendandahl Is this what you had in mind?

Yes, looks very good. Just the minor nits above.

…odeToGlyphId` map for TrueType fonts (bug 1132849, issue 6893, issue 6894) This patch adds a `getUnicodeForGlyph` helper function, which is used to recover Unicode values for non-standard glyph names. Some PDF generators, e.g. Scribus PDF, use improper `uniXXXX` glyph names which breaks the glyph mapping. We can avoid this by converting them to "standard" glyph names instead. Fixes https://bugzilla.mozilla.org/show_bug.cgi?id=1132849. Fixes 6893. Fixes 6894.

Snuffleupagus · 2016-03-09T18:55:13Z

Yes, looks very good. Just the minor nits above.

Thanks for the review, I've addressed the comments.

/botio test

pdfjsbot · 2016-03-09T18:55:14Z

From: Bot.io (Windows)

Received

Command cmd_test from @Snuffleupagus received. Current queue size: 0

Live output at: http://107.22.172.223:8877/635be36d4835bb1/output.txt

pdfjsbot · 2016-03-09T18:55:14Z

From: Bot.io (Linux)

Received

Command cmd_test from @Snuffleupagus received. Current queue size: 0

Live output at: http://107.21.233.14:8877/29f3c11931563c4/output.txt

pdfjsbot · 2016-03-09T19:15:40Z

From: Bot.io (Windows)

Success

Full output at http://107.22.172.223:8877/635be36d4835bb1/output.txt

Total script time: 20.43 mins

Font tests: Passed
Unit tests: Passed
Regression tests: Passed

pdfjsbot · 2016-03-09T19:17:23Z

From: Bot.io (Linux)

Failed

Full output at http://107.21.233.14:8877/29f3c11931563c4/output.txt

Total script time: 22.14 mins

Font tests: Passed
Unit tests: Passed
Regression tests: FAILED

Image differences available at: http://107.21.233.14:8877/29f3c11931563c4/reftest-analyzer.html#web=eq.log

brendandahl · 2016-03-10T17:25:31Z

r+
/botio makeref

pdfjsbot · 2016-03-10T17:25:33Z

From: Bot.io (Windows)

Received

Command cmd_makeref from @brendandahl received. Current queue size: 0

Live output at: http://107.22.172.223:8877/c58317f2e65a163/output.txt

pdfjsbot · 2016-03-10T17:25:33Z

From: Bot.io (Linux)

Received

Command cmd_makeref from @brendandahl received. Current queue size: 0

Live output at: http://107.21.233.14:8877/258818beaeb1a03/output.txt

pdfjsbot · 2016-03-10T17:45:46Z

From: Bot.io (Windows)

Success

Full output at http://107.22.172.223:8877/c58317f2e65a163/output.txt

Total script time: 20.21 mins

Lint: Passed
Make references: Passed
Check references: Passed

pdfjsbot · 2016-03-10T17:49:05Z

From: Bot.io (Linux)

Failed

Full output at http://107.21.233.14:8877/258818beaeb1a03/output.txt

Total script time: 23.52 mins

Lint: Passed
Make references: FAILED

brendandahl · 2016-03-10T18:37:22Z

/botio-linux makeref

pdfjsbot · 2016-03-10T18:37:24Z

From: Bot.io (Linux)

Received

Command cmd_makeref from @brendandahl received. Current queue size: 0

Live output at: http://107.21.233.14:8877/a88dee7cbd7205b/output.txt

pdfjsbot · 2016-03-10T19:00:41Z

From: Bot.io (Linux)

Failed

Full output at http://107.21.233.14:8877/a88dee7cbd7205b/output.txt

Total script time: 23.29 mins

Lint: Passed
Make references: FAILED

timvandermeij · 2016-03-10T19:23:41Z

/botio-linux makeref

pdfjsbot · 2016-03-10T19:23:42Z

From: Bot.io (Linux)

Received

Command cmd_makeref from @timvandermeij received. Current queue size: 0

Live output at: http://107.21.233.14:8877/56900e861a1f844/output.txt

pdfjsbot · 2016-03-10T19:45:30Z

From: Bot.io (Linux)

Success

Full output at http://107.21.233.14:8877/56900e861a1f844/output.txt

Total script time: 21.79 mins

Lint: Passed
Make references: Passed
Check references: Passed

Convert `uniXXXX` glyph names to proper ones when building the `charCodeToGlyphId` map for TrueType fonts (bug 1132849, issue 6893, issue 6894)

timvandermeij · 2016-03-10T19:46:04Z

Thank you!

Snuffleupagus added font-conversion font-truetype labels Mar 7, 2016

timvandermeij assigned brendandahl Mar 7, 2016

brendandahl reviewed Mar 9, 2016
View reviewed changes

timvandermeij merged commit 4784863 into mozilla:master Mar 10, 2016

Snuffleupagus deleted the TrueType-recover-glyphNames branch March 10, 2016 20:14

yurydelendik mentioned this pull request Mar 25, 2016

Investigate issue6961 test due to its frequent test failures #7112

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Convert `uniXXXX` glyph names to proper ones when building the `charCodeToGlyphId` map for TrueType fonts (bug 1132849, issue 6893, issue 6894) #7069

Convert `uniXXXX` glyph names to proper ones when building the `charCodeToGlyphId` map for TrueType fonts (bug 1132849, issue 6893, issue 6894) #7069

Snuffleupagus commented Mar 7, 2016

timvandermeij commented Mar 7, 2016

Snuffleupagus commented Mar 7, 2016

Snuffleupagus commented Mar 8, 2016

brendandahl commented Mar 8, 2016

Snuffleupagus commented Mar 8, 2016

Snuffleupagus commented Mar 9, 2016

pdfjsbot commented Mar 9, 2016

pdfjsbot commented Mar 9, 2016

brendandahl Mar 9, 2016

brendandahl commented Mar 9, 2016

Snuffleupagus commented Mar 9, 2016

pdfjsbot commented Mar 9, 2016

pdfjsbot commented Mar 9, 2016

pdfjsbot commented Mar 9, 2016

pdfjsbot commented Mar 9, 2016

brendandahl commented Mar 10, 2016

pdfjsbot commented Mar 10, 2016

pdfjsbot commented Mar 10, 2016

pdfjsbot commented Mar 10, 2016

pdfjsbot commented Mar 10, 2016

brendandahl commented Mar 10, 2016

pdfjsbot commented Mar 10, 2016

pdfjsbot commented Mar 10, 2016

timvandermeij commented Mar 10, 2016

pdfjsbot commented Mar 10, 2016

pdfjsbot commented Mar 10, 2016

timvandermeij commented Mar 10, 2016

Convert uniXXXX glyph names to proper ones when building the charCodeToGlyphId map for TrueType fonts (bug 1132849, issue 6893, issue 6894) #7069

Convert uniXXXX glyph names to proper ones when building the charCodeToGlyphId map for TrueType fonts (bug 1132849, issue 6893, issue 6894) #7069

Conversation

Snuffleupagus commented Mar 7, 2016

timvandermeij commented Mar 7, 2016

Snuffleupagus commented Mar 7, 2016

Snuffleupagus commented Mar 8, 2016

brendandahl commented Mar 8, 2016

Snuffleupagus commented Mar 8, 2016

Snuffleupagus commented Mar 9, 2016

pdfjsbot commented Mar 9, 2016

From: Bot.io (Linux)

Received

pdfjsbot commented Mar 9, 2016

From: Bot.io (Linux)

Success

Published

brendandahl Mar 9, 2016

Choose a reason for hiding this comment

brendandahl commented Mar 9, 2016

Snuffleupagus commented Mar 9, 2016

pdfjsbot commented Mar 9, 2016

From: Bot.io (Windows)

Received

pdfjsbot commented Mar 9, 2016

From: Bot.io (Linux)

Received

pdfjsbot commented Mar 9, 2016

From: Bot.io (Windows)

Success

pdfjsbot commented Mar 9, 2016

From: Bot.io (Linux)

Failed

brendandahl commented Mar 10, 2016

pdfjsbot commented Mar 10, 2016

From: Bot.io (Windows)

Received

pdfjsbot commented Mar 10, 2016

From: Bot.io (Linux)

Received

pdfjsbot commented Mar 10, 2016

From: Bot.io (Windows)

Success

pdfjsbot commented Mar 10, 2016

From: Bot.io (Linux)

Failed

brendandahl commented Mar 10, 2016

pdfjsbot commented Mar 10, 2016

From: Bot.io (Linux)

Received

pdfjsbot commented Mar 10, 2016

From: Bot.io (Linux)

Failed

timvandermeij commented Mar 10, 2016

pdfjsbot commented Mar 10, 2016

From: Bot.io (Linux)

Received

pdfjsbot commented Mar 10, 2016

From: Bot.io (Linux)

Success

timvandermeij commented Mar 10, 2016

Convert `uniXXXX` glyph names to proper ones when building the `charCodeToGlyphId` map for TrueType fonts (bug 1132849, issue 6893, issue 6894) #7069

Convert `uniXXXX` glyph names to proper ones when building the `charCodeToGlyphId` map for TrueType fonts (bug 1132849, issue 6893, issue 6894) #7069