Enhance otf2ttf for ttc, parallel, and file wildcards #1000

msoxzw · 2019-10-16T13:19:58Z

I mainly pick these from my repo

From my view, converting ttc font to the first font in ttc is very confusing. Hence, processing ttc font as a whole is a better way compared with splitting font, then merging it.

Moreover, calculating TrueType outline from PostScript outline is time-consuming. So, I add data parallel and file wildcards input for otf2ttf.

msoxzw · 2019-10-16T13:24:30Z

However, I do not know how to make a test ttc font for unit test. So, unit test for this PR is missing.

msoxzw · 2019-10-16T17:43:53Z

The test error is expected, because multiprocessing module will mess up the logging module, particularly in Windows.

>       assert (b"WARNING: Dropping glyph names, they do not fit in 'post' "
                b"table.") in output
E       assert b"WARNING: Dropping glyph names, they do not fit in 'post' table." in b"Dropping glyph names, they do not fit in 'post' table.\r\n"

miguelsousa · 2019-10-16T18:17:11Z

Looking at the failure makes me think that removing the WARNING: part from the expected result should make the test pass again.

Thanks for the PR, by the way. I have a few more comments to make but I need to run some validations first.

miguelsousa · 2019-10-16T18:22:40Z

@josh-hadley perhaps something to look into: CircleCI didn't run. I wonder if it didn't happen because the branch to be merged doesn't belong to the afdko repo.

miguelsousa

Looks good.
My only request is to handle non-OTF inputs and TTCs containing TTFs more gracefully. Right now the whole process stops at the assert ttFont.sfntVersion == "OTTO". It'd be better to skip the fonts that cannot be converted and log a warning.
Attached are fonts you can use for new tests.
pr1000.zip

msoxzw · 2019-10-18T14:04:35Z

@miguelsousa
Sorry, I do not fully understand your words. Given your attachments, I have some confusions:

font.ttf    -> font#1.ttf    or font.tff    -> none

otf_ttf.ttc -> otf_ttf#1.ttc or otf_ttf.ttc -> none
              (ttf_ttf) 

ttf_otf.ttc -> ttf_otf#1.ttc or ttf_otf.ttc -> none
              (ttf_ttf)

ttf_ttf.ttc -> ttf_ttf#1.ttc or ttf_ttf.ttc -> none

miguelsousa · 2019-10-18T18:02:44Z

@msoxzw if you look in https://github.com/adobe-type-tools/afdko/tree/develop/tests/otf2ttf_data/input there's only OTFs in there. The TTF and TTC files I provided in the ZIP are for adding new tests.

Here's how I expect otf2ttf to work with the new test input files:

font.ttf, skip this file — the font is TTF already
ttf_ttf.ttc, skip this file — the TTC contains 2 fonts and they're both TTF
otf_otf.ttc, convert this file — the TTC contains 2 fonts and they're both OTF
otf_ttf.ttc, convert this file, but skip one of its fonts — the TTC contains 2 fonts; the first font is OTF and the second is TTF
ttf_otf.ttc, convert this file, but skip one of its fonts — the TTC contains 2 fonts; the first font is TTF and the second is OTF

msoxzw · 2019-10-19T06:48:51Z

@miguelsousa
Thanks a lot. At present I have only a small question:
In otf_ttf.ttc or ttf_otf.ttc, whether or not the original TTF is also saved in the converted TTC font?

miguelsousa · 2019-10-19T16:53:32Z

Yes, keep the original TTFs inside the TTCs.
Thank YOU for taking the time to make these enhancements.

msoxzw · 2019-10-20T07:16:17Z

@miguelsousa
The test fonts that you provide give me much insight about OpenType.

I assume that the the results of otf_otf.ttc, otf_ttf.ttc, and ttf_otf.ttc are identical to ttf_ttf.ttc excluding checkSumAdjustment, created, and modified. However, the test error seems to detect the bugs in my codes. After a while, I think that I spot the cause of error: the 'hmtx' tables of test fonts are likely to not strictly conform to the OpenType spec, which makes some minor differences between ttf_ttf.tcc and others.

miguelsousa · 2019-10-21T04:54:43Z

I assume that the the results of otf_otf.ttc, otf_ttf.ttc, and ttf_otf.ttc are identical to ttf_ttf.ttc excluding checkSumAdjustment, created, and modified

Yes, I also expected that the TTX dump of ttf_ttf.ttc would be the same as the dump of the other TTCs after their conversion.
Looking at the failures I can see that the problem is actually in the TTCs I provided. They were made with subset fonts that didn't get all their values recalculated (e.g. hhea.advanceWidthMax, maxp.maxCompositeContours), so the TTCs wound up with inflated numbers. I'll regenerate the TTCs and provide new files soon.

python/afdko/otf2ttf.py

miguelsousa · 2019-10-21T04:59:02Z

tests/test_utils.py

+        font = TTCollection(font_path)
+
+    temp_path = get_temp_file_path()
+    font.saveXML(temp_path, tables=tables)


Please add font.close() after font.saveXML().

I have investigated this issue before I decided to remove "with" statement. "TTCollection" class does not provide the "close" method. Therefore, adding font.close() is impracticable. However, due to garbage collector of Python, the font object will be automatically closed and deleted, but heavily relying on garbage collector is a bad practice. Of course, there are at least two workarounds:

Re-implement the close method:

if font.reader is not None: font.reader.close()

Open font file by built-in open:

with open(font_path, 'rb') as f: try: font = TTFont(f) except TTLibError: font = TTCollection(f) temp_path = get_temp_file_path() font.saveXML(temp_path, tables=tables) return temp_path

Nevertheless, both methods, I think, will add maintenance burden if the "TTCollection" class implement close method in some day.
So, how do you think?

I think the best would be to implement a close() method in the TTCollection class.
Would you mind submitting such change to https://github.com/fonttools/fonttools ?

Not at all.

Moreover, if I submit such change and it is merged in "fonttools", this whole project will require the latest "fonttools". Hence, is it worthy to do so?

Yes. The afdko always updates to the latest version of FontTools.
A lot of other projects rely on FontTools so every opportunity to make the changes there benefits everyone.

miguelsousa · 2019-10-21T05:46:49Z

new_TTCs.zip

miguelsousa · 2019-10-21T17:28:04Z

@msoxzw code coverage on otf2ttf dropped quite a lot, most likely because of the parallelisation.

Do you know how to fix the reporting?

msoxzw · 2019-10-22T04:23:52Z

@miguelsousa
Thank you for digging out this code coverage issue from my PR. At present, this PR is waiting for the "fontTools" next release 4.0.3, and fonttools/fonttools#1751 has been merged.

miguelsousa · 2019-10-22T19:00:17Z

@msoxzw thanks for the FontTools contribution. We'll integrate the upcoming version shortly after it's released.

miguelsousa · 2019-11-19T00:07:45Z

@msoxzw FontTools v4.1.0 has been integrated

msoxzw · 2019-11-19T06:25:51Z

@miguelsousa I could not figure out why PEP 448 and PEP 498 are identified as invalid syntax in Codacy/PR Quality Review. Probably, the Python version is too old in the PR Quality Review.

miguelsousa · 2019-11-19T09:03:19Z

@msoxzw no worries, we're actually thinking of disconnecting Codacy.

miguelsousa

Thanks for all the good work.

Sion added 4 commits October 16, 2019 12:54

add support ttc conversion as a whole for otf2ttf

05d8e54

Adjust default value of face-index to -1 in otf2ttf

e72d88f

Support file wildcards input for otf2ttf

b13db92

Support data parallel conversion for otf2ttf since it is CPU intensive

7dbdb5e

miguelsousa requested changes Oct 16, 2019

View reviewed changes

Sion added 2 commits October 17, 2019 12:48

Fix special logging issue caused by multiprocessing

5fd954f

Skip non-OTF fonts

be1c0ea

Sion added 3 commits October 20, 2019 06:54

Simplify how to tell between otf and otc

38670cc

Skip TTF fonts in TTC fonts

1905e85

Add unittest for otf2ttf

766c8eb

Remove useless else on loop in otf2ttf

e423260

miguelsousa reviewed Oct 21, 2019

View reviewed changes

python/afdko/otf2ttf.py Outdated Show resolved Hide resolved

miguelsousa reviewed Oct 21, 2019

View reviewed changes

Sion added 3 commits October 21, 2019 06:36

Update test fonts

508504d

Adjust indentation in differ

b0e87a2

Revert to single check OTF

fb73aef

Sion added 2 commits October 22, 2019 03:42

Fix code coverage issue caused by multiprocessing

1bd5b40

Add more unittest for otf2ttf

1d458ad

msoxzw added 3 commits November 19, 2019 03:55

Close font using 'with' statement

ac8d0ff

Merge branch 'develop' into develop

294a592

Ignore FontTools version when converting ttc

c94bf16

miguelsousa approved these changes Nov 19, 2019

View reviewed changes

miguelsousa merged commit d50fb71 into adobe-type-tools:develop Nov 19, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enhance otf2ttf for ttc, parallel, and file wildcards #1000

Enhance otf2ttf for ttc, parallel, and file wildcards #1000

msoxzw commented Oct 16, 2019

msoxzw commented Oct 16, 2019

msoxzw commented Oct 16, 2019

miguelsousa commented Oct 16, 2019

miguelsousa commented Oct 16, 2019

miguelsousa left a comment

msoxzw commented Oct 18, 2019

miguelsousa commented Oct 18, 2019

msoxzw commented Oct 19, 2019

miguelsousa commented Oct 19, 2019

msoxzw commented Oct 20, 2019

miguelsousa commented Oct 21, 2019

miguelsousa Oct 21, 2019

msoxzw Oct 21, 2019

miguelsousa Oct 21, 2019

msoxzw Oct 21, 2019

msoxzw Oct 21, 2019

miguelsousa Oct 21, 2019

miguelsousa commented Oct 21, 2019

miguelsousa commented Oct 21, 2019

msoxzw commented Oct 22, 2019

miguelsousa commented Oct 22, 2019

miguelsousa commented Nov 19, 2019

msoxzw commented Nov 19, 2019

miguelsousa commented Nov 19, 2019

miguelsousa left a comment

Enhance otf2ttf for ttc, parallel, and file wildcards #1000

Enhance otf2ttf for ttc, parallel, and file wildcards #1000

Conversation

msoxzw commented Oct 16, 2019

msoxzw commented Oct 16, 2019

msoxzw commented Oct 16, 2019

miguelsousa commented Oct 16, 2019

miguelsousa commented Oct 16, 2019

miguelsousa left a comment

Choose a reason for hiding this comment

msoxzw commented Oct 18, 2019

miguelsousa commented Oct 18, 2019

msoxzw commented Oct 19, 2019

miguelsousa commented Oct 19, 2019

msoxzw commented Oct 20, 2019

miguelsousa commented Oct 21, 2019

miguelsousa Oct 21, 2019

Choose a reason for hiding this comment

msoxzw Oct 21, 2019

Choose a reason for hiding this comment

miguelsousa Oct 21, 2019

Choose a reason for hiding this comment

msoxzw Oct 21, 2019

Choose a reason for hiding this comment

msoxzw Oct 21, 2019

Choose a reason for hiding this comment

miguelsousa Oct 21, 2019

Choose a reason for hiding this comment

miguelsousa commented Oct 21, 2019

miguelsousa commented Oct 21, 2019

msoxzw commented Oct 22, 2019

miguelsousa commented Oct 22, 2019

miguelsousa commented Nov 19, 2019

msoxzw commented Nov 19, 2019

miguelsousa commented Nov 19, 2019

miguelsousa left a comment

Choose a reason for hiding this comment