Tool to split/merge TTC

[behdad]

As requested by Adam. Would be useful. Possibly as a separate tool as well as with support in ttx.

[miguelsousa]

The FDK now includes otc2otf and otf2otc tools.

[miguelsousa]

@twardoch ^^^

[twardoch]

Cool :) I'm leaving this one open so Behdad remembers that he intends to still improve fontTools a bit (ability to handle TTC as a whole, he mentioned).

[behdad]

@miguelsousa Is there documentation about those tools in FDK? I can provide the same API.

[miguelsousa]

@behdad the documentation that exists is in the scripts themselves and can be obtained by using the -h option.

otc2otf otc2otf.py  [-r] <font.ttc>

Extract all OpenType fonts from the parent OpenType Collection font.
-r  Optional. Report TTC fonts and tables, Will not write the output files.

example:
 AFDKOPython otc2otf.py  -r  LogoCutStd.ttc

The script may be invoked with either the FDK command:
  otc2otf
or directly with the command:
  python <path to FDK directory>/FDK/Tools/SharedData/FDKScripts/otc2otf.py

otf2otc  -t <table tag=src font index> -o <output ttc file name> <input font file 0> ... <input font file n>

-t <table tag=source font index>  Optional. When this option is present, the matching table from the specified font file is
used for all the font files.

example:
  otf2otc -t 'CFF '=2 -o LogoCutStd.ttc LogoCutStd-Light.otf LogoCutStd-Medium.otf LogoCutStd-Bold.otf LogoCutStd-Ultra.otf
# The 'ttc' file will contain only one CFF table, taken from  LogoCutStd-Bold.otf.

The script may be invoked with either the FDK command:
  otf2otc
or directly with the command:
  python <path to FDK directory>/FDK/Tools/SharedData/FDKScripts/otf2otc.py

Build an OpenType Collection font file from two or more OpenType font
files. The fonts are ordered in the output 'ttc' file in the same order
that the file names are listed in the command line. If a table is
identical for more than one font file, it is shared.

For each file, check that it looks like an sfnt file.

For each file, read in tables. Build a font object list, and dict of table tags to table data blocks.

A font object contains;
  font file name
  list of [tableTag, table data, shared] pairs

as the font tables are being read in, they are compared against the list of already seen data items.
If any match, the font table reference is replaced by the reference to the table already seen.

[anthrotype]

I'd like to add support for collections to the WOFF2 encoder. However we first need to decide on an API for TTCollection.
Any suggestions?

[behdad]

Let me upload whatever I have and leave it to your mighty hands...

[anthrotype]

👍

[twardoch]

BTW, note that the otc tools in FDK are completely standalone, simple, dependency-free pure-Python tools:
https://github.com/adobe-type-tools/afdko/blob/master/FDK/Tools/SharedData/FDKScripts/otc2otf.py
https://github.com/adobe-type-tools/afdko/blob/master/FDK/Tools/SharedData/FDKScripts/otf2otc.py

I think this could be easily lifted, except that FDK is Apache 2 while fontTools is MIT-licensed. Not sure how this affects things.

A.

[behdad]

Note that Doug also wrote one in nototools:
https://github.com/googlei18n/nototools/blob/master/nototools/ttc_utils.py
https://github.com/googlei18n/nototools/blob/master/nototools/decompose_ttc.py

I agree we want to add API though. Unfortunately I cannot find what I had :(. It wasn't much though, was TTCollection indeed. For decompiling, it's easy. I think we want to set something like collection.fonts to be a list of TTFont() objects. Add iteration to the collection object as well such that we can just do for "font in collection:".

Compiling becomes more complicated, but I leave that to you!

Also, maybe we should add a TTFontOrCollection() function that creates either a TTFont or TTCollection based on the font data. We also need to add a simpler way to detect what the data represents. Currently one has to try to open...

TTCollection should preferably have much of the same API as TTFont, in terms of compile(), save(), fromXML(), etc.

Thanks!

[rsheeter]

FWIW I did a test implementation (not realizing this issue existed) of TTC for fontTools and I found needing to branch what I was creating and worry about whether I had a TTFont or a SomethingElse annoying. I found it more convenient to make TTFont capable of representing either a regular font or a collection. If its a collection it has a fonts attribute with the TTFont's within the collection. This results in compile, save, to/from XML working with minimal changes.

[behdad]

I found it more convenient to make TTFont capable of representing either a regular font or a collection. If its a collection it has a fonts attribute with the TTFont's within the collection. This results in compile, save, to/from XML working with minimal changes.

Since it's Python, you can have two different types that implement the same save, XML, compile, ... API...

[rsheeter]

The convenience factor was mostly avoiding having to open the file to decide which object I was creating. As far as the API, my thinking was just that if it's the same entry points it doesn't have "much of the same API", it has exactly the same API and there is less tendency for those APIs to diverge over time. Either way is certainly workable.

However, IMHO the more interesting question is how to handle editing shared tables. For example, if fonts 0 and 1 share glyf, can I reach in and edit by both fonts[0]['glyf'] and fonts[1]['glyf']? How does one check if a table is shared or cause it to become shared?

[behdad]

@anthrotype Want to take this one? Perhaps summarize what we discussed in person? Thanks.

[anthrotype]

Sure, I was planning to do it as soon as I finish the logging thing I'm working on.

[HinTak]

While implementing DSIG checking a while ago, I noticed some TTCs have individual DSIG's in their member fonts, and I have always wondered if I can split them up and verify the individual ones. MS Font Validator has some code for writing fonts (which itself does not use), so this is probably do'able.

[HinTak]

@rsheeter : about sharing tables. Sharing glyf table is messy (or have a superset that includes all the edits), but sharing most other tables could work at a copy-on-write manner, I guess; and just before writing, one needs to calculate the table checksum anyway, and I seem to have seen some code, either in ttx or in FontVal, which avoid writing duplicate tables of identical checksums.

Hmm, it is in FontVal, OTFontFile/OTFile.cs:WriteTTCFile(), line 566 onwards - it does a barbaric binary check.

[HinTak]

@behdad @davelab6 @anthrotype @aaronbell @twardoch @miguelsousa @n8willis : A mere 30 lines (include blank lines, minus comments), IronPython Fontval-based example to split ttc's.

https://gist.github.com/HinTak/33cdcb8e1558538389cb5e0475674acc

I have only tried it on mingliu.tttc and the output seems to be valid. This is just to check that the Writefont functionality inside FontVal (which it does not use itself - obviously it has no use of that sort of routines) is functional. I might update it a bit to do what I have in mind - see if after splitting, whether individual DSIG in member fonts are valid in some such files I see.

[anthrotype]

You shouldn't bother verifying the individual DSIGs of TTC's members. They should have never been there in the first place. TTC may only have a single Format-1 DSIG table for the whole collection.

From https://www.microsoft.com/en-us/Typography/dsig.aspx

Individual fonts included in a TrueType collection should not be individually signed as the process of making the TTC could invalidate the signature on the font.

[HinTak]

You shouldn't bother verifying the individual DSIGs of TTC's members.

I know - I am just curious whether the merge happens immediately after the signing, or if there are changes between signing and merging.

[HinTak]

The corresponding FontVal-based IronPython script to merge truetype fonts into a truetype collection is about the same length, and 2/3 the same - they use the same modules, etc, obviously.

https://gist.github.com/HinTak/96a686572a824d78f8e1967b464821aa

Ran the splitter and merger against mingliu, @rsheeter , and there was already de-duplication on write, which is, I am glad to see, functional - note re-merge is a lot smaller than sum of all members:

32217124 original
27349496 member0
27349516 member1
27387460 member2
32210100 re-merged version

The 7k difference between the original and the new-merge is the TTC DSIG. The old version had all the directories up front, while the new has 2nd directory after tables of 1st font, etc.

The 32MB TTC is about 22MB of glyphs, common to all three members, but contains two versions of EBDT at 5MB . So that's how all 3 member fonts are about 27MB each.

[behdad]

Here's the API Cosimo and I came up with:

fonts = TTCollection(ttcFilename, shareTables=False, ...)

fonts.fonts # is a plain list of the TTFont objects
fonts.importXML(...)
fonts[idx] # __getitem__, __setitem__, __delitem__
fonts.saveXML(...) # TTX XML with <TTCollection> root tag.

fonts.save(...) # ttc
fonts.flavor = 'woff2'
fonts.save(...) # woff2 collection

list_of_fonts = [ttFont1, ttFont2, ...]
ttc = TTCollection()
ttc.fonts = list_fonts
ttc.save(...)
fonts.save()

To get there, here's a task list:

• Implement TTCollection() constructor / open,
• Implement sharing objects in TTColection() open,
• Implement TTCollection.save() without table sharing,
• Implement table sharing in TTCollection.save() (always enabled; why would anyone want to disable?),
• Implement saveXML(),
• Implement fontTools.ttc2ttf and fontTools.ttf2ttc commands,
• Implement importXML()
• Implement a function to open file and return TTFont or TTCollection based on contents,
• Same as above but for XML,
• Implement WOFF2 collections,
• Implement object-sharing in XML?

[behdad]

I've implemented basic reading, writing, and saveXML for TTCollection. If someone wants to go ahead and implement ttc2ttf and ttf2ttc, as well as the rest of the list above, would be great. I'm going to leave it for now.

[anthrotype]

Thank you @behdad! We'll pick it up from here 👍

[behdad]

This is done now by fonttools ttLib...

[kenmcd]

This is done now by fonttools ttLib...

Is there someplace where how to use these new split/merge features is explained?
Nothing in the official docs, and I do not see anything in the files (as mentioned above).
Thanks.

[behdad]

Is there someplace where how to use these new split/merge features is explained?

Try fonttools ttLib --help

This currently only supports extracting one face from a TTC at a time, using -y parameter to choose which face, and -o to save the output. Or building a TTC by listing multiple input fonts and using -o to save the output.

[kenmcd]

Is there someplace where how to use these new split/merge features is explained?

Try fonttools ttLib --help

This currently only supports extracting one face from a TTC at a time, using -y parameter to choose which face, and -o to save the output. Or building a TTC by listing multiple input fonts and using -o to save the output.

Already had updated to latest fonttools and had also ran that help command.
And am already familiar with extracting one-by-one.
Thought this added some features.
Thanks for the info.

[rsms]

@behdad Try fonttools ttLib --help

This does not seem to work with the latest release

$ fonttools ttLib --help
Traceback (most recent call last):
  File "<frozen runpy>", line 148, in _get_module_details
  File "<frozen runpy>", line 142, in _get_module_details
ImportError: No module named fontTools.ttLib.__main__

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/fonttools", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.11/site-packages/fontTools/__main__.py", line 31, in main
    runpy.run_module(mod, run_name="__main__")
  File "<frozen runpy>", line 222, in run_module
  File "<frozen runpy>", line 152, in _get_module_details
ImportError: No module named fontTools.ttLib.__main__; 'fontTools.ttLib' is a package and cannot be directly executed

However if I invoke the ttLib module directly it works

$ python3 -m fontTools.ttLib.__init__ --help
usage: fonttools ttLib [-h] [-o FILE] [-y NUMBER] [--lazy] [--no-lazy] [--flavor FLAVOR]
                       [font ...]

Open/save fonts with TTFont() or TTCollection()
...

Environment:

• Python 3.11.2 (CPython)
• macOS 10.15 x86_64
• fonttools 4.39.3

[behdad]

That seems to be picking up an older fonttools command it seems. How did you install the latest fonttools?

[rsms]

How did you install the latest fonttools?

I've tried both a "normal" system-wide install and installing in a virtual environment (same results)

I.e.

$ pip3 install fonttools
$ fonttools 2>&1 | head -n1
fonttools v4.39.3

and isolated with venv:

$ mkdir tmp1 && cd tmp1
$ python3 -m venv venv
$ . venv/bin/activate
(venv) $ which pip
/Users/rsms/tmp1/venv/bin/pip
(venv) $ pip install fonttools
Collecting fonttools
  Using cached fonttools-4.39.3-py3-none-any.whl (1.0 MB)
Installing collected packages: fonttools
Successfully installed fonttools-4.39.3
(venv) $ fonttools 2>&1 | head -n1
fonttools v4.39.3
(venv) $ fonttools ttLib --help
Traceback (most recent call last):
  File "<frozen runpy>", line 148, in _get_module_details
  File "<frozen runpy>", line 142, in _get_module_details
ImportError: No module named fontTools.ttLib.__main__
...

[anthrotype]

yeah, I can reproduce, thanks for the report. We need to add a __main__.py to fontTools/ttLib since it is a package (directory). I'll fix this