Encode unencoded glyphs as F0000 + hex(GID)

[davelab6]

I was chatting with @twardoch today about how glyph names are the 'primary key' for fonts, because in any contemporary font you have so many unencoded glyphs, accessed with OpenType Layout logic... But unencoded glyphs are tricky to precisely call, because OTL logic is per-font. I mentioned that I might like to use the Unicode Private Use Area to encode otherwise-unencoded glyphs.

Adam kindly mentioned he already thought about this, and he concluded that the Private Use Plane A (Unicode Plane 15) is ideal for this, as its U+F0000..U+FFFFD so you can use a value of F0000 + hex(GID) to cleanly, logically, encode all unencoded glyphs.

Let's do it!

• add a "3.10" cmap subtable

[twardoch]

All it requires is a simple tool which adds a "3.10" cmap subtable which maps glyph ids to the PUP A (sequentially, by adding 0xF0000 to the glyph ID). Because the codepoints will be in the same order as the glyph IDs, you can use the space-saving cmap format 12 which only defines the start and end of the cmap mapping range. So the added size overhead is small.

[vitalyvolkov]

@behdad Is there any way to define that glyph is unencoded using fontTools?

[davelab6]

@hash3g you can find the glyph IDs in the GlyphOrder table, eg https://github.com/hash3g/yesevaone/blob/master/YesevaOne-Regular.ttf.GlyphOrder.ttx and you can find glyphs that are encoded in the cmap table. I guess you need to make a set of the glyph names and a set of the encoded glyph names and compare them to get the set of unencoded glyphs.

[twardoch]

I would advocate that it would be much simpler (and more storage-effective) if you just encode ALL glyphs as U+F0000 + GID.

1. Check if there already is a cmap subtable with PID 3 EID 10 (3.10 for short). If it dors, skip to step 4.
2. Create a cmap subtable in format 12 and assign it to the cmap table as 3.10
3. Copy all mapping from the PID 3 EID 1 (3.1 for short) cmap subtable to the 3.10 subtable, as the spec requires 3.10 to be a superset of 3.1.
4. "Blindly" assign mapping to all glyphs from GlyphOrder from U+F0000 to U+F0000 + len(GlyphOrder) - 1.

This has the advantage that cmap subtable format 12 uses an efficient storage for continuous code-to-GID ranges. With my method, you'll only create one such range, so it'll only add a few bytes to the size, and will be very fast.

This approach has an additional benefit:
As a user of such font, I am are not forced to address the properly (i.e. via Unicode) glyphs encoded using the F0000+ codes.

I can still use the proper Unicodes. But if I do so, the browser/app will always perform the Unicode processing and default OpenType Layout shaping for complex scripts. So I won't really have the guarantee that the glyph I'm seeing is actually the glyph assigned to the Unicode codepoint in the font's cmap. It will be for most Unicodes but for some codepoints, the "Unicode+OTL magic" will kick in.

But if I address even the "properly" encoded glyphs using the U+F000+ codepoint, I will have a WYSIWYG guarantee. Even more: with harfbuzz.js, I can run a JS port of HarfBuzz in the browser, take the output GIDs, add F000+ to them and have my own explicit custom OTL processing if I need to. So I'm completely in control and independent of any "browser magic".

[twardoch]

Here is my code that does exactly what I described above.

#! /usr/bin/python
# -*- coding: utf-8 -*-
# 
# pyftaddspuaabygids.py
# Map all glyphs to the Supplementary PUA-A plane (U+F0000..U+FFFFF) 
# by 0xF0000 + glyphID
#  
# Copyright (c) 2014 by Adam Twardoch
# 
# Licensed to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

import fontTools.ttLib, sys, copy

def addSPUAByGlyphIDsMappingToCMAP(ttx):
    cmap = ttx["cmap"]
    # Check if an UCS-2 cmap exists
    for ucs2cmapid in ((3, 1), (0, 3), (3, 0)): 
        ucs2cmap = cmap.getcmap(ucs2cmapid[0], ucs2cmapid[1])
        if ucs2cmap: 
            break
    # Create UCS-4 cmap and copy the contents of UCS-2 cmap
    # unless UCS 4 cmap already exists
    ucs4cmap = cmap.getcmap(3, 10)
    if not ucs4cmap: 
        cmapModule = fontTools.ttLib.getTableModule('cmap')
        ucs4cmap = cmapModule.cmap_format_12(12)
        ucs4cmap.platformID = 3
        ucs4cmap.platEncID = 10
        ucs4cmap.language = 0
        if ucs2cmap: 
            ucs4cmap.cmap = copy.deepcopy(ucs2cmap.cmap)
        cmap.tables.append(ucs4cmap)
    # Map all glyphs to UCS-4 cmap Supplementary PUA-A codepoints 
    # by 0xF0000 + glyphID
    ucs4cmap = cmap.getcmap(3, 10)
    for glyphID, glyphName in enumerate(ttx.getGlyphOrder()):
        ucs4cmap.cmap[0xF0000 + glyphID] = glyphName

def usage():
    print "Map all glyphs to the Supplementary PUA-A plane (U+F0000..U+FFFFF) by 0xF0000 + glyphID"
    print "python %s inputfile[.otf|.ttf] outputfile[.otf|.ttf]" % sys.argv[0]

if len(sys.argv) == 3:
    inpath = sys.argv[1]
    outpath = sys.argv[2]
    ttx = fontTools.ttLib.TTFont(inpath, 0, verbose=0)
    addSPUAByGlyphIDsMappingToCMAP(ttx): 
    ttx.save(outpath)
    ttx.close() 
else: 
    usage()

[behdad]

I categorically reject this and think it's a bad idea. Nowhere in this report I see any reasoning for why this is needed or is a good idea.

[twardoch]

Ah, yes. We talked with Dave about this. Sorry it didn't become clear.

The idea is not to do this for production-ready fonts but for the purpose of development, to be used within the context of document-driven type design and similar such applications.

In a way, think of it as the "debug" mode of building fonts. Such debug mode might include other options that generate some redundant data (such as, well, glyph names! :) ) which is useful while designing but when building fonts in "release" mode, this stuff should not be included.

[behdad]

Ok, sure. Yeah, that would be useful.

[davelab6]

@behdad could you explain more about why you think this is a bad idea..? You think that if all Google Fonts have this feature, that it will increase the use of PUA characters and documents tightly bound to particular fonts in general usage?

[behdad]

@davelab6 for the same reasons that non-Unicode encodings are bad. This is even worse, this is full custom encoding, meaning any text encoded in those is illegible to any text processing use.