Standard mapping format for legacy music fonts

[rpatters1]

I have begun to think about how to map legacy font glyphs to SMuFL and back. Recent versions of Finale have provided mappings as json files for legacy Finale fonts as follows.

• the filename is <legacyfontname>.json
• the json is a dictionary by SMuFL glyphname containing codepoint, legacyCodepoint, description, and optionally nameIsMakeMusic.

Here are some example entries. From Broadway Copyist Text.json:

	"enclosureParenOverLeft": {
		"nameIsMakeMusic": true,
		"codepoint": "U+F723",
		"legacyCodepoint": "91",
		"description": ""
	},
	"dynamicMezzo": {
		"codepoint": "U+E521",
		"legacyCodepoint": "144",
		"description": ""
	}

And from Engraver Time.json:

	"timeSig0Large": {
		"codepoint": "U+FFFD",
		"legacyCodepoint": "48",
		"description": ""
	},
	"timeSig1Large": {
		"codepoint": "U+FFFD",
		"legacyCodepoint": "49",
		"description": ""
	}

I'm guessing nameIsMakeMusic indicates that the glyph name was assigned by MakeMusic for an optional glyph. However, other fonts (in particular Bravura) also seem to use some of these names.

It seems MakeMusic may have changed their approach mid-stride, because the second example shows a different approach for optional glyphs. The glyph name is correct and useful, but the codepoint is a catchall value that is useless (U+FFFD).

I would like to see a collection of legacy font mappings hosted here. Since so many legacy fonts are already using this format, I think it makes a good starting point. However, I think we need to define exactly how to handle mappings to codepoints in the optional range 0xF400-0xFFFF. I personally like adding a boolean, though it should be more generic, such as optional. And I really dislike mapping to a catchall codepoint. That adds a lot of overhead to any consuming program.

This conversation started over at mnx.

[samuelbradshaw]

I think this would be very useful.

[knutnergaard]

@rpatters1 I agree this would be useful in principle. but how do you imagine mappings to codepoints in the optional range should be defined, given that it's intended for discretionary mappings?

I'm also not quite clear on what a boolean would solve in this case. Nor do I understand the reasoning for using a catchall codepoint, but avoiding that shouldn't be controversial from a SMuFL point of view.

[rpatters1]

I don't feel strongly about an optional boolean, but I do think it should be possible to map legacy codepoints to the optional range of codepoints. Consider that MakeMusic was using their files to map Maestro to Finale Maestro, Broadway to Finale Broadway, etc. The optional range would be quite an important feature of these mappings.

Beyond that, it seems as though many optional-range glyphs and their names are shared, specifically, between Bravura and Finale Maestro. So even though the optional range is supposedly discretionary, there does appear to be some level of ad hoc cooperation going on. (Although, I could be mistaken.)

[knutnergaard]

At least at a glance, I could not find a lot of equivalence between the optional range mappings of Bravura and Finale Maestro. However, I know there is a tendency for font designers to follow Bravura's mappings for this range as well, even though this isn't SMuFL-specified.

Accross all the legacy fonts, there are probably a lot of glyphs that aren't semantically unique, and as such, do not fulfill the requirement for inclusion into SMuFL's recommended range. Most of these, I'm guessing, will have a variant in Bravura's optional range.

For simplicities sake, I would suggest that Bravura's optional range encoding and identification of optional glyphs (incl. canonical glyph name and description) be used as a starting point for a legacy font mapping, even if this might leave a lot to be desired in terms of organization. OTOH, if there really is only a small overlap between Bravura and legacy fonts in this regard, it might be best to try and define some sort of organizational blueprint for at least a reasonable subset of the optional range.

[rpatters1]

One thought I had is perhaps to specify the font for the optional mapping. That is, rather than just an optional boolean, provide the name of the font that the optional mapping goes to.

[rpatters1]

I should add that it appears that all of the MakeMusic fonts use the same set of optional glyphs, though of course not all of them are available in every font.

[knutnergaard]

Defining a schema is probably a necessary first step in this process.

[rpatters1]

Having thought about the Finale files, I feel that those mappings are backwards. They map SMuFL glyphnames to legacy codepoints. But what I think would be more useful is a mapping from legacy codepoints to glyphnames and (optionally) SMuFL codepoints.

Here is a strawman schema:

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "SMuFL Legacy Font Mapping",
  "type": "object",
  "additionalProperties": {
    "type": "object",
    "properties": {
      "glyphname": {
        "type": "string"
      },
      "codepoint": {
        "type": "string",
        "pattern": "^U\\+[0-9A-Fa-f]{4,6}$"
      },
      "description": {
        "type": "string"
      },
      "optionalSource": {
        "type": "string"
      }
    },
    "required": ["glyphname"],
    "additionalProperties": false
  },
  "propertyNames": {
    "pattern": "^(0x)?[0-9A-Fa-f]+$"
  }
}

Things to consider:

• This schema expects the key to be a decimal integer or a hex integer preceded by 0x. It is the value of the legacy codepoint.
• Should glyphname be a glyphnames array? (This could map a legacy paren acci, for example, to three standard SMuFL glyphs.)
• If so, then I suppose codepoint should become codepoints (but still optional)
• I imagine codepoint(s) to be a convenience field that provides the codepoints directly and can be used in situations where a glyphname mapping is unavailable or difficult to obtain.
• The optionalSource field would be a free-form string that identifies the source for optional glyph names. It might be the SMuFL font that contains the optional glyphs. But it also needs to accommodate Finale that provides a superset of optional glyphnames from which all of its SMuFL fonts draw their optional glyphs. So "finale" or "bravura" or "petaluma" or whatever.

FWIW: there is a great deal of overlap between the Finale set and the Bravura set of optional glyphs.

[knutnergaard]

To me the mapping of SMuFL names to legacy codepoints makes the most sence, given that each legacy codepoint can translate to several different SMuFL codepoints. This has the added benefit of providing more descriptive mappings than doing it the other way around. If I understand your proposal correctly, this would also take care of your second point, providing simple 1-to-1 key-value (or legacy-smufl) pairs.

I'm not totally clear on the reason for an outer additionalProperties keyword. Could you explain?

Would it be better to provide codepoints as integers rather than strings? If not, I think regular unicode scalar notation, like what SMuFL already uses (e.g., "U+E000"), would be more appropriate than the hexadecimal string representation with 0x prefix, used by many programming languages, especially since those representations may vary.

Defining an explicit optionalSource might be unecessary. I think it's a good idea to replecate the reference (Bravura) as far as possible, including for optional glyphs. Glyphs beyond those covered by Bravura/SMuFL, should either be ignored until included in the spec, or assigned to a single agreed-upon codepoint.

[rpatters1]

I would be happy to present the legacy codepoint keys as either integers or U+. The only reason I did not propose U+ in the first place is that most of these legacy fonts are not Unicode and their codepoints are not U codes. So my preference would integer.

To be useful to me, this needs to support the Finale optional glyphs. These are defined in a file distributed inside the macOS Finale.app bundle called glyphnamesFinale.json which has the same format as glyphnames.json. (Win Finale has them embedded in the .exe.) Since every Finale SMuFL font draws from this set, that means a sizeable percentage of all the SMuFL fonts that currently exist use them.

Beyond this, however, if Finale is any example, many SMuFL fonts (all of the Finale ones) are redesigns of legacy fonts that contain the same glyphs recoded as SMuFL. To be useful generally, this mapping file should be able to specify a specific SMuFL font it maps to. Perhaps instead of optionalSource we could have smuflFontName or something like it.

I make no claim to be an expert on the semantics of json schemas. I cobbled that together merely as a starting point for discussion. I thought additionalProperties was how to define the fields in each entry, e.g.:

{
  "0x41": {
    "glyphname": "noteheadBlack",
    "codepoint": "U+E0A4",
    "description": "Black notehead"
  },
  "0x42": {
    "glyphname": "noteheadHalf"
  }
}

If there's a better way, I would welcome the opportunity to learn it.

[knutnergaard]

To be useful to me, this needs to support the Finale optional glyphs. These are defined in a file distributed inside the macOS Finale.app bundle called glyphnamesFinale.json which has the same format as glyphnames.json. (Win Finale has them embedded in the .exe.) Since every Finale SMuFL font draws from this set, that means a sizeable percentage of all the SMuFL fonts that currently exist use them.

Beyond this, however, if Finale is any example, many SMuFL fonts (all of the Finale ones) are redesigns of legacy fonts that contain the same glyphs recoded as SMuFL. To be useful generally, this mapping file should be able to specify a specific SMuFL font it maps to. Perhaps instead of optionalSource we could have smuflFontName or something like it.

I see. Then I agree that the mapping of optional glyphs should be font dependant. Perhaps an explicit keyword like optionalGlyphsMapping would clarify this intention even more than smuflFontName?

According to the spec:

The value of "additionalProperties" MUST be a valid JSON Schema. [...] The behavior of this keyword depends on the presence and annotation results of "properties" and "patternProperties" within the same schema object. Validation with "additionalProperties" applies only to the child values of instance names that do not appear in the annotation results of either "properties" or "patternProperties". [...] Omitting this keyword has the same assertion behavior as an empty schema.

As I understand this, additionalProperties is appropriate insofar as to limit property names to the ones explicitly declared in the properties field.

If using codepoints (either in hex or scalar format) as property names, you would indeed need to provide an array of all the canonical SMuFL glyph names it maps to, rather than a single string value. This seems to me to be needlessly structurally complicated. Is there any particular reason why you prefer that rather than the other way around?

[rpatters1]

I would be fine with optionalGlyphsMapping.

If using codepoints (either in hex or scalar format) as property names, you would indeed need to provide an array of all the canonical SMuFL glyph names it maps to, rather than a single string value.

My goal is a schema specifying a layout as below (I've switched to decimal integer keys):

{
  "65": {
    "glyphname": "noteheadBlack",
    "codepoint": "U+E0A4",
    "description": "Black notehead"
  },
  "66": {
    "glyphname": "noteheadHalf"
  }
}

I don't see an additionalProperties that applies to the outer propertyNames key value (which is where, as I understand it, you are suggesting we might have to specify all possible values).

There is a "codepoint" key inside the additionalProperties. This is a SMuFL codepoint and its value (as I understand it, controlled by the pattern, is not a key.) So I apologize that I am having trouble understanding the issue you are raising.

[rpatters1]

Actually, thinking further about optionalGlyphsMapping, I'm wondering if it should named more generally. Yes, it would be the font for any optional glyphs mapped, but I am thinking of this more as an overall recommendation from the mapping. It tells the consuming program, "use this SMuFL font if you have it" for any glyph. Obviously the consumer can choose to take the recommendation or not.

[rpatters1]

Instead of a member of each mapped record, maybe the target SMuFL font recommendation could be an optional top-level key.

[knutnergaard]

My goal is a schema specifying a layout as below (I've switched to decimal integer keys):

If you are adamant about mapping legacy codepoints to SMuFL glyph names rather than the other way around, I think you have to allow for each codepoint to map to an array of multiple names, since each legacy glyph can apply to multiple codepoints in SMuFL.

In my view, however, this seems to make for an needlessly complicated structure. It's much simpler to just map each possible SMuFL glyph name to the appropriate legacy codepoint, especially since you then don't have to convert the type of the source codepoint or add an additional codepoint property to get the codepoint as an integer rather than a string.

If annotating the codepoint as a string value, I personally think we should use Unicode scalar notation, as long as this is what SMuFL itself uses. Providing codepoints as integer strings doesn't make that much sense to me, and kind of solidifies the need for my preferred mapping of glyph names to legacy codepoints.

I don't see an additionalProperties that applies to the outer propertyNames key value (which is where, as I understand it, you are suggesting we might have to specify all possible values).

There is a "codepoint" key inside the additionalProperties. This is a SMuFL codepoint and its value (as I understand it, controlled by the pattern, is not a key.) So I apologize that I am having trouble understanding the issue you are raising.

There seems to be a misunderstanding here. As far as I understand it, additionalProperties controls whether properties not listed in properties or patternProperties are allowed. In it's simplest use case, it means that additionalProperties could be set to false to disallow any properties except those listed in the other two property categories.

Property names, however, is intended to validate property names using a specified schema, e.g.:

{
  "type": "object",
  "propertyNames": {
    "pattern": "^[a-z_][a-z0-9_]*$"
  }
}

[rpatters1]

I misunderstood your comment about the mapping direction, thinking you were in agreement that legacy codepoints should map to smufl glyphnames. If we map from glyphnames (which is how Finale is already doing it), I don't see how this improves the many-to-1 situation. I would argue it makes it worse. (I already suggested we might well need arrays rather than single values for glyphs and smufl points in the from-legacy design.)

I am curious to know the use case you imagine where mapping glyphnames to legacy codepoints is preferable. I am imagining use cases that are exporting legacy files to modern programs, and that use case is much better off with mapping from legacy codepoints. I can make it work either way, though.

[knutnergaard]

Programmatically I don't seem much difference whether you map glyph names to codepoints or the other way around, so I'm myself curious why you think this is so much better). I have the following reasons for preferring a glyph-name-to-codepoint mapping:

1. Clarity: The keys are descriptive, and as such you will know which glyph it's referring to without having to know the legacy mapping by heart.
2. Simplicity: Avoids having to convert the keys from integers to strings and back and eliminates value duplication with a dedicated codepoint field.
3. Structure: In my book, an object of codepoints mapped to an array of glyph names (I know you already suggested this) is more complex than an object of glyph names mapped to a single codepoint.

[rpatters1]

I think we may be talking past each other, so let me lay out the situation from my perspective.

Prior Art

The prior art for this that I know of is the MakeMusic-supplied legacy font mappings. These are indeed keyed by glyphname, as you suggested. In theory this would allow more than one glyph name to map to the same legacy codepoint. However, in practice, not a single one of the legacy mappings from Finale does this. They are all exactly 1-to-1 mappings. (Usually only a subset of legacy codepoints are mapped, which is fine.)

Constraints on the Prior Art

The Finale files were constrained in that many of the glyphs in legacy Maestro (for example) could not be directly mapped to standard glyphnames. Because the SMuFL standard (correctly in my view) discarded glyphs that were combinations of other glyphs, many of the combo-glyphs in Maestro ended up mapped to glyphs in the optional range that kept the combo. This no doubt made Finale's transition to SMuFL much smoother.

Requirements

An absolute requirement of this mapping, for me, is that it provide a definitive answer to the question, "What glyphname (or sequence of glyphnames, see below) should I substitute for this legacy codepoint?" The current Finale mappings answer this question, because they are always strictly 1-to-1.

However, in general a mapping from glyphname to legacy codepoint does not answer the question definitively, so it would introduce a lot of complexity were I to try to use it. Complexity is what I'm trying to get rid of.

In addition, it would be nice if the new design removed the constraint on the prior art mentioned above. It would allow more mappings to stay in the standard codepoint range.

Proposed High-Level Design

• A mapping keyed on legacy codepoint guarantees at most one answer to the question that this design must answer, which is the best result we can expect.
• We could map the legacy codepoint to a sequence of glyphnames rather than always a single glyphname. This would allow more codepoints to be mapped in the standard range of glyphs. I do not feel strongly about having this, though.

[knutnergaard]

I see your point, and I agree that codepoint-to-glyph mappings fit better with the use case you're outlining.

[mscuthbert]

Two-cents from working with mnx and other JSON schemas -- consider implementing w3c-cg/mnx#428 (comment fields). And/or some sort of free-format optional "explanation" format, for notating things like "anchor point is on the bottom right of glyph, not bottom left as SMuFL expects".

Come to think of it, consider implementing optional anchor point x and y shifts. (blank = unknown not 0?). Or something like w3c-cg/mnx#429 so that people can add that later.

Please put in the schema a version number as number or string -- we may make mistakes that require revision -- and a $schema property (for the document in addition to the one in the $schema itself) which points to where on schemastore.org we'll eventually house this. A $schema property is especially important since the file name for these files will vary (current suggestion: <fontname>.json) so the appropriate schema can't be deduced from the filename alone (unlike things like .tsconfig.json). Also suggest calling it <fontname>-smuflmap.json or something like that with a standard filename-stem that can be searched for by name.