Strategy for extending our XCCS to Unicode mappings

[rmkaplan]

This is to capture the discussion that arose in the meeting 2/5/2025.

At least for the display, we now have the ability (thanks to @MattHeffron ) to convert Unicode BDF fonts to XCCS Strike fonts. This assigns the glyphs associated with a Unicode character to the corresponding XCCS code position, given our XCCS-to-Unicode mapping tables. The BDF fonts are much more complete than our current Class/Modern/Terminal fonts, and Unicode is much more complete than XCCS.

The proposal is to use the Unicodecode/glyph correspondences in the BDF fonts to extend our Unicode/XCCS mapping tables. If a BDF font has a glyph for unmapped Unicode ABCD, we can create a mapping between that Unicode and an otherwise unused code (reserved, undefined...) in the XCCS standard. We can do this cumulatively, as we read BDF fonts that are more or differently populated than fonts we have seen before.

We can start with a record that describes all the reserved/undefined XCCS character codes that we want to make available for assignment. This can be kept in a file (availablecodes, say) in a Unicode/xerox/extended directory. This file would just be a list of non-adjacent code-space intervals from which new codes can be peeled off.

The BDF font reads availablecodes file as it starts up. When it sees an unmapped unicode, it picks off the next code from the list, uses it to create a new Unicode mapping for the font being constructed, and updates the list.

After processing a given BDF font, the current availablecodes is written, and any new correspondences are also written out to new mapping files in the extended directory. ALL-UNICODE-MAPPINGS is then applied to update the cached numeric file of all the new mappings.

This would assign XCCS codes arbitrarily to otherwise unmapped Unicode codes, as driven by whatever order we happen to encounter them in the various BDF fonts. But the mappings, even if arbitrary, will be permanent: the next BDF font with a previously unmapped Unicode will get the XCCS that might have been assigned previously. And XCCS codes in Medley data structures will have stable interpretations whether or not particular fonts are loaded into a particular memory image.

[MattHeffron]

assign XCCS codes arbitrarily to otherwise unmapped Unicode codes

@rmkaplan I'd argue against using arbitrary XCCS codes. As you know, Unicode groups characters into blocks by language/purpose, much like character sets in XCCS. Preserving this organization would allow minimizing the .DISPLAYFONT files loaded when dealing with a particular language.
There is an additional, related problem: Unicode doesn't prevent the low 8-bits of a character code from being 255. In XCCS this is reserved for changing character sets. (Likewise, 255 isn't allowed as a character set number in XCCS, either.)
The low 8-bits issue will need to be addressed.

That being said, it seems that when mapping a Unicode glyph's code that is logically related to an existing XCCS character set, then arbitrarily assigning that code to an unused XCCS code in that character set would be reasonable.

Also, I was thinking that to deal with Unicode character codes > xFFFF (outside of the Basic Multilingual Plane), this could be dealt with by creating separate FONTDESCRIPTORs for these Unicode planes and appending an indicator to the font FAMILY name (and thus with different DISPLAYFONT file names.

The UnicodeData.txt has all of the defined codes and could be used, with a bit of work, to create a sensible mapping for extra characters.

[rmkaplan]

@MattHeffron , I think that "arbitrary" will mostly end up with the layout that you suggest, at least for Unicode characters that are grouped together in a language-related way. The sequence of next available codes will tend to be in a single part of the unused XCCS charcode space, or maybe 2 depending on the order in which we see the Unicode codes.

If unicode characters were grouped together for some purpose, they would tend to end up near each other in the (arbitrarily) assigned XCCS charsets. Otherwise, I don't know that we can determine whether a Unicode code is logically related to a code in an existing XCCS character set, without doing a lot of work. We could try to fill in advance from the UnicodeData.txt list, whether or not we had a font, but there is no point if we don't have at least one BDF font that has a glyph for a particular Unicode.

The lowbit issue should fall out from the initial list of available codes. For a completely empty XCCS charset xx we would see the interval (xx0 xx254) in the available code list, and after using xx254 it would automatically skip to whatever the next interval might be.

I think the main issue with going outside of the basic plane is to hash using IEQP instead of EQ, and then we can just assign a code in the usual (arbitrary) way.

[MattHeffron]

there is no point if we don't have at least one BDF font that has a glyph for a particular Unicode

I have the BDF file for GNU UniFont which has 57086 glyphs. I found a otf2bdf converter that can create BDF for any OpenTypeFont (e.g., the Noto fonts that @hjellinek is working with). We can get pretty much any Unicode glyph into BDF.

I think the main issue with going outside of the basic plane is to hash using IEQP instead of EQ

There's still going to be the issue of how to access these glyphs outside of the BMP in a portable/repeatable way. If we fill all of the "available" space with BMP glyphs/codes, how would/should we make the glyphs in other "planes" available?

[rmkaplan]

If we have glyphs for every possible Unicode code, then I suspect that we will run out of unused XCCS codes whether or not we go outside the basic plane. So we probably will have to make some editorial decisions--don’t assign XCCS codes to Unicodes in the basic plane that are really unlikely to ever be seen. And in doing that calculation, reserve some codes for characters outside the basic plane that we think would be more useful (like emojis).

…

On Feb 7, 2025, at 6:00 PM, Matt Heffron ***@***.***> wrote: there is no point if we don't have at least one BDF font that has a glyph for a particular Unicode I have the BDF file for GNU UniFont which has 57086 glyphs. I found a otf2bdf converter that can create BDF for any OpenTypeFont (e.g., the Noto fonts that @hjellinek <https://github.com/hjellinek> is working with). We can get pretty much any Unicode glyph into BDF. I think the main issue with going outside of the basic plane is to hash using IEQP instead of EQ There's still going to be the issue of how to access these glyphs outside of the BMP in a portable/repeatable way. If we fill all of the "available" space with BMP glyphs/codes, how would/should we make the glyphs in other "planes" available? — Reply to this email directly, view it on GitHub <#2025 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQSTUJKRPSC4G7VB6OMTNQT2OVQKRAVCNFSM6AAAAABWVJYRV2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNBUGQZDMMZZHA>. You are receiving this because you were mentioned.