Alternative way to access character variants?

[jsbien]

Hi!
I will use the font primarily with XeLaTeX which supports OpenType features, so I expect no problems.
However I would be happy to use it also in indexes such as https://github.com/jsbien/Zaborowski-index4djview, which are displayed by a practically orphaned program written in QT; even if QT supports the features (I don't know), the program definitely does not .

So the question (showing my ignorance about fonts...) is: does a character variant have also a code which can be used to access it? If not, at least if not automatically, can you make a policy to provide such access codes for all/some character variants?

[psb1558]

I've looked at index4djview, and while I'm not sure I understand everything about it, I get the problem of displaying complex text in an older application, or in an app with limited OpenType support (like MS Word). I hope you don't mind if I offer a bit of theory here.

In JuniusX, you can divide the character variants into three groups: those with Unicode code points, those with MUFI code points (from the Unicode Private Use Area), and others. You need a code point to enter a character variant directly (without invoking an OpenType feature).

In theory, an app should handle a character variant with a Unicode code point correctly, since Unicode data is built into the app. For example, if I enter this sequence directly:

with insular f (U+A77C) and d (U+A77A), generate a PDF, and open it in Mac Preview, I can find the word by searching on "findan" (with plain f and d).

If I directly enter a MUFI code point, e.g. "Whan" with U+F203, the gothic variant of a

I can't find the word with a search for "Whan," because a code point in the PUA has no data associated with it—it has no meaning to the app. In this way, the use of MUFI (PUA) code points introduces usability and accessibility problems.

JuniusX gets around these problems by preferring unencoded character variants, even for MUFI characters. For example, JuniusX has two versions of the gothic a (since duplicating a character requires only a few bytes), one with MUFI code point U+F203 and one without. When you use OpenType feature cv01[4] to produce the gothic a, you get the unencoded version, which inherits the encoding of the original (plain) a. And so the problems with searching and accessibility are solved. This is the biggest reason why the non-MUFI character variants (like the variants of ø on cv21) have no associated code points.

I could duplicate these characters the way I did for the MUFI variants, supplying versions with PUA code points, but doing that would introduce problems of its own: first, the very real possibility of a conflict with MUFI somewhere down the road, if the character was adopted by MUFI and assigned a different code point or if MUFI decided to use that code point for another purpose; second, encouraging the use of non-searchable, non-accessible code points. So I prefer not to do this.

The good news is that when text encoded in the way JuniusX recommends is displayed in an older app, the Unicode character is usually an acceptable fallback for the variant. If your text has a variant of ø, the older app will display the plain ø. This isn't ideal, but it may be good enough.

Sorry for the longish post. I'm starting to outline an article going into problems of accessibility and their implications for MUFI, and I hope I can explain things more clearly there.

[jsbien]

On Fri, Dec 11 2020 at 6:27 -08, Peter Baker wrote: Thanks for the explanation. [...]

I could duplicate these characters the way I did for the MUFI variants, supplying versions with PUA code points, but doing that would introduce problems of its own: first, the very real possibility of a conflict with MUFI somewhere down the road, if the character was adopted by MUFI and assigned a different code point or if MUFI decided to use that code point for another purpose;

I don't think the danger of a conflict with MUFI is serious. With Google I found the proverb "the early bird catches the worm". If you use a PUA character before MUFI, at least in principle this is the MUFI problem :-) More seriously, I have a question: are all PUA planes 15 and 16 practically usable? Looks like nobody uses them yet, so there is no risk of any conflict.

second, encouraging

You can provide them but at the same time deprecate them. That's what Unicode is doing for some characters. Assigning the characters to planes 15 or 16, if technically possible, makes them probably so inconvenient to use that deprecations is automatic :-)

the use of non-searchable, non-accessible code points.

I understand you refer to the limitations of some software. I guess in Emacs I can search and acces every character. On the other hand for some applications searchability doesn't matter, as the search is provided by separate indexes or by other means.

So I prefer not to do this.

I will be happy if in some future you will reconsider your position. But thanks to the font free license I don't intend to press you, as a fork with additional features can be created if really needed.

The good news is that when text encoded in the way JuniusX recommends is displayed in an older app, the Unicode character is usually an acceptable fallback for the variant. If your text has a variant of ø, the older app will display the plain ø. This isn't ideal, but it may be good enough.

For me it is not good enough. I have created indexes to historical texts and displaying historical variants by djview4poliqarp would be a great advantage. On the other hand it is quite possible that the user base is just me, so my needs are definitely less important than those of users of Words etc.

Sorry for the longish post.

It's OK.

I'm starting to outline an article going into problems of accessibility and their implications for MUFI, and I hope I can explain things more clearly there.

I will wait patiently for it.

[psb1558]

Can djview4poliqarp handle characters in planes 15 and 16? I understand that a lot of apps can't handle codes above U+FFFF. If it can, I'd be happy to encode the characters you need in those planes, where there's little possibility of conflict with MUFI.

(Long ago I had to change the code points of quite a few characters in Junicode because of conflicts with MUFI, so I take this potential problem seriously.)

[jsbien]

I think it is best tested empirically. Would you be so kind to prepare a mini-font with a character in plane 15 and/or 16?

[psb1558]

Not quite mini, but I've dropped two subsetted files, JuniusX-Regular-subset.ttf and JuniusX-Bold-subset.ttf, into the fonts directory. They've got the four unencoded variants of ø on cv21, as before, but also duplicated with code points U+F0000-U+F0003. I'll leave them there only till you say you've got them, and then I'll delete them.

[jsbien]

Thank you very much, but I don't see them. I guess you haven't commited or pushed them.

[jsbien]

I downloaded them, so you can already remove them. I will let you know soon about my test results.

[psb1558]

Oops. I had to resolve a confict. They're there now.

[jsbien]

To my pleasant surprise djview4poliqarp supports the plains.

[psb1558]

Since we're now talking about substantive changes to the font, I've opened an issue for this.

[jsbien]

Is it technically feasible to add automatically (e.g. with some script) all the variants to, say, plane 15 without any selection? This would be the simplest way when the variant are stable. I see only potential problem when a new variant is added. It should be added at the end of the block, so some index or a log would be needed.

[psb1558]

Possibly, but as the number of variants is (probably) not large, and the script would be used only once, I'm not sure it's worth the research and labor of writing it. I'm going to filter the glyphs in the font down to a likely set and just look through the set. I'll document everything I encode--first by noting it in the issue I've opened, and later by producing a document.

[jsbien]

PUA_test.txt
Debian Gnu/Linux stable (buster), vim 2:8.1.0875-5:

[psb1558]

I've just posted the complete static font with those few plane 15 characters in it, so you can get rid of the subsetted font. It will take a while to do the rest, since this is a busy season.

[jsbien]

Thank you very much. F0001, F002, F0003 and F000A look for me as the same character - is this intentional?
UPDATE: it's OK, just a stupid mistake of mine. It's OK for me as it is quite sufficient for testing, but I'm just curious.
I will wait patiently for the rest.

[psb1558]

No, they should give you the four different variants of ø. They are: F0000, F0001, F0002, F0003. This works in Aquamacs. I'll try it in Ubuntu/Emacs when I get a chance.

[psb1558]

F0000 and F0003 are different in your screenshot above.

[jsbien]

Sorry for bothering you, it was just a stupid mistake of mine.

[jsbien]

I'm unable to display F0004-F0018 neither in Emacs not in vim :-(
Anyway I would like to name the characters for use with Unihistext. I enclose my proposal for F000-F000C. One aspect is the naming of specific characters, another the general naming policy,
PUA_test.txt

[psb1558]

I thought U+F0000-U+F0003 were lowercase. They're sized that way, anyway. Are they supposed to be capital, or should there be matching capitals?

U+F000D-U+F0013 fill out MUFI's collection of letters with high overline, for use with roman numbers. Likewise U+F0014-U+F0016 fill our MUFI's collection of letters with medium-high overline, also for use with roman numbers. U+F0017 fills out MUFI's collection of alphabetic superscripts, and U+F0018 is a variant of eth for use with certain diacritics.

I notice now that I haven't added capitals yet.

[jsbien]

On Sun, Dec 13 2020 at 9:27 -08, Peter Baker wrote: I thought U+F0000-U+F0003 were lowercase. They're sized that way, anyway. Are they supposed to be capital, or should there be matching capitals?

Sorry, again a stupid cut-and-paste mistake of mine :-( They should be lowercase as they are now.

U+F000D-U+F0013 fill out MUFI's collection of letters with high overline, for use with roman numbers. Likewise U+F0014-U+F0016 fill our MUFI's collection of letters with medium-high overline, also for use with roman numbers. U+F0017 fills out MUFI's collection of alphabetic superscripts, and U+F0018 is a variant of eth for use with certain diacritics.

OK, I will add the names tomorrow. I still has doubt about the proper naming policy, but I will come back to this later.

[psb1558]

These names are useful if you're making a database for use by people designing XML schemas or the like, and I would also like to include them in the documentation, but they don't go into the font, which uses boring naming conventions like "uF0000".

I need to add capitals, I see. I thought I had done that.

[jsbien]

In MUFI the overline is over capitals, your glyphs look to me like minuscules. Personally I don't mind, I'm just curious.

[psb1558]

MUFI has a lowercase l with high overline (U+E58C, presumably for the purpose of clearing the top of the letter). At some time in the past—long ago, I think—I decided to fill out the collection of letters (at least those used in roman numbers) with high and medium overlines so that if one wanted a roman number like either of these

one could do it (with stylistic set ss04 or ss05). I don't know whether it was a good idea or not, but there it is. JuniusX has a choice of high or medium overline for lowercase letters used in roman numbers, and then the MUFI collection of capitals with high overline, e.g.

[jsbien]

I enclose my attempt to name the characters. As you can see, I was unable to identify U+F000D and U+F0015. I of course focus on characters needed for Polish. Comments are welcomed.
JuniusX_test.pdf

[psb1558]

This looks great, and thanks very much for doing it! A few notes:

1. I have added U+F001E and U+F001F, the "slightly different" variants you mention for U+F0000 and U+F0001. These were an oversight. (Updated font to be posted later today.)

2. U+F000D is a variant of j for when the letter is preceded by g or certain vowels with ogonek. If I remember correctly, I was advised a number of years ago that this was needed to prevent awkward collisions in Lithuanian. In most apps the substitution will be handled invisibly by calt (Contextual Alternates). (The story with U+F000B is similar: calt substitutes it e.g. before vowels with dieresis.)

3. U+F0015 is superscript q, that being the only letter of the basic Latin alphabet without a superscript variant in Unicode. This and other superscript letters and numbers are produced by Opentype sups (Superscripts).

4. U+F0011-U+F0014 are small letters

5. U+F0011 is showing as A with stroke in the document.

[jsbien]

Thanks for the corrections, I again made a lot of cut and paste error :-(. Other comments I will take into account soon. I will post the new version here, but FYI I keep the backup of (quick and dirty) sources at https://github.com/jsbien/unicode4polish.

[jsbien]

Next draft:
JuniusX_test.pdf

[jsbien]

Are the overline variants accessed only by ss04 i ss05?

[psb1558]

So far. I didn't put them in the Character Variants (cvNN) because I assumed they'd always be used together, as in roman numbers. But I could.

Of course, they all have code points now, and that's a method of access too.

[jsbien]

It's OK for me. I asked to be sure how to describe them in my note.