Improve F2 text symbol palette

[lasconic]

This PR reorganises the F2 palette. The radio buttons and codepage are gone. The palette now contains 3 tabs.

• One for common symbols, exactly the same one that we used to be
• One for musical symbols, it currently displays the BravuraText font. A list is display on the right for direct access to Harp symbols for example.
• One for the current text font in the selected text. Similar to the All radio button before the change except it also features a list of Unicode ranges.

Feedback welcome. I will cross post on the tech preview forum.

[MarcSabatella]

I love it! I guess it might be nice if SMuFL were to define "human readable" names for the category names. For that matter, would be good to have those for the glyphs themselves.

[mgavioli]

IMHO, the new arrangement makes a lot of sense!

A few suggestions / comments:

• The "Musical Symbols" list at the left side looks "programmish": would it be possible to have the names spelt in a more "human-readable" way? ("accidentalsHelmoltzHellis" => "Accidentals Helmoltz-Hellis")
• The "Musical Symbols" list seems to lack some sections (for instance, all/most of the Medieval/Renaissance ranges are not there or I could not find them); is this intentional? or simply a work in progress?
• The "Unicode Symbols" lists ranges in code point order; this is nice for peoples familiar with Unicode arrangement, but seems hard to understand for everybody else. Would it make sense to sort them in alphabetical order? (I understand alphabetical order is language dependent, but at least this would gather all the "Latin ..." or all the "Arabic ..." ranges together in practically all the languages).
• Does the "Unicode Symbols" table looks for glyphs in some specific fonts? Many ranges for which I have fonts installed are shown as collections of question marks. Examples are: N'Ko, Telugu, Kannada, most CJK ranges in the BMP, ...
• I think the "Uncode Symbols" should not include the 3 Surrogate ranges, as no character can exist there.
• Finally, for "Unicode Symbols" again: for almost anybody on this planet, the range list will include a lot of ranges for which no glyph is installed, which may look disconcerting to the average user (I expect questions like "MuseScore does not work! Many characters are missing!"). I have no practical suggestion, but it might be worth thinking a little about...

[MarcSabatella]

I wonder, maybe even an automated camelcase -> title case converter would be sufficient. Eg, algorithmically turn "thisIsCamelCase" into "This Is Camel Case".

[lasconic]

• For the music symbol categories, I will try a camel case conversion. These categories are coming directly from the classes.json file of SMuFL so we have no control on the translation, the name of the category or the presence or absence of a given category. SMuFL also provides a ranges.json file that could be more suitable since it gives "User readable description". I will implement ranges.json to see the difference.
• The unicode tab uses the current selected font in the text (and the eventual fallback found by Qt). It could be good (or not) to hide ranges without any symbols or even to add symbols if they are not present but I couldn't find an easy way. FreeSerif has quite some symbols since it's supposed to be a unicode font... I will remove the surrogate ranges.

[lasconic]

• surrogate ranges are gone
• Now use the ranges.json file instead of classes.json
• Added a splitter between the category list and the palette.

[MarcSabatella]

Looks really, really good!

"Splitter" meaning you can resize the panes?

[lasconic]

Yes, http://doc.qt.io/qt-5/qsplitter.html#details

[mgavioli]

@lasconic : great! SMuFL ranges look much better now. And I could easily find the ranges I though were missing, which tells things are easier to use.

I still believe that Unicode ranges should be in alphabetical order. Even without messing with the sources, this can be achieved with QListWidgetItem::setData() and QListWidgetItem::data(). For instance, replace https://github.com/musescore/MuseScore/pull/2196/files#diff-d6181d404519b7c488db5c6ea17e15b0R607 with:

  lwu->setSortingEnabled(true);
  for (i=0; i < unicodeRangeNames.length(); i++) {
        QListWidgetItem* newItem = new QListWidgetItem(unicodeRangeNames.at(i));
        newItem->setData(Qt::UserRole, i);
        lwu->addItem(newItem);
        }

Then, replace the first line of TextPalette::populateUnicode() with:

  int row = lwu->currentItem()->data(Qt::UserRole).toInt();

This keeps (most) related ranges together and makes for easier lookup in the list. The only con is that the Unicode tab defaults to "Aegean numbers" (in English), which is likely not the most looked up range. In case, it can be improved.

Testing this uncovered that there are two "Unified Canadian Aboriginal Syllabics" ranges (line 326 and 335); the second should have " Extended" added.

FreeSerif is rather good as a Unicode font, but it is far from complete (as practically all Unicode fonts are: Unicode is really big!); but apparently Qt fallback is not so good, as it fails to find substitutes for many ranges for which I have fonts installed. Might it be some of our general settings about font management? In my opinion, quite concerning are the CJK ranges, which affect a sizeable part of Earth population.

[lasconic]

Regarding Unicode. According to me, the main purpose of this palette is too be able to insert a given character while it's not in your keyboard, or you are not used to enter it. Think about inputting the name of a Russian composer in cyrillic because user wants to be true to the original. CJK users are better served by the native input method of their system, and I hope nobody will type the full lyrics of a song using this unicode table. I think about this tab as a workaround for things we didn't think of putting in the common tab...

[mgavioli]

@lasconic : "the main purpose of this palette..." Exactly! For this you need to find the relevant range and, unless you know the Unicode sequence of ranges, this may not be easy or quick (as there is quite a number of them). I have a decent knowledge of Unicode sequence and still I would prefer to look in an alphabetically sorted list of ranges.

---

"CJK users are better served by the native input method": true. This remark of mine was more about Qt shortcomings, than about the palette itself: if Qt cannot find the characters for the palette, I suspected it will likely be unable to find the characters for displaying the text, however it has been entered.

However, further tests showed that there is no fallback at all: the palette shows exactly the code points supported by the font selected in the text properties widget when the it has been open (select a CJK font -- or whatever--, open the palette and CJK glyphs -- or whatever -- are there!), nothing more and nothing less. This is at least simple to explain and to understand and may even be a useful guide for the user to know in advance if the specific characters he is looking for are supported by that font or not. So, to me this seems a reasonable setup: problem closed, as far as I am concerned.

[MarcSabatella]

I guess if oyu already knwo the name of the range you are looking for, then an alphabetic listing could make sense, but I wouldn't have a clue what I was looking most of the time and would prefer a logically sorted list - which I'd like to think the default list is.

That is, if I am looking for a symbol but don't know the name of its range, I might scroll about until find a likely looking candidate, try it, then if i's not in that range, I would expect other likely candidates to be nearby so I can try them next. I kind of doubt an alphabetic listing would be better for this for than the raw ordering.

But I say this as someone unlikely to use this tab much at all; I'm msotly be using Common and Musical.

[lasconic]

http://shapecatcher.com/ :)

[MarcSabatella]

Extremely cool!

[mgavioli]

@MarcSabatella : the Unicode range sequence did grow organically and it is practically meaningless: CJK Unified (first) Extension is before CJK Unified, while many other extension are after the 'main' range, sometime much later; Latin extensions (there are many of them) are all over the BMP and some even in other planes; non-alphabetic ranges (punctuation, symbols, ...) and minority scripts are also randomly placed. Possibly there was a rationale in Unicode 1.0, but since then ranges have been put where there was room for them.

I have extensively used Windows built-in (once upon a time), Babel and Linux Gnome character maps: the first two are by range sequence, the last by alphabetic sequence and I am pretty sure the latter is easier to use.

(EDIT: as an example try to locate the Welsh ẃ glyph, not to look too much away...)

But, if I am the only one to feel this needing, then it possibly fulfil specific usages...

Finally, you say you will not use that tab so much; "appetite comes while eating" we say in Italian: I use quite often the (Linux built-in) character map for punctuation (dashes, quotes, guillemets, ...) and PUA, even if for the most comm I have made my own XCompose macros and I will likely switch to MS own palette once merged; you possibly will find your own uses too...