Hyphen displayed as "tofu"

[ghost]

Bug report

Describe the bug

Here is a clear and concise description of what the problem is:

Music files with tag information, containing a hyphen, will not be displayed correctly. Not the hyphen but "tofu" (the empty box, □) will be displayed.
N.b.: Enabled tag reading (Settings / Media / Music) is active.

Example for tag data with hyphens in MusicBrainz: "Fu‐Gee‐La" does not use ordinary hyphen-minuses ('-', ASCII 0x2D) but the more correct hyphens ('‐', U+2010).

It's on Kodi 18.3 with default skin (Estuary), running in Windows 10 (German).
The problem only exists with the default language font of the skin. With Arial based font the hyphen will be displayed correctly.

Expected Behavior

Here is a clear and concise description of what was expected to happen:

A hyphen (U+2010) is a common symbol and should be displayed correctly.

Common punctuation and special characters should be displayed in any case.

Actual Behavior

Not the hyphen ('‐', U+2010) but "tofu" (the empty box, □) will be displayed.

Possible Fix

Update the default font, Noto Sans, to a recent version.

After some investigations, I found that the default font, NotoSans-Regular.ttf, version 1.06, doesn't have the symbol for the hyphen. This version of the font file seems to be outdated.
In the most recent version of the font, downloaded from [Google] (https://www.google.com/get/noto/), out of "NotoSans-hinted.zip", version 2.0, it has the symbol for the hyphen (and also other new symbols like stars [filled and unfilled]).

After replacing the font files (C:\Program Files (x86)\Kodi\addons\skin.estuary\fonts\) with recent versions the hyphen will be displayed correctly.

Note: With the new fonts (NotoSans-Regular.ttf and NotoSans-Bold.ttf copied to C:\Program Files (x86)\Kodi\addons\skin.estuary\fonts\) the hyphen is correct but the bullet operator (U+2219?) in the "Sort by:" line is missing (is "tofu")!
Maybe it's not so easy to replace only the font file. Or maybe I've used the wrong files. ...
The bullet operator (U+2219) is in the old font file, but missing in the new file! It's in
NotoSansSymbols2 (now)!

To Reproduce

Steps to reproduce the behavior:

1. Tag a music file with a hyphen (e.g. for the title, like in "Fu‐Gee‐La" (see MusicBrainz)
2. Run Kodi with default skin and tags from files.
3. You should see "tofu" instead of the hyphen in Kodi.

Debuglog

The debuglog can be found here:
kodi.log

Screenshots

Here are some links or screenshots to help explain the problem:

Missing hyphens:

Comparison of the fonts:

Kodi with new font files:

Additional context or screenshots (if appropriate)

Here is some additional context or explanation that might help:

With Arial based font I have the hyphen and also the bullet operator at "Sort by:".

Your Environment

Used Operating system:

• Android

• iOS

• Linux

• OSX

• Raspberry-Pi

• Windows

• Windows UWP

• Operating system version/name: Windows 10 Professional, 64 Bit, v1903

• Kodi version: 18.3

note: Once the issue is made we require you to update it with new information or Kodi versions should that be required.
Team Kodi will consider your problem report however, we will not make any promises the problem will be solved.

[DaVukovic]

I have to check at home later for myself. I own that album as well, but I'm not on Windows at home. But I'll check how this album has been tagged as I'm using MusicBrainz as well.

Example for tag data with hyphens in MusicBrainz: "Fu‐Gee‐La" does not use ordinary hyphen-minuses ('-', ASCII 0x2D) but the more correct hyphens ('‐', U+2010).

I'm not that deep into those kind of things. But who says that the one or the other is more (or less) correct? Let's assume MusicBrainz will probably change the hyphen they use to something Kodi is able to work with the current font.

Updating the fonts might be a different story. The "dot" which is then shown as tofu could also probably be changed. Maybe @ronie or @HitcherUK could share an idea if we could/would consider to change the font and probably use something else for that "dot".

[ghost]

I'm also not very familiar with typographic "rules" but maybe the Unicode Consortium is the master (of disaster).
As we can read on Wikipedia: "The hyphen (‐) is a punctuation mark used to join words, ..." but often (most of the time) the hyphen-minus will be used. The hyphen-minus is hyphen and minus sign, but it's only a compromise from times of (7-Bit) ASCII.

It's a little bit like with typographic and typewriter signs (like apostrophe, prime, etc.) and the correct way of using them.

It's not a play of MusicBrainz which symbol will be used. The user chose the symbol at entering the data (artists, releases, tracks). Often this data will only be copied and, I guess, most users don't know which type of punctuation they copy. And most of the users don't know the difference between a hyphen and a hyphen-minus.

Because the bullet operator is in NotoSansSymbols2 (now), it seems that it's not enough to use just one TTF-file.

Note: In MusicBrainz Picard you can enable or disable the convert from Unicode punctuation to ASCII. Only with disabled converting you will get the "correct" typographic symbols (hyphen, apostrophe, etc.).

[Hitcher]

I'd suggest we simply update that font file (and any others that need it) and replace the bullet operator with a bullet.

[ghost]

I'd suggest we simply update that font file (and any others that need it) and replace the bullet operator with a bullet.

This sounds not bad but I'm not sure, if this is as easy.

In the new font files (downloaded from Google) there are also other symbols missing, like the black star (U+2605) and white star (U+2606). The fix of #12954 will not work with this new font files!
In addition, also many other symbols are missing now in the recent version from Google.

I'm not sure about the font files used by Kodi/Estuary. Are these "official" versions from Google? Or are this files modified and other symbols have been added manually?
If the files have been modified, it seems to be a kind of lack, because you cannot easily upgrade to a new version from Google.
I don't know if it's easily possible to use more than one font file at once, the Regular (and Bold) and also the Symbols and/or Symbols2. I suspect that this might be a bigger change.

Note: This might also affect the skin Estouchy.

[ronie]

The fix of #12954 will not work with this new font files!

i haven't looked into it yet, but out of curiosity.. why wouldn't that work anymore?

[ghost]

The fix of #12954 will not work with this new font files!

i haven't looked into it yet

Pls., look into #12954 first and, I guess, it would be clear.

[ronie]

if it was i wouldn't have asked...
what am i missing?

[ghost]

if it was i wouldn't have asked...
what am i missing?

What do you want?
You nag around, without reading the linked issue. To read the linked issue is the first thing, you have missed!
If you are not willing to contribute to this issue seriously (and reading is the first mandatory task), it might be better to keep out.

[Hitcher]

@NeroRome You've linked a PR not an issue.

[ronie]

ok, please accept my apologies for trying to help out..

/unsubscribe

[ghost]

You've linked a PR not an issue

Okay, it's not an Issue, in kind of the GitHub entity type, but technically (as GitHub entity) it's a Pull request.
It's just an issue in kind of the sophisticated meaning of the word (problem, case, etc.). There is no Issue in GitHub for this Pull request but only the issue in the forum (linked in the request).
Therefore, basically, my postings are not wrong.

The Pull Request #12954 is also very short and easy to understand (and it's not necessary to read the linked thread in the forum).

However, if people are not willing and not able to read (and understand) just two sentences...
...for me it makes no sense to continue...

---

I've created this detailed Issue, I suggested possible solutions (incl. a lack with the bullet operator), I've hinted to a lack with just updating modified font files to a recent version from Google (with #12954 as example).
Now, after one week, we have no progress, one pissed-off and we are talking about what an issue is.

[DaVukovic]

Don't like that attitude at all if a dev wants to help out.

And what also might be relevant (probably)...the linked PR is not even merged. The PR regarding that topic which is merged is: #12956

Anyway, I'm out here as well, as I can't help.