Show ⠀, ⠿, ⣿, a question mark or the hexadecimal value (Liblouis 3.8 style, HUC8, HUC6) for undefined characters in braille

[DrSooom]

Example character:

♯ (U+266F, music sharp sign)

Actual behaviour:

• de-de-comp8.ctb: ⠠⡌⠭⠣⠫⠫⠋⠠ ('\x266f', 8 cells) is shown for this character.
  Workaround: Add "sign \x266f 3456" at the end of this table and restart NVDA. And don't forget that the very last line of those files must be blank.
• fr-bfu-comp8.utb: ⠀ (no dots, 0, one single cell) is shown for this character.
• UEBC-g1.utb: ⠼⠩ (dots 3456 and 146, two cells) is shown for this character.
• Other braille tables weren't tested.

Expected behaviour:

• Add a combobox called "Show undefined characters as" in the Braille NVDA Settings with the following five values: (Please see the update below too.)
  • ⠀ (U+2800, no dots, 0)
  • ⠿ (U+283F, dots 1-6, 123456)
  • ⣿ (U+28FF, dots 1-8, 12345678)
  • Question mark [undefined characters are shown as a question mark, different dots depending on the current used braille table]
  • Hexadecimal Unicode value [⢸**** and ⢸*****, ⢸ = U+28B8, * = value between 0 and f]
    Note: Please see the update regarding this entry below too.
• The rules in all braille tables are overwritten by this new function.
• We have to test this new function also with multi-byte characters higher than U+FFFF like this smiley emoji 😀 (U+1F600). In the German 8-dot table this smiley is shown as ⠠⡌⠭⠙⠳⠩⠙⠠⠠⡌⠭⠙⠑⠬⠬⠠ ('\xd83d''\xde00', 16 cells just for one single character).

Update 2019-03-01 17:02 CET:

The fifth value in the combobox called "Hexadecimal Unicode value" should be replaced with the following four values:

• Extensive hexadecimal value [Liblouis 3.8 style for undefined characters like '\xhhhh']
• HUC8 Braille Tables [displays ⣭⡤⣺ or ⣥⣆⡉⣥⢂⣺ for the Unicode character "Grinning Face" (U+1f600)]
• HUC6 Braille Tables [displays ⠿⠤⠵⠺ or ⠿⠆⠛⠆⠿⠂⠼⠞ for the Unicode character "Grinning Face" (U+1f600)]
• Use the Braille Table behaviour [no changes to the actual behaviour]

See also:

PR #9044 and the documentation about the Hexadecimal Unicode Characters Braille Tables (which is the result of liblouis/liblouis#688 and liblouis/liblouis#689)

System configuration:

NVDA installed/portable:

Both

NVDA version:

NVDA 2013.2 (as far as I can remember based on the German 8-dot table) to 2018.3 Beta 3

[LeonarddeR]

Output like '\x266f' is generated by liblouis for undefined characters. Working around this within NVDA is very complex, as it involves a recalculation of several variables received by liblouis, including the input character to braille output positions. I don't think that this would be the right way to do. I'd rather see liblouis include something like a replacement character for undefined characters, but that would mean a backwards incompatible change to the lou_translate. @dkager, thoughts?

marking this as "needs external fix for now". aving said that, pretty sure that there is already a similar ticket.

[dkager]

Cases 1-3 and 5 can already be accomplished with minimal effort. Case 4 is the tricky one.
I wonder if overwriting the behavior of all braille tables might "violate" braille standards that specify their own sign for characters outside of that standard.

[LeonarddeR]

@dkager commented on 1 Sep 2018, 14:24 CEST:

Cases 1-3 and 5 can already be accomplished with minimal effort.

Could you elaborate on how to accomplish this?

[dkager]

From the manual:

undefined dots
If this opcode is used in a table any characters which have not been defined in the table but are encountered in the text will be replaced by the dot pattern. If this opcode is not used, any undefined characters are replaced by '\xhhhh', where the h’s are hexadecimal digits.

[DrSooom]

GitHub replaces 😀 and 😀 automatically with a graphic, unless they are putting within a code section. So I updated it above a few minutes ago to get these emoji replacements out of my text.

btw: On iOS 10.3.3 the German table shows the English question mark (⠹, U+2839, dots 1456) for undefined characters and two question marks for emojis and other characters higher than U+FFFF. The German question mark is shown as ⠢ (U+2822, dots 26) in NVDA/Liblouis and on iOS and Fire OS (BrailleBack). I guess that BrailleBack is also using Liblouis, but I'm not sure.

Update 2019-03-01 17:04 CET:

Liblouis is part of BrailleBack. I took a look into the appk file and found its tables there.

[DrSooom]

Related issues: #537, #2949, #5091 and #5763

[LeonarddeR]

@dkager commented on 1 sep. 2018 15:03 CEST:

From the manual:

undefined dots
If this opcode is used in a table any characters which have not been defined in the table but are encountered in the text will be replaced by the dot pattern. If this opcode is not used, any undefined characters are replaced by '\xhhhh', where the h’s are hexadecimal digits.

Could we throw this opcode at liblouis at runtime? I assume we can compile it with lou_compileString, but then, how do we include it in our translation routines?

[dkager]

You can do that, or write a small supplementary table to include after the real translation table.

[LeonarddeR]

It looks like lou_compileString works ok to change the undefined opcode, however it seems difficult to reset it to the default, other than by calling lou_free.

[dkager]

Did you try passing just undefined?

[LeonarddeR]

@dkager commented on 8 sep. 2018 20:33 CEST:

Did you try passing just undefined?

Jup, negative. Also tried undefined =

[dkager]

I thought the empty opcode was supposed to reset it. Maybe we should build that.