non visible characters are displayed on braille displays, how to avoid that?

[Adriani90]

Steps to reproduce:

Reported by several users on different groups.
i.e.

1. Use a braille display and open the twitter website
2. Switch to focus mode with nvda+space
3. Navigate through the tweets with j and k and see what is displayed on the braille display

Actual behavior:

Characters like '\xfffc' (object replacement character) or 'XFeFF' (zero with no break space) are displayed on the braille display. These and more such characters are not only displaed on Twitter but also on Windows explorer and other programs.

Expected behavior:

Such invisible characters should not be processed by the braille displays.

System configuration

NVDA installed/portable/running from source:

Installed and portable

NVDA version:

2019.3 Beta 1, 2019.2.1, 2019.1.1

Windows version:

mainly Windows 10

Name and version of other software in use when reproducing the issue:

Firefox, Google Chrome, Windows explorer and others.

Other information about your system:

Other questions

Does the issue still occur after restarting your PC?

yes

Have you tried any other versions of NVDA? If so, please report their behaviors.

no

[Adriani90]

cc: @bramd, @dkager, @Andre9642, @LeonarddeR

[DrSooom]

U+FEFF is displayed in Firefox if an UTF-8 with BOM encoded html/php file is included. This isn't the case if you include UTF-8 without BOM encoded html/php files. Eliminating this Unicode character has to be done in my opinion by the web browsers – but always as an option. This is NOT a Liblouis or braille issue. These Unicode characters are also sent to the TTS.

See also: https://en.wikipedia.org/wiki/Byte_order_mark#Byte_order_marks_by_encoding

PS: I guess that there already exist similar NVDA issues regarding these Unicode characters. Maybe I find them later.

[Adriani90]

However, in my view it would be cool if there is a possibility to avoid displaying these characters from NVDA or Liblouis perspective. You can also see it as a feature if you like, for me it is a bug because the braille translates characters which are not visible for sighted people for example. This is also how we treat pronounciation of non visible elements.

[DrSooom]

@Adriani90: I guess you missed this sentence, which I added to my previous comment a few minutes later after publishing.

These Unicode characters are also sent to the TTS.

Thus I suggest to add a new list with checkboxes called "Ignored Unicode code points". Example Unicode code points: U+00FF, U+200E, U+200F, U+FEFF, U+FFFC and U+FFFE

I'm not sure how we should implement such a list. Shall we make it generally for all TTS and all braille tables/displays at the same time or separate lists for each TTS and braille table/display? All these checkboxes should be enabled by default, as non-technical users won't miss these Unicode code points. But I would miss them! And web developers would also miss them as they need them to check if html/php files are correctly included.

[DrSooom]

@Adriani90: Please also read question No. ⠣ in the HUC Braille Tables documentation.

[Adriani90]

It would be good if some people chould share their experiences on this. In my view this is kind of annoying when working with a braille display and such non visible characters are using the limited space on the display.

[gregjozk]

just a suggestion for this issue:
create ticket/pr at liblouis with these characters presented with virtual dots (from 9 to f) globally, so it would be included in all tables.

[DrSooom]

@gregjozk: Your suggestion would also influence all end applications in which Liblouis is included. I wouldn't want to do that.

[bramd]

I'm quite sure related issues have been opened earlier, but I can't find them right now.

There are a few solutions to this:

1. Map more characters to braille dot combinations. This should be done in liblouis and its' tables
2. Liblouis has the undefined command to define a dot pattern that should be shown for unmapped characters. In theory, liblouis can accept a list of tables to process and we could provide an NVDA preference for this and generate the appropriate table containing this command. We also need a UI to see which character has actually been replaced with the dot pattern

[michaelDCurran]

Agreed that seeing things like \xfffc is annoying and a waste of space on a display. However, these characters are still valid, and need to take up one braille cell at an absolute minimum, so that they can be routed to. Of course in the cases where they are only presentational within non-interactive text, then they should be stripped much earlier in NvDA, before it even reaches braille.
But for this, we need very clear real-world use cases of where they appear.