[Feature Request] Make ZERO WIDTH SPACE have some visual component

[GhbSmwc]

Try sorting this:

https://pbs.twimg.com/media/
https://twitter.com/
​https://pbs.twimg.com/media/

Somehow, the last URL is positioned wrong.

Wait.., there is a zero-width character on the third (before the h in https) which is "​" (select to see that character). I think NP++'s show symbol should also show such characters.

[xylographe]

The character is U+200B (Unicode name: ZERO WIDTH SPACE).

[sasumner]

I think NP++'s show symbol should also show such characters.

What's your opinion on what it should look like?

[GhbSmwc]

A yellow line, much like when you make it show spaces and tabs.

[sasumner]

@GhbSmwc I took the liberty of renaming your issue; hopefully that is okay with you. :-)

[sasumner]

This is going to depend upon Scintilla.

[sasumner]

This was "addressed" by the Scintilla project back in 2013: https://sourceforge.net/p/scintilla/feature-requests/980/

Their response was to add an API so that you can represent any character you want, "specially":
https://www.scintilla.org/ScintillaDoc.html#SCI_SETREPRESENTATION

The representation mechanism is the "white text on black box" variety, much like line-endings look when visually enabled in Notepad++:

Here's an example of what it would look like to set the representation to "" (an empty string), the narrowest possible thing, which is harmonious with a "zero-width" character:

I suppose it is "workable" but I'd say it isn't ideal.

[Daksol]

This request has come up more than once if you look back through the Issues history. I encountered this problem when "screen scraping" data out of certain HTML pages.

If this is not taken up, then maybe add a Caution to NPP interface to say that the option "View-> Show Symbol -> Show All Characters" does NOT show characters such as ZeroWidthSpace and friends.

Only way to detect a Zero Width Space is to move a cursor along a line of text while watching the character position number in the status bar,

[Daksol]

Other issues raised covering the same topic

• Closed Issues raised on the same topic, going back to 2015
  • Show All Characters #5062, Feature Request - Visualize Zero-width spaces in "show all chars" mode. #4731, Show White Space and TAB does not show all whitespace #827
• Open (as at 20-nov-2020)
  • [Feature Request] Display Zero-width non-joiner and Zero-width joiner as their ISO keyboard symbols #8530

[sasumner]

@Daksol

This request has come up more than once

And note that it remains an "open" request

"View-> Show Symbol -> Show All Characters" does NOT show characters such as ZeroWidthSpace and friends.

The "Show All Characters" verbage is historical and comes from a time when "space", "tab" and "end-of-line" characters were the only things that weren't shown. Today that menu item might be "Show space, tab, CR, LF"

The author of N++ has declined my proposal for making such things as you mention visible as a core Notepad++ optional feature, but has kept the door somewhat open by this: #5062 (comment)

You can have these characters visible RIGHT NOW if you are willing to use a Pythonscript to control them.

[donho]

@sasumner

The author of N++ has declined my proposal for making such things as you mention visible as a core Notepad++ optional feature, but has kept the door somewhat open by this: #5062 (comment)

The issue #5062 talked about the BOM, which is visible, but removed by Notepad++, so I won't reopen it since it's not an issue. OTOH, I agree the invisible Unicode character is an issue.
So let's add Show Zero Width Non Breaking Space menu command between Show End Of Line & Show All Character, and make Show All Character command shows also ZWNBS chars.

[sasumner]

@donho

Correct about BOM. BOM should never be visible, except I guess HexEditor plugin? (Not our problem!)

5062 was also used as a springboard for the general discussion of invisible characters.

let's add Show Zero Width Non Breaking Space menu command

But...there are many more invisible UTF-8 characters than that, that users could want.
Hmmm...
Let me come up with a proposal for how to best handle this?

[donho]

Correct about BOM. BOM should never be visible, except I guess HexEditor plugin? (Not our problem!)

Yes, HexEditor does some dirty hacking to show these 3 bytes.

But...there are many more invisible UTF-8 characters than that, that users could want.

Then let's add "Show all invisible character" menu command - I don't think users care about showing which kind of invisible characters.

Let me come up with a proposal for how to best handle this?

OK. Please keep it simple & stupid.

[Daksol]

@sasumner. Thanks for getting this back on the program, so to speak.

Some thoughts on what would be best way of handling hidden characters - this derived from the usecases I was encountering when I first discovered zero width space and friends.

My real world interaction with hidden chars has often been when troubleshooting web pages, urls et al. And what I have often found myself doing is putting a string into (say Excel) and adding formulae to show the Unicode code for each item. That then shows up the non-breaking spaces, zero-width spaces etc. Not very convenient to say the least.

• For the most common hidden, unprintable and whitespace characters
  • Examples: CR, LF, Tab, Space as at present, (suggest adding NonBreakingSpace here)
  • With Show-All-Chars:
    • show as descriptive block - exactly what happens in Notepad++ currently
• For other unprintable chars
  • Examples: Zero width space
  • With Show-All-Chars:
    • either show as dark-shaded rectangular block, and with onmouseover show the hex value
    • or as highlighted group of characters showing including Unicode value, eg %200B

[sasumner]

@Daksol said

Some thoughts on what would be best way of handling hidden characters...

Notepad++ only has limited control over the display; the display is primarily controlled by Scintilla.
The only possible thing that N++ can do with these characters is what is shown with CR and LF in my previous posting HERE.
So any such characters would have to look like those.

[ArkadiuszMichalski]

@sasumner But indicator for this not_visible_chars must be the orange dot (same as for standard space)? I would prefer to somehow distinguish a normal space from the other characters.

[sasumner]

@ArkadiuszMichalski said:

must be the orange dot

Scintilla controls the "orange dot" behavior, for but only for U+0020 character.
The best Notepad++ can do for making any new characters visible is like what it does for CR and LF.
In the default theme this means whiteish characters on a black background.

[sasumner]

So here's a sampling of what could be done, stolen from a forgotten discussion thread on the Community site:

[ArkadiuszMichalski]

OK thx, looks good, at least for me.

[sasumner]

Here's the complete list as of now, that I plan to implement:

For some of these, I just made up an abbreviation so that the representation, the white letters on the black background, doesn't become too big/wide. I don't know if there is a standard abbreviation...does anyone?

Comments are most welcome.

More "simple and stupid" spec'cing out of this feature to come...

[donho]

@sasumner
Are they (in the list above) all not displayed currently in Notepad++ ?

[sasumner]

Are they (in the list above) all not displayed currently in Notepad++ ?

Correct.
I do not know if the list is complete, but all of those currently in the above list are currently invisible in N++.
Unless someone says "You forgot about 'foo' !" we can start with the above list.

[sasumner]

@donho

Here's the file the above image was made from, if you want to "see" how N++ displays the invisible characters:

invisible_utf8_chars.txt

[sasumner]

BTW, here is what is displayed for the invisible_utf8_chars.txt in Windows' notepad.exe :

[donho]

@sasumner
For me FF, NEL, LS & PS are displayed:

My debug info:

Notepad++ v7.9.1   (32-bit)
Build time : Nov  2 2020 - 01:03:56
Path : C:\Program Files (x86)\Notepad++\notepad++.exe
Admin mode : OFF
Local Conf mode : OFF
OS Name : Windows 10 Enterprise (64-bit) 
OS Version : 2004
OS Build : 19041.630
Current ANSI codepage : 1252
Plugins : DSpellCheck.dll HexEditor.dll mimeTools.dll NppConverter.dll NppExport.dll NppXmlTreeviewPlugin.dll 

[ArkadiuszMichalski]

With BOM I got this:

[sasumner]

@ArkadiuszMichalski I see no changes in your two screenshots, except the lines that have been removed from the second file. This is not a problem, I'm just stating it for the record.

[sasumner]

@donho said:

I'm confused... now 2060, 2066, 2067, 2068 and 2069 in both files you provided don't display on my 1st laptop.

5 Unicode chars in both files you provided do display on my 2nd laptop.

That confuses me as well. :-(

[donho]

config.xml matters!

@sasumner
use this one to check:
config.zip

[sasumner]

@donho said:

config.xml matters!

I presume that writeTechnologyEngine="0" in that config.xml is what is important?

---

So this N++ preference setting is going to impact how these characters appear:

To hopefully be very clear, this is how the invisible_utf8_chars_2_w_bom.txt file appears for me with "direct write" unticked:

And here's what it looks like for me with "direct write" ticked:

I'm not at all sure what this means for the future of this feature.

@donho How long is the "direct write" preference setting going to remain? Forever?
I turned it on and it has remained on forever for me.
No downside, only advantages.
To get good data on users that it wouldn't work well for, it should have been made to be on by default, and users could turn it off.
The way it was done, nobody notices it to turn it on and try.
Thus we get no feedback about users that it causes a problem for.

[donho]

@sasumner

How long is the "direct write" preference setting going to remain? Forever?

I did read feedback that in some environments there's the performance issue when "direct write" is ON. So our experience can not be representative for the whole community. Furthermore, I don't want to scarify existing option (which could concern the performance issue) for a new feature which is benefit for less people.

The workaround for me is just treat 2060, 2066, 2067, 2068 and 2069 as invisible chars so there's no ambiguity whether if "direct write" is ON. What do you think?

[sasumner]

The workaround for me is just treat 2060, 2066, 2067, 2068 and 2069 as invisible chars so there's no ambiguity whether if "direct write" is ON

Yes, that can work.
Or, we detect when direct-write is "on" and then don't include those in the list of characters to set a new representation for.

[donho]

@sasumner

we detect when direct-write is "on" and then don't include those in the list of characters to set a new representation for.

Yes, it could work if Scintilla has a message for that. The Notepad++'s option regarding "direct write" in Parameters class cannot be counted on because it's applied to next session.

[sasumner]

if Scintilla has a message for that.

SCI_GETTECHNOLOGY

[sasumner]

Another user request: https://community.notepad-plus-plus.org/topic/20428/highlight-non-breaking-spaces

[sasumner]

A couple of older but related Community links:

• https://community.notepad-plus-plus.org/topic/18900/persistent-highlight-of-characters-e-g-no-break-space
• https://community.notepad-plus-plus.org/topic/15649/visualization-for-zero-width-characters

[sasumner]

One more that this feature would probably help: https://community.notepad-plus-plus.org/topic/20490/invisible-spaces

[alankilborn]

@ArkadiuszMichalski I think the "scintilla dependent" label can be removed from this issue. Scintilla already supports everything needed to allow any normally-invisible character to be shown. I suppose it is left to the programs that embed Scintilla to decide how they want to do this; at least up to now Scintilla has not added a default visual component to more than the "low ASCII" range (NULL, SOH, STX, ..., GS, RS, US).

[Daksol]

@ArkadiuszMichalski - Good to see some movement on this one. fyi There is another Open Issue #8530 which covers the same ground as #8284 which could perhaps also be updated.

• Title:- Display Zero-width non-joiner and Zero-width joiner as their ISO keyboard symbols
• Link [Feature Request] Display Zero-width non-joiner and Zero-width joiner as their ISO keyboard symbols #8530
• There is a more detailed discussion of the issues - and links to a related Scintilla issue (which was closed in Scintilla without action).
• The "scintilla dependent" tag is still attached to 8530 - could you remove that one too? - via same logic

Edit: 25Aug2022 - corrected typo in link

[alankilborn]

Good to see some movement on this one.

What "movement"??
:-)

[Daksol]

This is great, fantastic to get this new feature. Question - can you also close #8530 - which is part of the same Swarm ?? thanks | | | | | | | | | | | [Feature Request] Display Zero-width non-joiner and Zero-width joiner as... This may be similar #8284 https://en.wikipedia.org/wiki/Zero-width_non-joiner https://en.wikipedia.org/wiki/Zero... | | | John Edge ***@***.*** On Sunday, 5 February 2023 at 04:20:57 CET, Don HO ***@***.***> wrote: Closed #8284 as completed via aaab190. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>