Format-Hex should not try to render unicode control characters that affect the display

[SteveL-MSFT]

Format-Hex currently filters out control characters in the ASCII range that would affect the console display. Needs to be updated to handle Unicode control characters. Also consider replacing use of period for non-printable characters with the Unicode symbol for non-printable to avoid confusion with actual periods.

[iSazonov]

Is "`u{00}" expected output format?

[SteveL-MSFT]

Any unprintable character (beyond the control characters) should be replaced with the box with question mark symbol.

[ThreeFive-O]

@iSazonov I think @SteveL-MSFT means this one:

U+FFFD � REPLACEMENT CHARACTER used to replace an unknown, unrecognized or unrepresentable character

from https://en.wikipedia.org/wiki/Specials_(Unicode_block)

[iSazonov]

@ThreeFive-O Thanks for clarify.

Will this symbol be well displayed in the Windows 7 console with default config?

[iSazonov]

With Raster fonts (default on Windows 7) the replacement symbol is not displayed.
On other Windows (by default) the symbol is displayed as an empty square.

It does not look good.

[SteveL-MSFT]

We might have to special case Win7 and do something like detect if the font can display and and maybe just show a question mark.

[iSazonov]

Determining the capabilities of a font at runtime looks impossible.
We could add new parameter ReplacementCharacter with standard default U+FFFD � REPLACEMENT CHARACTER.

[SteveL-MSFT]

@iSazonov does U+FFFD render as whitespace on Win7? That might be good enough since the telemetry shows that only a minor % of customers are on Win7 and it's probably a small set of those customers using format-hex.

[iSazonov]

does U+FFFD render as whitespace on Win7?

Raster font is default on Windows 7 console and the symbol is displayed as whitespace. User have to select TrueType font to see the symbol. I personally always do this.
If Unix consoles use TrueType font by default I think we could use the symbol.

Also I tried Char.IsControl() in code you link and get surprised results. It is again a problem on Windows 7. I don't know can we accept this for new Windows version and Unix-s.

[SteveL-MSFT]

I think this should be pretty straight forward as I expect the unicode control characters to be documented on the internet and just needs to be added to the already existing filter out list