ANSI escape handling and PowerShell #13071
Comments
SteveL-MSFT
added
Issue-Enhancement
|
Possibly stupid question here: Will I be able to use |
|
@adamskt good question. I don't see any reason why you couldn't serialize |
|
So in summary. I liked the early draft of this idea, and I like this even more. A simple thumbs up wasn't enough :-) Since common parameters set the preference variables for the local scope, what would your view be on adding a common parameter which sets output-rendering for a single command ? Among other things that would mean if one really hates the authors choice of colors a simple entry in $PSDefaultParamterValues gets rid of it. I also wonder if more choices should be available so that a user can set their own warning, error, emphasis styles and the author then says "format this as a warning" not "use orange text" |
|
If PowerShell is going to tackle theming I'd really really like to see some extra settings that a TUI might be able to tap into. Taking bootstrap as an example, I'd like to see the styles be under names like:
Every time I've started making a TUI of any sort, trying to make something that looks nice and also keeps to the spirit of the user's color choices is very challenging. |
|
Re terminology: I wonder if we should use "Virtual Terminal (VT) [Escape] Sequences" rather than "ANSI escape codes" in order to avoid confusion with the "ANSI" character encoding - though, arguably, that confusion is more likely for Windows users and, conversely, Unix users may be less familiar with the term "Virtual Terminal". FWIW, looking at a few Avoiding "ANSI" would also mean: Re output modes (controlling when to use the sequences, Note that GNU
GNU macOS/BSD In short: I think these modes are sufficient and I think they're worth adopting - including the terminology. Use of environment variables, new CLI parameter, new common parameter to control coloring: We should consider exposing setting Update: #10811 (comment) mentions @lzybkr proposing a new CLI parameter:
@jhoneill above suggested a new common parameter for cmdlet-individual control (caveat is that unless this parameter is opt-in, conflicts with existing user-defined parameters are possible). |
This comment has been minimized.
This comment has been minimized.
|
This already seems kind of similar to what @Jaykul has done with PANSIES, only less flexible and harder to work with. In that module, a PSProvider is added that is capable of generating escape codes for common (named) colors as well as arbitrary hex color values. That can then be used mid-string to insert values in a flexible way. If users are still going to have to look up the escape sequences for these settings / download a third party module to handle that anyway, I'm not sure this really adds much. Having a base theming framework is a good start, but without some more user-friendly features (VT escapes are not user friendly, period) I don't see it getting a ton of use. |
|
@mklement0 you're right, had the wrong URL in my clipboard at the time. Fixed. |
|
@vexx32 the |
|
@SeeminglyScience I think those are interesting style names and can see how they may be more future proof. Were you thinking those would replace the ones I have under the |
|
@vexx32, as implied by @SteveL-MSFT's comment, unless you want to redefine the default colors (which probably isn't a good idea) or redefine the However, in order to better support these - advanced - use cases, perhaps we could extend |
The way I have it in my head is something like this: class Theme
{
public Style Default { get; set; }
public Style Primary { get; set; }
public Style Secondary { get; set; }
public Style Success { get; set; }
public Style Danger { get; set; }
public Style Warning { get; set; }
public Style Info { get; set; }
public Style Light { get; set; }
public Style Dark { get; set; }
public Style Muted { get; set; }
}
class Style
{
public Color Foreground { get; set; }
public Color Background { get; set; }
public Decoration Decoration { get; set; }
}
class Color
{
// Maybe just the RGB value here.
public bool TryWriteAnsiPrefix(Span<char> buffer, out int amountWritten, bool isBackground = false);
public bool TryWriteAnsiReset(Span<char> buffer, out int amountWritten, bool isBackground = false);
}
class Decoration
{
public bool IsUnderlined { get; set; }
// etc
public bool TryWriteAnsiPrefix(Span<char> buffer, out int amountWritten);
public bool TryWriteAnsiReset(Span<char> buffer, out int amountWritten);
}Where I realize this is a drastically different design in a few places that aren't relevant, it's just for illustration purposes. |
I assume this means, as also implied by @jhoneill's comment, that the engine will have to check every output object for whether it is a string and, if so, whether it contains VT escape sequences - do we need to worry about performance, especially with sophisticated detection (see below)?
I suggest a more sophisticated test, along the lines of using regex At any rate, we need to clearly document what escape sequences we do detect as |
|
$PSStyle.OutputRendering would also solve #12703 |
We forget about Jason's PSMore idea. Formatting system based on the idea will be great. I do not think that it will require more efforts than this PSStyle proposal. Also I'd think about generalizing of the rendering - |
|
The API looks good to me, but the proposal seems to be focused a lot on creating "decorated strings" for the pipeline and doesn't talk about the distinction between formatting and pipeline data. Colors are imo a formatting concern and should generally not be in strings in the pipeline. The API can of course be used inside formatting files, so there isn't a problem with the API, I just wished the proposal here targeted that use case explicitly and less so putting strings with ANSI sequences into the pipeline. Some people will do that of course, and that is ok, because the distinction doesn't matter for every quick shell script hacked together. But I expect PowerShell itself and many modules to use this feature the most, and they'll want to do it the proper way. Since this issue is very broadly named "ANSI escape handling and PowerShell", I'd also like to point out that there are some big issues with the way the formatting system handles color escapes that would need to be solved too (as others have also pointed out): #7744 (comment) Some comment on the API: It also seems like it would make sense to add Overall would love to see such an API though! |
YES! I have marked myself out the position of "the curmudgeon who wants monochrome" but data has no colour - we add it to give humans something better to look at - therefore it should be applied at the very last moment via the formatting xml. Nothing in this prevents that - I think it makes things easier. |
|
@felixfbecker the |
|
I second the comment about "Reset" being a better name than "AttributesOff". (I also prefer "VT sequences" to "ANSI escape sequences", but not quite as strongly.) Re: the Oh wait... did you say this class would be internal only? That seems like it would be a shame... the PowerShell engine is not going to be the only thing that wants to be able to deal with this sort of stuff. (Feel free to borrow any code from my related CaStringUtil class.) The "fg" and "bg" PSProviders in jaykul's Pansies has a really nice terse syntax-- Side note (only very tangentially related to this): another thing we need is editors that understand VT SGR sequences. Once or twice I've taken to embedding SGR sequences in my comment-based help comments, which looks great when displayed, but gets real ugly real quick when reading the comments in the source and trying to edit it. |
|
@jazzdelightsme updated to also include You should open that editor complaint in the VSCode repo! |
I think this is a very important point. Ideally the API would be designed in a way that doesn't make it appear that the best practice is to just output decorated strings. With the current design I think we'd see an uptick in this sort of pattern.
I can definitely think of scenarios, but none that wouldn't be solved easily enough with
Should PowerShell wait then? Once the dotnet version comes out we'll have one public API that doesn't work with PS and one internal type we still can't use right? Realistically PowerShell is probably going to need it's own version anyway since the dotnet one will likely be pretty byref-like heavy. So my opinion would be make the PowerShell-centric version public, and just switch the implementation to use the dotnet version once it becomes available. Maybe make them implicitly convertable to each other for extra points.
I mean we can use reflection if we don't care about using unsupported API's. I feel like pubternal API's don't really work in PowerShell, most users won't understand what that means. Especially with the already mixed messaging around them via |
|
Forgive my ignorance but why are not solving this in the formatting layer like #11890? PowerShell has kept data and formatting separate and it seems like this is trying to combine formatting (color) into the data (strings). |
|
To add to the comments by @felixfbecker, @SeeminglyScience, and @ThomasNieto re the problematic use of VT strings in the pipeline:
The problem is that, by virtue of using namespace notation via a provider-based implementation, @Jaykul himself has proposed a single, hierarchical
Re SGR (VT escape sequences) in comment-based help: Given that we want to move away from raw VT escape sequences, I think a more promising thing to consider is the use of (perhaps a limited subset of) Markdown in comment-based help. |
|
It's a mistake to talk about "moving away from raw VT escape sequences" -- that is impossible, this is just UX. Also, remember the color palette is always owned by the terminal. Practically all terminals support a base 16 color palette which can be configured. Most support changing the full 256 color xterm palette, nowadays. A growing number support using specific RGB values directly without a palette. But in no case should the shell try to redefine any of those palette colors. As far as the user experience goes, I think there are two separate concerns in the original pos that should perhaps be separated. The first part is the The second part is theming, and I think the design of a struct/class with specific named properties that are primarily for use in formatting files, and configured by the user according to their own preferences is a good one. There's no need for people to worry about when and where these things will be put into strings. We all know where things are converted into strings in PowerShell, and this does not change that. As a corollary, any discussion about alternate ways of applying formats is still tangential here! We've always had an enum for colors for the output streams, but it doesn't support xterm or full color, and I like the idea of not only extending the color to "any sequence supported by your terminal" but also extending the list of colors from Default, Warning, Error, Verbose, Debug, Progress to include a few extra colors like Success, and even some "accent" colors like PrimaryAccent, PrimaryMuted, SecondaryAccent, SecondaryMuted, or whatever. If we add these, then everything PowerShell outputs by default should only use the values defined here -- so that users can create specific combinations that they can read, and expect them to be respected throughout the shell. For the sake of usability and accessibility, I'm not sure it's a good idea to have the foreground and background defined separately for these -- there's very little chance that combining the foreground from one with a different background is a good idea, so why bother requiring people to write both out every time they want to use one of these styles? Wouldn't it be better to just have a "Warning" style which can include a foreground and background, depending on the preference of the user? I am not excited about the extra named formats (i.e. PSStyle) -- I think most of these are still too rarely functional to be of any use (for instance, bold is normally just "bright", which is actually rendered as a totally different color, and only supported on the base 8 colors -- see this comment). Frankly, we're probably better off using more generic names like "PrimaryAccent" and letting people define those to include blinking if they really want to ... In case it's not obvious from what I've already written: PowerShell should not define it's own base 16 color palette!That is, please do not implement anything like the proposed PSStyle+ForegroundColor. It would be a mistake. The color palette belongs to the terminal, not the shell. Implementing a palette where the names of the base 16 colors are defined as a different VT escape sequence than the default 16 color escape sequences would just confuse people and would wreck interaction with native apps. If you want to expose color names for simplicity, do what I did in Pansies and expose a shortcut way to use color names alongside rrggbb values to get the VT escape sequence for a color name, so that people don't need to learn the sequences -- but don't make the sequence customizable. |
|
Good points, @Jaykul, but just to be clear re:
What the quoted statement meant is (a) that we don't want to require that users deal directly with the underlying VT escape sequences (they should only have to deal with abstractions) and (b) for those advanced users that do, we should make it easy to generate such sequences with something like |
|
@SteveL-MSFT as long as those Foreground.Red colors are not settable, and always result in the simple 16 color codes I totally agree that "ANSI should only be used with formatting or text output and not as part of string members of objects" but given that statement, I don't understand what you mean about having "ANSI controlled at individual cmdlets"? If we "control" ANSI outside of the formatting or output commands, wouldn't that break the last point? I'm really not sure I'm totally on board with all this work going in to basically just create sets of colors. I can do that in a module and just tack it on to the Host.PrivateData for people to find.... Personally, I think it would be best to add ANSI parameters to the formatting commands and color properties to the formatting objects they output. Enhancing things with color there would be far less backward compatible (you couldn't just ship <ListItem>
<Label>green</Label>
<ScriptBlock>([PoshCode.Pansies.RgbColor]"Green").ToVtEscapeSequence() + "Green" + "$([char]27)[49;39m"</ScriptBlock>
</ListItem>I would much rather be able to write something like: <ListItem>
<Label>green</Label>
<PropertyName Foreground="Black" Background="Green">Green</PropertyName>
</ListItem>Or even <Label>green</Label>
<PropertyName Foreground="Black" Background="00EE00">Green</PropertyName>Actually, I really want to be able to get the colors with script: <TableColumnHeader>
<Alignment>Left</Alignment>
<Label>Green</Label>
<Width>5</Width>
<ForegroundColor>Black</ForegroundColor
<BackgroundColor>Green</BackgroundColor>
</TableColumnHeader>
...
<TableColumnItem>
<PropertyName>Green</PropertyName>
<BackgroundColorScript>$_.Green</BackgroundColorScript> <!-- changes the background of JUST THIS CELL to my custom green -->
</TableColumnItem>We want more than schemes. We want to color things simply, maybe even have a Of course, we also need the formatting engine to clear the colors after each cell (or at least, set them back to the colors for the column or row ... |
|
@Jaykul if the desire is to have color be first classes in formatting, then we could add With the current proposal, people could use any library they want for decoration as long as the end result is ANSI escape sequences. The main change in the engine is in formatting to use the new The point about "cmdlet controlled" is that ANSI is still the de facto language, but specific cmdlets may have options to strip ANSI from received strings or strip ANSI (or not include ANSI) on output. I personally prefer having stripping of ANSI just part of |
Generalization of formatting and rendering should be our priority. |
I think that sounds good on paper, but it seems like we've already tried that experiment, and it failed (never gained any traction): the current formatting cmdlets don't And after having implemented my own "Formatting+Output" engine, I can see why--somebody else's "generalized" format is usually not going to be quite what you want, anyway. People want the source object itself, or the final rendering; nobody actually wants a generalized, un-rendered representation of formatting. Or put another way, the source objects themselves are much better "intermediate forms" than any other intermediate form you could come up with. If you want to render them in a different way; that's great. But I don't think anybody actually wants or needs any generalized middle step to do that. |
|
I think ANSI escape sequences has already won the war. You already have xterm.js that renders ANSI to HTML. And Jupyter Notebooks with .NET Interactive also renders ANSI to HTML. CI/CD systems also render ANSI to HTML. |
|
Won the war? I would say the opposite. People prefer the Web that won. But tons of old applications that generate ANSI control codes are forced to work on the Web through extra adapters. Why should we go this crooked path? :-) Today PowerShell does not issue extra escapes and can output to (1) dumb terminals, (2) web engine like chromium. |
|
@iSazonov The current design would allow for translation of ANSI to HTML if that is required, but for console, ANSI is the de facto standard for authors of tooling |
|
By console, you mean a thin dumb teletype (tty) terminal. But at the same time we want coloring, multiple windows, background tasks, progress bar, power command line editor and so on - we actually want a thick powerful terminal. I am at a loss that we are spending so much effort to overcome the limitations of tty (here and in PSReadline). (Note that these escape sequences are contrary to the object-based nature of PowerShell.) This approach is counterproductive - we are forced to perform complex string conversions at the application level, then at the terminal level, then at the OS level. My strong belief is that PowerShell should only offer a generic interface and delegate the rest to the formatting system and host. The formatting system must understand the target is either a console or a file to apply appropriate formatting. |
|
I would be inclined to agree that expending effort to letting users handle this in code itself manually would be massively wasted if it were not easily and readily accessible to formatters (if not outright designed for that first tbh). |
|
🎉This issue was addressed in #13758, which has now been successfully released as Handy links: |
|
Might want to add some kind of validation when setting certain sub-properties. For example, see how I was able to set the value of
|
|
Using With an array containing two string elements, I would expect there to be only two lines in the output.
|
|
Thanks for filing the new feedback. Upvoted. Unfortunately this new feature is exposing a bug in the core tool. Honestly I never had a need to use |
|
@pcgeek86 You can use the @(
"$($PSStyle.Foreground.Blue)Hello$($PSStyle.Reset)",
'string2'
) `
| Out-String `
| Set-Content -Path test2.txt -NoNewline |
User scenario
Since we've added more ANSI escape sequences for coloring in PS7, if you pipe or redirect that text output, the embedded ANSI escape sequences are sent down the pipeline. This can make log files harder to read. Also, for those wanting to leverage ANSI escape sequences, they are difficult to create/read for scripters and also harder to use formatting presets rather than hardcoding specific colors.
Fundamental philosophy
On Linux, if you use
ls --colorthen different file types use ANSI escape sequences as color indicators. If you pipe this output toless, then you get paging while retaining the color information. If you redirect this output to a file, that file contains the ANSI escape sequences. If you then use thecatcommand on the file, you see the coloring as the ANSI escape sequences are rendered by the terminal.On macOS, the similar command is
ls -G, however, piping this output tolessor redirecting to a file you lose the ANSI escape sequences so it is just plain text. I believelson macOS is detecting if output is redirected and turning off coloring automatically.So for PowerShell, we should have a consistent experience regardless if Windows, Linux, or macOS and interop with native commands that may output ANSI escape sequences in addition to cmdlets/formatting.
Since ANSI escape sequences are de facto standardized, we should not introduce a new intermediate format and should just embed ANSI escape sequences within strings. This will ensure compatibility with other tools that emit or handle ANSI escape sequences.
Proposed technical implementation details
A new automatic variable
$PSStylewill be added:$PSStyle
The base members return ANSI escape sequences mapped to their names. These are also settable so the user can change bold to underlined, for example. This makes it easier for scripters to author decorated strings with tab completion:
"$($PSStyle.Background.LightCyan)Power$($PSStyle.Underlined)$($PSStyle.Bold)Shell$($PSStyle.Reset)"$PSStyle.OutputRendering
This is of type
System.Management.Automation.OutputRenderingwhich is an enum with the values:$PSStyle.Formatting
This effectively replaces
$Host.PrivateDataas the way to read or configure colors for formatting rendering.$Host.PrivateDatawill continue to exist for backwards compatibility, but is not connected to$PSStyle.Formatting.One difference here is that instead of being of type
ConsoleColor, these are all strings and doesn't separate foreground and background colors. This means that a single member can have foreground and background colors defined as well as other attributes like bold, underlined, etc...Scripters can easily leverage this:
"$($PSStyle.Formatting.ErrorAccent)Power$($PSStyle.Formatting.Verbose)Shell$($PSStyle.AttributesOff)"$PSStyle.Foreground and $PSStyle.Background
These members contain the standard 16 console colors as well as a RGB() methods to specify 24-bit color. For the colors, the values are settable and because they are strings can be any string content and any number of ANSI escape sequences.
PowerShell engine changes
The formatting system, pipelining, and redirection will be updated to respect the value of
$PSStyle.OutputRendering.StringDecorated type
This is an internal type to handle ANSI escaped strings.
Constructor
Single constructor taking a string as input.
bool IsDecoratedpropertyReturns if the string contains ANSI escape sequences based on if the string contains ESC or C1 CSI.
int LengthpropertyReturns the length of just the text content of the string sans ANSI escape sequences.
StringDecorated Substring(int contentLength)methodReturns a substring starting at index 0 up to the contentLength for content that is not part of an ANSI escape sequence. This is needed for table formatting to truncate strings preserving ANSI escape sequences that don't take up printable character space.
string ToString()methodReturns the plaintext version of the string.
string ToString(bool Ansi)methodReturns the raw ANSI embedded string if
Ansiparameter is true, otherwise returns plain text with ANSI escape sequences removed.Out-StringupdateTo enable scripts to easily allow plain text redirection,
Out-Stringwill have a-RemoveAnsiswitch.Select-StringupdateBy default, it would make sense for
Select-Stringto ignore ANSI escape sequences, perhaps a-IncludeAnsiswitch should be added.$Host.PrivateData
This will be available for legacy scripts/modules that read from it. However, the engine changes (and console host changes) would mean these settings will no longer be observed and instead use the settings from
$PSStyle.Alternate considerations
Original prototype was built on top of
System.CommandLine.Rendering, however, that is still a work in progress and is not targeted to be 1.0 in time for 7.1. It is also geared towards C# developers and the user experience is not great for scripts.Related Issues
#10811
#7744
#3611
The text was updated successfully, but these errors were encountered: