Make Out-String and Out-File keep string input unchanged

[daxian-dbw]

PR Summary

Fix #17452

The changes in this PR update the behavior of Out-File and Out-String regarding the RenderingOutput setting to be the following:

• when the input object is pure string, these 2 cmdlets keep the string input unchanged regardless of the RenderingOutput setting
• when the input object needs to have formatting views applied to it, these 2 cmdlets remove escape sequences from the formatting output strings based on the RenderingOutput setting.

This change is a breaking change to these 2 cmdlets' existing behavior regarding RenderingOutput.

PR Checklist

• PR has a meaningful title
  • Use the present tense and imperative mood when describing your changes
• Summarized changes
• Make sure all .h, .cpp, .cs, .ps1 and .psm1 files have the correct copyright header
• This PR is ready to merge and is not Work in Progress.
  • If the PR is work in progress, please add the prefix WIP: or [ WIP ] to the beginning of the title (the WIP bot will keep its status check at Pending while the prefix is present) and remove the prefix when the PR is ready.
• Breaking changes
  • None
  • OR
  • Experimental feature(s) needed
    • Experimental feature name(s):
• User-facing changes
  • Not Applicable
  • OR
  • Documentation needed
    • Issue filed: Change the behavior of Out-File and Out-String regarding $PSStyle.OutputRendering MicrosoftDocs/PowerShell-Docs#8892
• Testing - New and feature
  • N/A or can only be tested interactively
  • OR
  • Make sure you've added a new test if existing tests do not effectively test the code changed
• Tooling
  • I have considered the user experience from a tooling perspective and don't believe tooling will be impacted.

[iSazonov]

We need to update docs https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_ansi_terminals?view=powershell-7.2

about ANSI terminals - PowerShell

Describes the features of PowerShell that use ANSI escape sequences and the terminal hosts that support them.

[SeeminglyScience]

LGTM! Thank you for making it still possible to retain formatting when output rendering is Ansi ❤️

[daxian-dbw]

@PowerShell/powershell-committee Need the committee to review the breaking change comparing with 7.2.x.
Below is a summary of the breaking change. The corresponding issue is #17452.

In 7.2.x, Out-File and Out-String remove escape sequences from the passed-in strings when $PSStyle.OutputRendering is Host or PlainText.

When the passed-in object has formatting views applied to it, removing escape sequences from the resulted formatting text makes perfect sense when the output rendering option is Host and PlainText.

However, when the passed-in objects are pure strings, those 2 cmdlets shouldn't make any changes to the string objects, because

1. that will make the scenarios like (Get-Content file-a -Raw).Replace("`r`n", "`n") > file-b not working. Another example is that Set-Content and Out-File behave differently on string input under those output rendering options but they should really behave the same:

   ## Out-File changes the input strings when output rendering is `Host` or `PlainText`
   "`e[31mabc`e[0m" | Set-Content -Path .\abc.txt
   "`e[31mabc`e[0m" | Out-File -Path .\abc.txt
   

2. that also cause potential security problems as we are altering the string input when doing redirection. This was the reason why we changed the default OutputRendering option from Host to Ansi in v7.2.1

3. given that we have changed the default output rendering option back to Host in 7.3 preview, "`e[31mabc`e[0m" > .\abc.txt won't work the same as in 7.0.x and 7.2.x by default. That is a regression to 2 LTS releases.

I think the $PSStyle.OutputRendering option should only affect formatting output when it comes to Out-File and Out-String. When the input objects are pure strings, they should not be altered by Out-File or Out-String regardless of the output rendering option.

[ImportTaste]

Is this going to get rid of the extra linebreak that happens too?

[daxian-dbw]

@ImportTaste This PR only affects escape sequences.

[SteveL-MSFT]

Really great change!

[pull-request-quantifier-deprecated]

This PR has 66 quantified lines of changes. In general, a change size of upto 200 lines is ideal for the best PR experience!

---

Quantification details

Label      : Small
Size       : +60 -6
Percentile : 26.4%

Total files changed: 5

Change summary by file extension:
.cs : +6 -4
.ps1 : +54 -2

Change counts above are quantified counts, based on the PullRequestQuantifier customizations.

Why proper sizing of changes matters

Optimal pull request sizes drive a better predictable PR flow as they strike a
balance between between PR complexity and PR review overhead. PRs within the
optimal size (typical small, or medium sized PRs) mean:

• Fast and predictable releases to production:
  • Optimal size changes are more likely to be reviewed faster with fewer
    iterations.
  • Similarity in low PR complexity drives similar review times.
• Review quality is likely higher as complexity is lower:
  • Bugs are more likely to be detected.
  • Code inconsistencies are more likely to be detetcted.
• Knowledge sharing is improved within the participants:
  • Small portions can be assimilated better.
• Better engineering practices are exercised:
  • Solving big problems by dividing them in well contained, smaller problems.
  • Exercising separation of concerns within the code changes.

What can I do to optimize my changes

• Use the PullRequestQuantifier to quantify your PR accurately
  • Create a context profile for your repo using the context generator
  • Exclude files that are not necessary to be reviewed or do not increase the review complexity. Example: Autogenerated code, docs, project IDE setting files, binaries, etc. Check out the Excluded section from your prquantifier.yaml context profile.
  • Understand your typical change complexity, drive towards the desired complexity by adjusting the label mapping in your prquantifier.yaml context profile.
  • Only use the labels that matter to you, see context specification to customize your prquantifier.yaml context profile.
• Change your engineering behaviors
  • For PRs that fall outside of the desired spectrum, review the details and check if:
    • Your PR could be split in smaller, self-contained PRs instead
    • Your PR only solves one particular issue. (For example, don't refactor and code new features in the same PR).

How to interpret the change counts in git diff output

• One line was added: +1 -0
• One line was deleted: +0 -1
• One line was modified: +1 -1 (git diff doesn't know about modified, it will
  interpret that line like one addition plus one deletion)
• Change percentiles: Change characteristics (addition, deletion, modification)
  of this PR in relation to all other PRs within the repository.

---

Was this comment helpful? 👍  :ok_hand:  :thumbsdown: (Email)
Customize PullRequestQuantifier for this repository.

[JamesWTruher]

I think we should seriously consider adding a parameter to out-string and out-file which retains our current behavior. Perhaps a parameter -AsPlainText or some such will allow users to have the older behavior while we provide a safer default behavior.

[daxian-dbw]

I agree that we should consider so, but maybe wait until we get issue reports that someone really gets affected by this breaking change.
Also, -AsPlainText may be misleading because it means "leaves the string as is" in other places where this name is used, not "removing ANSI sequences", but we can discuss this more when we really get to consider adding this parameter.

[SteveL-MSFT]

@PowerShell/powershell-committee reviewed this, we would recommend adding a parameter to enable users to get back the previous behavior.

[daxian-dbw]

@SteveL-MSFT Is that a requirement for this PR or could do in separate PR? Also, is that supposed to be done now or should wait until there is a report about needing the previous behavior?

[daxian-dbw]

Given that the recommendation is a separate feature, I think it would be fine to use a separate PR if we really decided to add a parameter for the previous behavior. I will merge this PR and open an issue for that.

Opened the issue #17509. I also left my comments there.

[ghost]

🎉v7.3.0-preview.5 has been released which incorporates this pull request.:tada:

Handy links:

• Release Notes

[adityapatwardhan]

/backport to release/v7.2.6

[github-actions]

Started backporting to release/v7.2.6: https://github.com/PowerShell/PowerShell/actions/runs/2784883343

GitHub

Make `Out-String` and `Out-File` keep string input unchanged · PowerShell/PowerShell@0e4d2e4

PowerShell for every system! Contribute to PowerShell/PowerShell development by creating an account on GitHub.

[github-actions]

@adityapatwardhan backporting to release/v7.2.6 failed, the patch most likely resulted in conflicts:

$ git am --3way --ignore-whitespace --keep-non-patch changes.patch

Applying: Make Out-String and Out-File keep string input unchanged
.git/rebase-apply/patch:91: trailing whitespace.
            (Get-Verb -Verb Get | Out-String).Contains("`e[") | Should -BeTrue 
warning: 1 line adds whitespace errors.
Using index info to reconstruct a base tree...
M	src/System.Management.Automation/FormatAndOutput/common/ILineOutput.cs
M	test/powershell/engine/Formatting/PSStyle.Tests.ps1
Falling back to patching base and 3-way merge...
Auto-merging test/powershell/engine/Formatting/PSStyle.Tests.ps1
CONFLICT (content): Merge conflict in test/powershell/engine/Formatting/PSStyle.Tests.ps1
Auto-merging src/System.Management.Automation/FormatAndOutput/common/ILineOutput.cs
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 Make Out-String and Out-File keep string input unchanged
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".
Error: The process '/usr/bin/git' failed with exit code 128

Please backport manually!

[adityapatwardhan]

Backport done via #17859

[ghost]

🎉v7.2.6 has been released which incorporates this pull request.:tada:

Handy links:

• Release Notes