Option for LF vs. CR LF Piping To Match Line Endings Of Running Script

[AE1020]

Summary of the new feature / enhancement

I work with source code and files for cross-platform projects that are standardized to all UNIX line endings (LF) instead of "historical Windows" (CR LF) _{(Notepad.Exe supports LF only since 2018)}

My files are kept on shared filesystems that are accessed simultaneously by Windows, Linux, and MacOS systems.

Where possible I use bash for Windows when scripting, falling back on CMD.EXE occasionally. Yet increasingly PowerShell is needed, e.g. deprecations of things like ODBCCONF.EXE means I have to access "cmdlets" like Add-OdbcDsn.

But there appears to be no way to ask Powershell not to use CR LF line endings on Windows when doing redirects. In piping it will translate incoming LF to CR LF, which also leads to binary file corruption (see #1908)

Personally I never want this on any platform. I consider LF endings canon, and certainly prefer no-op pipes produce the same bytes they get in.

However, I'd assume that a global setting asking for LF on Windows always would break expectations of some packaged functionality. That functionality is why I'm using PowerShell in the first place, so no good.

Proposed technical implementation details

I propose an OPTIONAL mode of operation in which the LF / CR LF usage of the source code of the running powershell script itself dictate its behavior. If the script source is LF-only, then it thinks it lives in an LF-only world and likely wants to live by LF-only expectations for piping. If the script is CR LF, then its redirection and piping favor that.

This would allow PowerShell's behavior to piggy-back on the existing git setting for autocrlf translation that cross-platform developers already use. Windows users who turn it on probably want CR LF in redirects when they run a script. Windows users who turn it off probably want just LF on their filesystem always.

I don't know how something like Add-OdbcDsn works, but if it did piping on some .INI file that is CR LF based on Windows, my hope would be it would still work under this strategy...as a library script presumably is a CR LF file itself. So I could use a packaged piece of functionality like that in a script which itself was written to use LF only.

Maybe if it this option worked out well it could someday become a default. It would mean less conditional code between the Unix and Windows versions. As a first step I'd like it fine as something I could turn on globally and leave on.

[jhoneill]

Basing it on what the current script users isn't the greatest of ideas. What happens when one script calls another and one is unix-style LF and the other windows style CR-LF .
Where are you going to set the option? Presumably in the script anyway, so an option which says "look at my code and see if there are CR characters in it" might as well just do the thing.
A preference variable would make sense, similar to $OutputEncoding

• I think a lot of stuff looks at [system.enviroment]::newline which is read only. The problem as described would equally well apply to someone running a script on Linux and wanting Windows-style output.
• The > operator is really a wrapper for out-file (try 1,2,3 | out-file O:\foo.txt and the error comes from out-file) and a preference which sets it for out-file should apply to the > operator (and probably more). .
• Out-File and Out-String both support a -NoNewLineParameter but not a "Use the new line of the other OS" parameter.
• The workround of using Out-String and then doing a regex replace of "\r\n" with "\n" is ugly (Out-string also put extra blank lines around the text)

[btjwork]

I would like a global "don't corrupt my text data without my express permission" preference for PowerShell. It still confuses me that the current behavior isn't considered a bug.

All output should be treated as binary unless I explicitly pipe it into a PowerShell command that expects its input to be objects rather than data. I.e., the behavior should be based on what is consuming the data, not based on having been invoked by PowerShell.

If the data is being consumed by the filesystem (i.e., mycommand.exe >foo.txt), then at an OS level the stdout of mycommand.exe should be directly connected to a filesystem file descriptor. It should be solely up to mycommand.exe to control encoding and line endings.

[rkeithhill]

There has been some discussion on this issue particularly when piping from one native app to another - PS should step out of the way. However, the scenario that @btjwork brings up is a bit different. The > foo.txt is just a syntax sugar for ... | Out-File foo.txt. So that isn't a native to native scenario. For Out-File you have to either pick an encoding or Out-File will use its default (utf8NoBom) on PS >= 6.

[btjwork]

Yes, I was aware. I was hinting that I wish I could override > being a synonym for piping to Out-File. But, I realize that's complicated because of PS's historical choices. But, that is more of a discussion for the original issue:

#1908

(Note that the title of the original issue includes "or file".)

[AE1020]

Basing it on what the current script users isn't the greatest of ideas. What happens when one script calls another and one is unix-style LF and the other windows style CR-LF .

But this is exactly my point.

I only use PowerShell because I have to in order to use prepackaged functionality that requires it.

If library scripts I am calling expect the piping behavior to be generating CR LF (let's say because an internal tool or function would croak if passed data with only LF) then I don't want those to break.

But I want my scripts to not be generating CR LF files, because I don't want them anywhere on the shared drives.

The proposal has the possibility of converging to a place where if there are no CR LF scripts on a system, there will be no CR LF piping.

I'm not saying it's a strategy that would please everyone. But in my world it allows for a way that people collaborating a project can make the decisions for themselves based on how a file is represented on their drive while using the same script code. If you choose to use this option then you can collaborate with people who insist on CR LF while being able to maintain LF-only in your own world.

[AE1020]

I would like a global "don't corrupt my text data without my express permission" preference for PowerShell. It still confuses me that the current behavior isn't considered a bug.

Yes, though as mentioned some prepackaged functionality may depend on the corruption to operate. :-/

It's in some sense a separate point from whether CR LF are generated from thin air. but PowerShell itself isn't a neutral party if it generates any text with newlines at all which can be redirected. (e.g. some kind of multi-line echo of material from within a script). If it injects CR LF in that without any way to stop it...even if the source script was LF only...then that is undesirable.

So I'm proposing "Letting People Vote With Their Feet" ...e.g. how they represent line endings in their script on disk. This even permits individuals working with the same code to make different choices, and I think the Git autocrlf setting dovetails nicely with this.

(My true wish is that along with standardizing on UTF-8 with no byte order mark (as it should be!), the cross-platform focus had taken that extra step to standardizing on LF only so the files could be exchanged between platforms without transformation. That would have been ideal, and solving the binary corruption would be a nice bonus of that. But I would guess that is a bridge too far, so this is a proposal that could let people inch stepwise toward that world.)

[rkeithhill]

If Out-File had a -LineTerminator parameter on it, then you could configure your scripts to use this:

$PSDefaultParameterValues['Out-File:LineTerminator'] = 'Lf' # THIS PARAMETER DOES NOT EXIST ATM

This would work with both Out-File and >.

Currently, Out-File is calling through to .NET's TextWriter.WriteLine() which uses .NET's value for Environment.NewLine for the platform. With this new parameter, Out-File could be updated to use the TextWriter.Write() method and tack on the user-supplied line terminator sequence.

Caveat - any other cmdlets that write to files e.g. New-ModuleManifest, etc are likely to use Environment.NewLine so ymmv.

[btjwork]

I like the direction @rkeithhill is going for this particular scenario - I think this should be an explicit setting that the user and/or script writer can control. And @AE1020, this would be consistent with the git behavior you use as an example: it doesn't "guess" what line ending to use based on other files on the same filesystem, it instead makes an explicit consistent decision based on user preference settings.

This would also allow the top-level script to include the logic to "look around" and automatically initialize the setting based on whatever rules the developer chooses. For example, could literally look at the git settings, rather than trying to infer them.

[btjwork]

BTW, sorry to all if my first responses were confusing - I got here from @AE1020's comment on #1908, where it seemed that this was being suggested as an alternative solution to that scenario. I personally think this is an unrelated (though valid) scenario. I've been waiting years for #1908 to be resolved and didn't want to see it get side-tracked by this proposal.

[jhoneill]

Basing it on what the current script users isn't the greatest of ideas. What happens when one script calls another and one is unix-style LF and the other windows style CR-LF .

But this is exactly my point.

If you have a preference variable which is $LineTerminatorMatchScript = True and script A has cr/lf and script B has lf termination if A calls B which script sets the preference ?

And since you will need to set that preference in a script why ask PowerShell to examine the code and try to work out if what it is and why not just have $LineTerminatorPreference = Unix / $LineTerminatorPreference = Windows.

I only use PowerShell because I have to in order to use prepackaged functionality that requires it.

If library scripts I am calling expect the piping behavior to be generating CR LF (let's say because an internal tool or function would croak if passed data with only LF) then I don't want those to break.

That isn't the behaviour. Windows files (like DOS files before them and CP/M files before that) use CR LF as a line separator. PowerShell does have < to lash a file to std-in and ">" is (as already said) syntactic sugar for | out-File so the pipeline isn't merging < and > to make writing (txt) to stdout become (pseudo keyboard) input, it's designed for passing objects, and we get some behaviour differences. One is is for input, it's not a long stream of text with LF or CR LF pairs in it. PowerShell treats it as multiple text strings. Try IPConfig | measure (or IFConfig | measure on linux) and you'll see it is multiple items. The same when you read a file.

When you have a bunch of text strings and you want to send them to a non-Powershell program, or to a file, PowerShell inserts the local OS's new line (LF or CR LF).

But I want my scripts to not be generating CR LF files, because I don't want them anywhere on the shared drives.

So you have a quest to prevent Windows programs writing output in the way Windows programs naturally write. If you work with a sufficiently small set you might be able to achieve that. PowerShell on Linux (or in WSL) would do what you want, but PowerShell on Windows thinks it is being helpful, and isn't doing what you want.

The proposal has the possibility of converging to a place where if there are no CR LF scripts on a system, there will be no CR LF piping.
Yeah. Most modules you install will use CRLF so good luck with that.

I'm not saying it's a strategy that would please everyone. But in my world it allows for a way that people collaborating a project can make the decisions for themselves based on how a file is represented on their drive while using the same script code. If you choose to use this option then you can collaborate with people who insist on CR LF while being able to maintain LF-only in your own world.

Like I said it works both ways. If you come against someone who demands all LF and you're running on a Windows machine you should be able to please them, and if you're on a linux machine and the demand is for all CRLF that should be possible as well. Right now if the default for the OS you're on isn't the desired one you're stuck. There are plenty of harder things to change.

[AE1020]

$LineTerminatorMatchScript = True

I'm proposing a global setting for powershell. (It doesn't make sense to put this in the scripts themselves.)

If you come against someone who demands all LF and you're running on a Windows machine you should be able to please them, and if you're on a linux machine and the demand is for all CRLF that should be possible as well.

As the problem statement says that's what I'd like to see.

And I proposed an implementation path for this that doesn't require putting anything in the scripts themselves. (Based on git crlf settings, the crlf setting isn't something considered "in the script" but an out-of-band characteristic that version control can manage invisibly.)

If there's another way to do it, great. But I think my suggestion would work.