Unify cmdlets with parameter 'Encoding' to be of type System.Text.Encoding #5080

JamesWTruher · 2017-10-10T21:53:02Z

This unifies file encoding across the inbox cmdlets to be UTF-8 without a BOM for all platforms. This is a breaking change as cmdlets on windows have a number of different encodings. This supports better interoperability with tradition Linux shells as we are using the same encoding.

Use ArgumentCompletionsAttribute for the parameter to accept the currently used strings.
Register encoding providers on all platforms. This seems safe enough, and there are defintely more encodings available on CoreFX after this is done, which will be useful for some of the web cmdlet scenarios

Validate that files are created with UTF-8 encoding without BOM
Update tests to validate Encoding parameter to new type and create new tests for
parameter type validation.

Breaking Change The -Encoding Byte has been removed from the filesystem provider cmdlets. A new parameter -Byte is now the signal that a byte stream is required as input or output will be a stream of bytes.

adityapatwardhan · 2017-10-10T22:14:39Z

src/System.Management.Automation/engine/Utils.cs


+        // BigEndianUTF32 encoding is possible, but requires creation
+        internal static Encoding BigEndianUTF32Encoding = new UTF32Encoding(true,true);


Use named parameters instead of true, true

adityapatwardhan · 2017-10-10T22:17:58Z

src/System.Management.Automation/namespaces/FileSystemProvider.cs

-            {
-                return Encoding == FileSystemCmdletProviderEncoding.Byte;
-            } // get
+            get { return _usingByteEncoding; }


Please use automatic property and remove _usingByteEncoding.

adityapatwardhan · 2017-10-10T22:18:13Z

src/System.Management.Automation/namespaces/FileSystemProvider.cs

@@ -7688,9 +7631,14 @@ public bool WasStreamTypeSpecified
        {
            get
            {
-                return (Encoding != FileSystemCmdletProviderEncoding.String);
-            } // get
+                return _wasStreamTypeSpecified;


Same as above.

adityapatwardhan · 2017-10-10T22:19:32Z

src/System.Management.Automation/utils/ClrFacade.cs

@@ -109,16 +104,17 @@ internal static Encoding GetDefaultEncoding()

        /// <summary>
        /// Facade for getting OEM encoding
+        /// OEM encodings work on all platforms, or rather codepage 437 is available on both Windows 


Probably the comment intended to be codepage 437 is available on both Windows and Non-Windows

indeed so, fixed

adityapatwardhan · 2017-10-10T22:25:57Z

src/System.Management.Automation/utils/PathUtils.cs

+        internal class ByteEncoding : System.Text.Encoding
+        {
+            // the encoding for this is not bigendian and not BOM
+            public Encoding ActualEncoding = new UnicodeEncoding(false, false);


Use named parameters

adityapatwardhan · 2017-10-10T22:41:04Z

test/powershell/engine/Basic/Encoding.Tests.ps1

+            [pscustomobject]@{ Key = $testStr } | export-csv $outputFile
+            $bytes = Get-FileBytes $outputFile
+            $bytes[0,1,2] -join "-" | should not be ($utf8Preamble -join "-")
+            $bytes -join "-" | should match ($utf8bytes -join "-")


Should this be MatchExactly

not needed (as they're numbers) - MatchExactly isn't the same as Be, it's just a case-sensitive match -cmatch I just need to be sure that my string is in the output somewhere

adityapatwardhan · 2017-10-10T22:41:22Z

test/powershell/engine/Basic/Encoding.Tests.ps1

+            [pscustomobject]@{ Key = $testStr } | export-clixml $outputFile
+            $bytes = Get-FileBytes $outputFile
+            $bytes[0,1,2] -join "-" | should not be ($utf8Preamble -join "-")
+            $bytes -join "-" | should match ($utf8bytes -join "-")


Should this be MatchExactly

not needed - for the same reason as above, i'm just making sure that the magic bytes are in the file somewhere

adityapatwardhan · 2017-10-10T22:41:40Z

test/powershell/engine/Basic/Encoding.Tests.ps1

+            ${testStr} >> $outputFile
+            $bytes = Get-FileBytes $outputFile
+            $Expected = $( $ExpectedWithNewline; $ExpectedWithNewline )
+            $bytes -join "-" | should be ($Expected -join "-")


Should this be BeExactly

nope - they're numbers so case doesn't provide value

adityapatwardhan · 2017-10-10T22:42:41Z

test/powershell/engine/Basic/Encoding.Tests.ps1

+        }
+        It "Encoding parameter of command '<Command>' is type 'Encoding'" -testcase $testCases {
+            param ( $command )
+            $command.Parameters['Encoding'].ParameterType.FullName | Should Be "System.Text.Encoding"


Should be BeExactly?

it could be - BeExactly is a case-sensitive string compare, but the case of the response is not the point, only that it is the proper fullname.
I'm in the file so I'll update it, but we should be cautious about being overly pedantic.

adityapatwardhan · 2017-10-10T22:45:46Z

src/Microsoft.PowerShell.Commands.Utility/commands/utility/Send-MailMessage.cs

@@ -523,4 +523,4 @@ public override object Transform(EngineIntrinsics engineIntrinsics, object input
    }

    #endregion
-}


Missed adding the change for the Encoding parameter?

dantraMSFT · 2017-10-10T22:46:44Z

internal static class EncodingConversion

I think this class should be in its own CS file; it's not a path utility and discoverability is problematic.

In fact, I think this and the associated ArgumentToEncodingTransformationAttribute should be moved into an encoding file.

dantraMSFT · 2017-10-10T22:44:17Z

src/System.Management.Automation/utils/PathUtils.cs

        internal const string Utf7 = "utf7";
        internal const string Utf32 = "utf32";
        internal const string Default = "default";
        internal const string OEM = "oem";
+        internal const string Byte = "byte"; // This will return unicode, but 'byte' is special as it uses unicode encoding without a BOM
+        internal static ByteEncoding byteEncoding = new ByteEncoding();


This field should be static readonly and use the same casing as the constants defined above it (internal static readonly ByteEncoding ByteEncoding).

There are numerous references to this field in the various cmdlet's Encoding parameters but the purpose of it is opaque. I suggest a comment here or, perhaps, an accessor method that does the pattern instead of placing it in the various cmdlets or even initialize the field to new ByteEncoding().ActualEncoding.

dantraMSFT · 2017-10-11T17:02:00Z

src/Microsoft.PowerShell.Commands.Utility/commands/utility/CSVCommands.cs

+            }
+            set
+            {
+                if ( value == EncodingConversion.byteEncoding )


I believe you want to use Object.ReferenceEquals or override Equals in your ByteEncoding class and handle equality there; otherwise, you're relying on Encoding.Equals which may not do what you expect.

dantraMSFT · 2017-10-11T17:03:44Z

src/Microsoft.PowerShell.Commands.Utility/commands/utility/CSVCommands.cs

+            }
+            set
+            {
+                if ( value == EncodingConversion.byteEncoding )


These set methods need a comment as to why the comparison to EncodingConversion.byteEncoding is needed to provide a guide when creating new cmdlets that provide an Encoding parameter.

Maybe add a helper, something like:

EncodingConversion.SetEncodingParameterHelper(ref _encoding, value);

Or:

_encoding = EncodingConversion.EncodingParameterHelper(value);

In reply to: 144073559 [](ancestors = 144073559)

dantraMSFT · 2017-10-11T17:07:36Z

src/System.Management.Automation/utils/PathUtils.cs

+        // We need a way to provide an encoding and also
+        // a way to notify that the user wants "byte" encoding
+        // which is just a stream
+        internal class ByteEncoding : System.Text.Encoding


Should this class and the EncodingConversion.byteEncoding moved here as a public field?

In other words, is the problem you're solving with this custom instance be something third-party cmdlets will need to solve as well?

this is really a dodge for backward compatibility. -Encoding Byte should have never existed, but it does and I don't want to have this be a difference between 6 and 5.x. This byte encoding is really needed for the *-content cmdlets where the parameter is used as a flag indicating that the user is looking for a stream of bytes, I would rather not make this public

Since we're already making this big breaking change, I think we should fix this by not having ByteEncoding as Byte is an artificial "encoding". The *-Content cmdlets should have a different parameter to output or receive a byte stream.

It would more clear to users to have -Byte or -Binary parameter.
@mklement0 @rkeithhill What do you think?

I kind of prefer -AsByteArray myself.

cc @HemantMahawar @joeyaiello

Purely from a usability perspective, I don't have a problem with -Encoding Byte, because even though it is technically not a text encoding, the -Encoding parameter can be conceptualized as being about how to interpret the data on reading and writing, and -Encoding Byte (a byte stream) is one way of (non-)interpreting .

This keeps all considerations regarding encoding/non-encoding focused on a single parameter, which simplifies things (it doesn't introduce another parameter that you have to learn about and remember), both in terms of use and documentation.

How can we interpret Get-Content -Encoding Utf8 | Set-Content -Encoding 'CP1251' -Byte?

dantraMSFT · 2017-10-11T17:18:33Z

test/powershell/Language/Parser/RedirectionOperator.Tests.ps1

-        $CR  = [text.encoding]::unicode.GetBytes($asciiCR)
+        # create the expected - utf8 encoding without a bom
+        $encoding = [text.utf8encoding]::new($false)
+        $BOM = $encoding.GetPreamble()


This is very misleading; the test is verifying with a BOM yet the test is requesting the BOM. If the Encoding class has a bug or was created incorrectly, the test will still pass. In other words, if the array is empty, as expected, the test passes. If the array is non-empty, the test will still pass.

I suggest removing $BOM completely and not relying on the implicit empty array.

dantraMSFT · 2017-10-11T17:20:07Z

test/powershell/Language/Parser/RedirectionOperator.Tests.ps1

-        $TXT = [text.encoding]::unicode.GetBytes($asciiString)
-        $CR  = [text.encoding]::unicode.GetBytes($asciiCR)
+        # create the expected - utf8 encoding without a bom
+        $encoding = [text.utf8encoding]::new($false)


Suggest matching the casing of the type [Text.UTF8Encoding].

I fixed up the cases of everything while I was there

dantraMSFT · 2017-10-11T17:32:18Z

test/powershell/Modules/Microsoft.PowerShell.Management/Get-Content.Tests.ps1


        $testPath = Join-Path -Path $TestDrive -ChildPath 'TailWithEncoding.txt'
-        $content | Set-Content -Path $testPath -Encoding $encoding
+        $content | Set-Content -Path $testPath -Encoding $encodingName


Should this test cover the ByteEncoding path? It doesn't appear to.

covered by the tests in engine/Basic

iSazonov · 2017-10-11T15:48:07Z

src/System.Management.Automation/utils/PathUtils.cs

+                }
+                else if ( encoding.IndexOf(wordToComplete, StringComparison.InvariantCultureIgnoreCase) == 0 )
+                {
+                    encodings.Add(new CompletionResult(encoding, encoding, CompletionResultType.Text, encoding));


What is different from Line 635?

iSazonov · 2017-10-11T16:21:03Z

src/System.Management.Automation/utils/PathUtils.cs

+                {
+                    // the encodingName is guaranteed to be valid, so it is safe to pass null to method
+                    // Convert(Cmdlet cmdlet, string encoding) as the value of 'cmdlet'.
+                    return EncodingConversion.Convert(null, encodingName);


EncodingConversion.Convert() do this comparitions too - I believe we could directly call Convert() and catch an exception or change EncodingConversion.Convert to return null or create TryConvert().

iSazonov · 2017-10-11T16:32:02Z

src/System.Management.Automation/namespaces/FileSystemProvider.cs

-    /// Defines the values that can be supplied as the encoding parameter in the
-    /// FileSystemContentDynamicParametersBase class.
-    /// </summary>
-    public enum FileSystemCmdletProviderEncoding


It seems we shouldn't remove public API. Maybeuse [ObsoleteAttribute].

Jim, can you check if this API is in PowerShell Standard 3.0? This enum isn't used at all now, right? So leaving it in with Obsolete doesn't help.

iSazonov · 2017-10-11T17:15:34Z

src/System.Management.Automation/utils/ClrFacade.cs

-                uint currentAnsiCp = NativeMethods.GetACP();
-                s_defaultEncoding = Encoding.GetEncoding((int)currentAnsiCp);
-#endif
+                s_defaultEncoding = new UTF8Encoding(false);


I don't like the change. Previously (discussion #3248, PR #3467) I fixed this for Windows because we're still very sensitive to the OEM encoding. We have tons applications in enterprises that doesn't work if we don't set the correct OEM encoding on the system. If MSFT wants to cut it off, do it first in Windows 10 kernel and get feedback.

@iSazonov where are tests which provide validation for the changes you made in PR3467? The discussion in #3248 doesn't seem to have any concrete examples of misbehavior, which I would really like to understand. As for your last, I would rather get feedback on breaking behavior outside of the Windows ship cycles.

@iSazonov, @PowerShell/powershell-committee has already decided to go with UTF8 (No BOM) as default. Can you give me a specific example where this is known to break? If OEM is needed, perhaps we could add a setting in the powershell json file but still default to utf8

PR #3467#3467 haven't tests because (1) we did not have a full set of encoding tests in the time (2) we waited the encoding RFC (3) we only restore Windows PowerShell behavior by removing a breaking change.
About concrete examples. I speak about business applications. I have no idea how I can demonstrate it concrete. I understand that you hardly ever changed the OEM encoding on your computers. But ask yourself why are customers forced to do so? If MSTF wants to see it and understand it, it needs TAP with non-english enterprises. As I said earlier, the simple answer is that we have many business applications that only work with a specific OEM encoding and read/write files in OEM. PowerShell also works in this environment! The #3467 was caused by the fact that the scripts were no longer working and we needed to re-encode the files. The value of PowerShell Core disappears completely.

We can look at it on the other side. A year ago there was a categorical position that we should be backward compatible and that we should avoid making breaking changes as much as possible. The position was later changed so we want to facilitate the adaptation of UNIX users. This led to the creation of many incompatible changes. The current change from this series. These two principles have proved to be contradictory. We need to decide which one's in charge.

If the first means that we want to allow multiple Windows users to use PowerShell on Unix without much effort, and to get feedback to start the process of evolutionary convergence of platforms.

The second actually means that we're creating a brand new product that cannot be version 6, but must be PowerShell Core 1.0. It's fair to say that Windows users will have to migrate to a new product instead of using it directly. In this case, I agree that we can be ahead of the Windows 10 system and even its Notepad and use Utf8 by default like all other changes. Only then will it not be expected that the community will begin to use PowerShell Core extensively before version 3.
The second way is very dangerous. Microsoft can break the Windows PowerShell community and not build a new one-there are already well-known and popular alternatives (PHP, JS, Perl) that have proven their reliability and performance and have already formed a significant community. I've seen some examples of Microsoft already quickly destroying communities. The recovery took several years. It's sad. If it were my business, I would prefer to keep today's real customers, not future abstracts, and make the first step similar WSL like Windows PowerShell for Unix to capture a reliable bridgehead. Only then the community would be prepared to accept any breaking changes as understandable and necessary. Sorry for many words :-)

It seems again that a viable option is to have a setting for powershell.runtimeconfig.json to have this be OEM for compat reasons.

I haven't followed this closely, but what happened to the $PSDefaultEncoding = 'WindowsLegacy' proposal from the RFC?

And, just to clarify: for backward compatibility, it's both OEM and "ANSI" code pages that must be considered.

@mklement0 the simplified version we scoped for 6.0.0 does not include WindowsLegacy

@JamesWTruher can you open an issue to cover @iSazonov 's concern?

@mklement0 WindowsLegacy was pulled waiting for customer need - opened #5204 to track.

iSazonov · 2017-10-11T17:34:46Z

src/System.Management.Automation/utils/PathUtils.cs

@@ -459,7 +478,7 @@ internal static Encoding Convert(Cmdlet cmdlet, string encoding)
        {
            if (string.IsNullOrEmpty(encoding))
            {
-                // no parameter passed, default to Unicode (OS preferred)
+                // no parameter passed, default to  (OS preferred)


indeed so, and incorrect (it should be UTF8) - fixed

iSazonov · 2017-10-11T17:44:57Z

src/System.Management.Automation/utils/PathUtils.cs

+        // We will provide a partial list for tab completion we will convert to an encoding.
+        // If the user has an encoding object, we will cheerfully use it, but we can't use
+        // it as a validation set, as a user may have an specialty encoding which we have
+        // no idea about.


I think we could use ArgumentCompletionsAttribute #4835

iSazonov · 2017-10-11T17:46:20Z

src/System.Management.Automation/utils/PathUtils.cs

-                new string[] { Unknown, String, Unicode, BigEndianUnicode, Ascii, Utf8, Utf7, Utf32, Default, OEM });
-            string msg = StringUtil.Format(PathUtilsStrings.OutFile_WriteToFileEncodingUnknown,
-                encoding, validEncodingValues);
+                new string[] { Unknown, String, Unicode, BigEndianUnicode, Byte, Ascii, Utf8, Utf8Bom, Utf8NoBom, Utf7, Utf32, Default, OEM });


Can we reuse TabCompletionResults ?

iSazonov · 2017-10-11T17:49:55Z

src/Microsoft.PowerShell.Commands.Utility/commands/utility/CSVCommands.cs

-        [ValidateSetAttribute(new string[] { "Unicode", "UTF7", "UTF8", "ASCII", "UTF32", "BigEndianUnicode", "Default", "OEM" })]
-        public string Encoding { get; set; }
+        [ArgumentToEncodingTransformationAttribute()]
+        [ArgumentCompleter(typeof(EncodingArgumentCompleter))]


I think we could use ArgumentCompletionsAttribute #4835

lzybkr · 2017-10-12T23:32:34Z

src/System.Management.Automation/utils/PathUtils.cs

@@ -470,6 +489,9 @@ internal static Encoding Convert(Cmdlet cmdlet, string encoding)
            if (string.Equals(encoding, String, StringComparison.OrdinalIgnoreCase))
                return System.Text.Encoding.Unicode;

+            if (string.Equals(encoding, Byte, StringComparison.OrdinalIgnoreCase))
+                return byteEncoding;


If we aren't using a dictionary for fast lookup, we can at least order the tests from most likely (guessing of course) to least likely.

e.g., is Unknown really the most likely case? I hope not.

lzybkr · 2017-10-12T23:35:24Z

src/System.Management.Automation/utils/PathUtils.cs

+            // the encoding for this is not bigendian and not BOM
+            public Encoding ActualEncoding = new UnicodeEncoding(false, false);
+            public override unsafe int GetByteCount(char* c, int i1) { throw new NotImplementedException(); }
+            public override int GetByteCount(char[] c, int i1, int i2) { throw new NotImplementedException(); }


I believe it's not hard for someone to get an instance of this encoding, so maybe we should be implementing all of these methods, assuming it's straightforward to do so (and mostly it seems like it is.)

SteveL-MSFT · 2017-10-13T23:00:50Z

src/System.Management.Automation/utils/PathUtils.cs

+        // We need a way to provide an encoding and also
+        // a way to notify that the user wants "byte" encoding
+        // which is just a stream
+        internal class ByteEncoding : System.Text.Encoding


Since we're already making this big breaking change, I think we should fix this by not having ByteEncoding as Byte is an artificial "encoding". The *-Content cmdlets should have a different parameter to output or receive a byte stream.

SteveL-MSFT · 2017-10-13T23:03:05Z

src/System.Management.Automation/utils/ClrFacade.cs

-                uint currentAnsiCp = NativeMethods.GetACP();
-                s_defaultEncoding = Encoding.GetEncoding((int)currentAnsiCp);
-#endif
+                s_defaultEncoding = new UTF8Encoding(false);


@iSazonov, @PowerShell/powershell-committee has already decided to go with UTF8 (No BOM) as default. Can you give me a specific example where this is known to break? If OEM is needed, perhaps we could add a setting in the powershell json file but still default to utf8

mklement0 · 2017-10-15T16:03:03Z

src/System.Management.Automation/utils/PathUtils.cs

        internal const string Utf7 = "utf7";
        internal const string Utf32 = "utf32";
        internal const string Default = "default";
        internal const string OEM = "oem";
+        internal const string Byte = "byte"; // This will return unicode, but 'byte' is special as it uses unicode encoding without a BOM


Not knowing the inner workings: How will this return Unicode? With Get-Content, you get a byte array, and on output (Set-Content / Add-Content) you're forced to provide a byte array as input too (however you constructed that).

JamesWTruher · 2017-10-17T18:43:48Z

@adityapatwardhan @dantraMSFT @SteveL-MSFT I've believe I've made updates that address your blocking issues

SteveL-MSFT · 2017-10-18T23:35:22Z

@PowerShell/powershell-committee reviewed this and has agreed on:

name the parameter -AsByteArray
output warning if -Encoding is used with -AsByteArray that Encoding is not being used

mklement0 · 2017-10-19T03:22:23Z

A message from the COSP (Crying Over Spilt Milk) department:

output warning if -Encoding is used with -AsByteArray that Encoding is not being used

This results in an awkward user experience, because users will expect incompatible parameters to be in different parameter sets, with attempts to combine them resulting in an error, not a warning.

A step backward in usability, and a breaking change to boot - and no gain that I can discern.

I also agree with @iSazonov that forgoing compatibility with the legacy encoding behavior on Windows altogether will dismay many Windows users, especially those whose "ANSI" / OEM code pages aren't based on the Latin alphabet.

If the idea is to focus on the Unix crowd first, I see trouble as well: broken quoting, broken CLI, broken globbing, lack of && and ||, ...

SteveL-MSFT · 2017-10-19T20:27:17Z

@mklement one of the considerations the committee acknowledged is the complexity of using multiple parametersets to make parameters mutually exclusive so we were ok with using a warning as a simplification until we can revisit the recalled RFC on making it easier to both define as well as understand mutually exclusive parameters

…oding Created support classes so tab completion would continue to wor Create an ArgumentTransformationAttribute for the parameter to accept the currently used strings Register encoding providers on all platforms. This seems safe enough, and there are defintely more encodings available on CoreFX after this is done, which will be useful for some of the web cmdlet scenarios Validate that files are created with UTF-8 encoding without BOM Update tests to validate Encoding parameter to new type and create new tests for parameter type validation.

refactor encoding class and other encoding utilities into new file update redirection operator tests to convert string matching for encoding into dictionary, which should make it more efficient

Add a new parameter to the filesystem provider cmdlets to indicate whether we want a byte stream Unify Send-MailMessage with other cmdlets

Created a warning when both -AsByteStream and -Encoding are used together updated tests to check that the warning is emitted

Also improve encoding completion to handle wildcards Also make sure that Encoding parameters are properties rather than fields. Remove unused OpenStreamReader which takes a string since it's no longer in use

…bute for encoding attribute. Simplify logic for converting string to encoding

daxian-dbw · 2017-10-23T20:04:58Z

src/Microsoft.PowerShell.Commands.Utility/commands/utility/CSVCommands.cs

+            EncodingConversion.Utf8Bom,
+            EncodingConversion.Utf8NoBom,
+            EncodingConversion.Utf32
+            )]


How about use ArgumentCompletions(EncodingConversion.TabCompletionResults) here? It would be easy for future updates too.

Same comments for other instances of ArgumentCompletions usage.

that doesn't work, because arrays aren't constant (I tried that first)

OK, then we are good here. #Closed

daxian-dbw · 2017-10-23T20:23:23Z

@dantraMSFT Can you please take another look and update your review?

Comments has been addressed by later commits.

daxian-dbw · 2017-10-24T02:42:11Z

I'm getting this PR merged so that it can be validated by today's daily builds, and hopefully, it can catch Beta.9 release.
Actually, this PR should have the last commit marked with '[Feature]' tag to trigger full build run.

JamesWTruher added the Breaking-Change breaking change that may affect users label Oct 10, 2017

JamesWTruher assigned daxian-dbw Oct 10, 2017

JamesWTruher requested review from daxian-dbw, SteveL-MSFT, dantraMSFT and adityapatwardhan October 10, 2017 21:53

JamesWTruher requested review from anmenaga, BrucePay, lzybkr, PaulHigin and vors as code owners October 10, 2017 21:53

adityapatwardhan requested changes Oct 10, 2017

View reviewed changes

dantraMSFT suggested changes Oct 11, 2017

View reviewed changes

iSazonov reviewed Oct 11, 2017

View reviewed changes

lzybkr reviewed Oct 12, 2017

View reviewed changes

SteveL-MSFT requested changes Oct 13, 2017

View reviewed changes

SteveL-MSFT added the Review - Committee The PR/Issue needs a review from the PowerShell Committee label Oct 13, 2017

mklement0 reviewed Oct 15, 2017

View reviewed changes

JamesWTruher force-pushed the jameswtruher/fileEncodingMin branch from 6ca1339 to b8322f4 Compare October 16, 2017 19:36

JamesWTruher force-pushed the jameswtruher/fileEncodingMin branch from d2f4c8d to a131575 Compare October 18, 2017 17:46

SteveL-MSFT added Committee-Reviewed PS-Committee has reviewed this and made a decision and removed Review - Committee The PR/Issue needs a review from the PowerShell Committee labels Oct 18, 2017

JamesWTruher mentioned this pull request Oct 19, 2017

a warning should be presented if -AsByteStream and -Encoding are used together #5170

Closed

JamesWTruher force-pushed the jameswtruher/fileEncodingMin branch from a131575 to 068ee02 Compare October 19, 2017 19:46

JamesWTruher added 8 commits October 23, 2017 11:49

Update to follow coding style guidelines

431370a

Fix cmdlet casing to follow guidelines

53b31e9

Change the encoding to utf8 when the converter gets an empty string

6ef8580

refactor encoding class and other encoding utilities into new file update redirection operator tests to convert string matching for encoding into dictionary, which should make it more efficient

Change Encoding to remove byte as a possible encoding value

e29baac

Add a new parameter to the filesystem provider cmdlets to indicate whether we want a byte stream Unify Send-MailMessage with other cmdlets

Changed -Byte parameter to -AsByteStream

b59b0ee

Created a warning when both -AsByteStream and -Encoding are used together updated tests to check that the warning is emitted

put ValidateNotNullOrEmpty attribute on Encoding parameters

840343d

Also improve encoding completion to handle wildcards Also make sure that Encoding parameters are properties rather than fields. Remove unused OpenStreamReader which takes a string since it's no longer in use

Change from IArgumentCompleter attribute to ArgumentCompletions attri…

007bae4

…bute for encoding attribute. Simplify logic for converting string to encoding

JamesWTruher force-pushed the jameswtruher/fileEncodingMin branch from 65d31df to 007bae4 Compare October 23, 2017 19:32

daxian-dbw requested changes Oct 23, 2017

View reviewed changes

daxian-dbw approved these changes Oct 23, 2017

View reviewed changes

daxian-dbw merged commit be70072 into PowerShell:master Oct 24, 2017

mklement0 mentioned this pull request Oct 31, 2017

Get-Content not support -Encoding Byte in beta 9 #5279

Closed

TylerLeonhardt mentioned this pull request Nov 3, 2017

Windows PowerShell Support and InvokeBuild PowerShell/Polaris#62

Merged

zjalexander added the Documentation Needed in this repo Documentation is needed in this repo label Nov 15, 2017

stevenzeck mentioned this pull request Nov 20, 2017

Latest PowerShell Core: Cannot bind parameter 'Encoding'. Cannot convert the "byte" value of type "System.String" to type "System.Text.Encoding" DTW-DanWard/PowerShell-Beautifier#33

Closed

matt9ucci mentioned this pull request Jan 3, 2018

Update the default/acceptable values for Encoding parameter in v6.0 MicrosoftDocs/PowerShell-Docs#2000

Closed

7 tasks

stknohg mentioned this pull request Apr 4, 2018

Converting from windows 1252 to UTF8 #6550

Closed

iSazonov mentioned this pull request Aug 16, 2018

Common encoding pattern for Utility Cmdlets #3628

Closed

iSazonov mentioned this pull request Oct 11, 2018

Removal of "Encoding Byte" is a breaking change #7986

Closed

joeyaiello mentioned this pull request Oct 15, 2018

Docs needed for 'Unify cmdlets with parameter 'Encoding' to be of type System.Text.Encoding' MicrosoftDocs/PowerShell-Docs#3076

Closed

joeyaiello removed Documentation Needed in this repo Documentation is needed in this repo labels Oct 15, 2018

KalleOlaviNiemitalo mentioned this pull request Jun 6, 2020

Make console windows fully UTF-8 by default on Windows, in line with the behavior on Unix-like platforms - character encoding, code page #7233

Open

JamesWTruher deleted the jameswtruher/fileEncodingMin branch May 11, 2022 18:27

xtqqczze mentioned this pull request Oct 23, 2022

Replace UTF8Encoding(false) with Encoding.Default part 2 #18356

Merged

unfurl-links bot mentioned this pull request Nov 11, 2022

"IBM437 is not supported encoding name" exception when saving Excel workbook on daily PS 7.3 build #18537

Closed

5 tasks


		// BigEndianUTF32 encoding is possible, but requires creation
		internal static Encoding BigEndianUTF32Encoding = new UTF32Encoding(true,true);

Unify cmdlets with parameter 'Encoding' to be of type System.Text.Encoding #5080

Unify cmdlets with parameter 'Encoding' to be of type System.Text.Encoding #5080

Conversation

JamesWTruher commented Oct 10, 2017 • edited by daxian-dbw

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dantraMSFT commented Oct 10, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

iSazonov Oct 16, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JamesWTruher Oct 12, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mklement0 Oct 15, 2017 • edited

Choose a reason for hiding this comment

JamesWTruher commented Oct 17, 2017

SteveL-MSFT commented Oct 18, 2017

mklement0 commented Oct 19, 2017 • edited

SteveL-MSFT commented Oct 19, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JamesWTruher Oct 23, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

daxian-dbw commented Oct 23, 2017

JamesWTruher commented Oct 10, 2017 •

edited by daxian-dbw

dantraMSFT commented Oct 10, 2017 •

edited

iSazonov Oct 16, 2017 •

edited

JamesWTruher Oct 12, 2017 •

edited

mklement0 Oct 15, 2017 •

edited

mklement0 commented Oct 19, 2017 •

edited

JamesWTruher Oct 23, 2017 •

edited

daxian-dbw commented Oct 24, 2017 •

edited