[release/3.0] Avoid MemoryMarshal.Cast when transcoding from UTF-16 to UTF-8 while escaping in Utf8JsonWriter.

[ahsonkhan]

Port of #40996 to fix https://github.com/dotnet/corefx/issues/40979

Description

Instead of using MemoryMarshal to re-interpret cast a span of UTF-16 chars to bytes (to pass them to APIs expecting UTF-8 data), call the JavascriptEncoder API that expects UTF-16 chars instead. Casting char to byte doesn't transode it from UTF-16 to UTF-8 which was the previous intention. Doing so results in certain invariants in the code to break since the resulting index that points to the first character to escape would be incorrect (or even out of the bounds of the original span), which results in a negative value being passed in to stackalloc (and hence a stackoverflow). The issue is in the code-path where a custom encoder is passed in (and wouldn't happen by default).

Customer Impact

The bug was customer-reported where the user observed a stackoverflow in an ASP.NET WebAPI (which uses a custom encoder) when trying to serialize a string that contained non-ascii characters (for example chinese caracters). Generally, any use of the JsonSerializer or Utf8JsonWriter where a custom encoder is involved for writing .NET strings as JSON is affected. It is imperative that the escaping behavior of the serializer is functionally correct.

Regression?

Introduced in .NET Core 3.0 - preview 8

Risk

The risk of this change is around the escaping behavior changing when the user passes in a custom escaper. Be default, the JSON stack uses the default escaper which isn't affected by this change. However, the ASP.NET defaults to a custom escaper so end-users are more likely to be affected by this escaping behavior and fix. There is no easy workaround for the user since passing in a custom/default encoder wouldn't always work either.

Tests run / added

• Regression test for the serializer was added along with targeted tests of the writer itself (which the serializer uses under the covers).
• Verified end-to-end in an ASP.NET Web API app that writing non-ascii strings with the custom escaper works as expected.

cc @steveharter, @GrabYourPitchforks, @pranavkm, @ericstj

[ahsonkhan]

MacOS Build x64_Debug test failures are unrelated:
https://dev.azure.com/dnceng/public/_build/results?buildId=348664&view=ms.vss-test-web.build-test-results-tab&runId=10473744&resultId=168822&paneView=debug

System.Security.Cryptography.OpenSsl.Tests on netcoreapp-OSX-Debug-x64-OSX.1014.Amd64.Open

System.Security.Cryptography.OpenSsl.Tests Total: 649, Errors: 0, Failed: 565, Skipped: 14, Time: 1.083s

https://github.com/dotnet/corefx/issues/40262
https://github.com/dotnet/corefx/issues/38498

Example test failure:

System.Security.Cryptography.Dsa.Tests.DSAKeyFileTests.ReadWriteDsa1024EncryptedPkcs8_PasswordBytes [FAIL]
      System.TypeInitializationException : The type initializer for 'Crypto' threw an exception.
      ---- System.TypeInitializationException : The type initializer for 'CryptoInitializer' threw an exception.
      -------- System.DllNotFoundException : Unable to load shared library 'System.Security.Cryptography.Native.OpenSsl' or one of its dependencies. In order to help diagnose loading problems, consider setting the DYLD_PRINT_LIBRARIES environment variable: dlopen(libSystem.Security.Cryptography.Native.OpenSsl, 1): image not found
      Stack Trace:
           at Interop.Crypto.DsaKeyCreateByExplicitParameters(SafeDsaHandle& dsa, Byte[] p, Int32 pLength, Byte[] q, Int32 qLength, Byte[] g, Int32 gLength, Byte[] y, Int32 yLength, Byte[] x, Int32 xLength)
        /_/src/Common/src/System/Security/Cryptography/DSAOpenSsl.cs(112,0): at System.Security.Cryptography.DSAOpenSsl.ImportParameters(DSAParameters parameters)
        /_/src/System.Security.Cryptography.Algorithms/src/System/Security/Cryptography/DSA.cs(349,0): at System.Security.Cryptography.DSA.ImportEncryptedPkcs8PrivateKey(ReadOnlySpan`1 passwordBytes, ReadOnlySpan`1 source, Int32& bytesRead)
        /_/src/Common/src/System/Security/Cryptography/DSAOpenSsl.cs(132,0): at System.Security.Cryptography.DSAOpenSsl.ImportEncryptedPkcs8PrivateKey(ReadOnlySpan`1 passwordBytes, ReadOnlySpan`1 source, Int32& bytesRead)
        /_/src/Common/tests/System/Security/Cryptography/AlgorithmImplementations/DSA/DSAKeyFileTests.cs(593,0): at System.Security.Cryptography.Dsa.Tests.DSAKeyFileTests.<>c__DisplayClass23_0.<ReadBase64EncryptedPkcs8>b__0(DSA dsa, ReadOnlySpan`1 source, Int32& read)
        /_/src/Common/tests/System/Security/Cryptography/AlgorithmImplementations/DSA/DSAKeyFileTests.cs(652,0): at System.Security.Cryptography.Dsa.Tests.DSAKeyFileTests.ReadWriteKey(String base64, DSAParameters& expected, ReadKeyAction readAction, Func`2 writeArrayFunc, WriteKeyToSpanFunc writeSpanFunc, Boolean isEncrypted)
        /_/src/Common/tests/System/Security/Cryptography/AlgorithmImplementations/DSA/DSAKeyFileTests.cs(589,0): at System.Security.Cryptography.Dsa.Tests.DSAKeyFileTests.ReadBase64EncryptedPkcs8(String base64EncPkcs8, Byte[] passwordBytes, PbeParameters pbeParameters, DSAParameters& expected)
        /_/src/Common/tests/System/Security/Cryptography/AlgorithmImplementations/DSA/DSAKeyFileTests.cs(174,0): at System.Security.Cryptography.Dsa.Tests.DSAKeyFileTests.ReadWriteDsa1024EncryptedPkcs8_PasswordBytes()
        ----- Inner Stack Trace -----
        /_/src/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.Initialization.cs(40,0): at Interop.CryptoInitializer.Initialize()
        /_/src/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.Initialization.cs(19,0): at Interop.Crypto..cctor()
        ----- Inner Stack Trace -----
           at Interop.CryptoInitializer.EnsureOpenSslInitialized()
        /_/src/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.Initialization.cs(27,0): at Interop.CryptoInitializer..cctor()