Implement TranscodingStream, a streaming equivalent of Encoding.Convert

[GrabYourPitchforks]

Resolves #30260.

The general idea is that if you have an inner Stream which represents machine-readable text under one encoding (say, shift-jis), this allows creating a wrapping Stream which represents that same machine-readable text under a different encoding (say, UTF-8). The primary scenario for this is as an adapter where you have System.Text.Json (as UTF-8) working on the outer Stream but where the client needs the response payload in a different encoding.

Usage (showing writing)

Encoding responseEncoding = GetResponseEncoding();
Stream responseStream = OpenResponseStream();

if (responseEncoding is UTF8Encoding)
{
    // write directly to the response
   await JsonSerializer.SerializeAsync<T>(responseStream, ...);
}
else
{
    // insert our adapter
    Stream transcodingStream = Encoding.CreateTranscodingStream(
        innerStream: responseStream,
        innerStreamEncoding: responseEncoding,
        outerStreamEncoding: Encoding.UTF8 /* this is what System.Text.Json emits */,
        leaveOpen: true /* if needed */);
    await JsonSerializer.SerializeAsync<T>(transcodingStream, ...);
    await transcodingStream.DisposeAsync();
}

Note: When writing, the call to Stream.Close / Stream.Dispose / Stream.DisposeAsync is the thing that flushes the underlying Encoder / Decoder instances. This shouldn't normally matter as long as the caller isn't trying to write ill-formed data to the outer Stream. But if the caller does emit an incomplete multi-byte sequence (say, the UTF-8 bytes [ E0 BF ]) at the end of the stream, the fallback mechanism won't be invoked until the outer stream is disposed.

[Dotnet-GitSync-Bot]

Note regarding the new-api-needs-documentation label:

This serves as a reminder for when your PR is modifying a ref *.cs file and adding/modifying public APIs, to please make sure the API implementation in the src *.cs file is documented with triple slash comments, so the PR reviewers can sign off that change.

[ghost]

Tagging subscribers to this area: @tarekgh
Notify danmosemsft if you want to be subscribed.

[GrabYourPitchforks]

And yes, I know I have array pool rentals inside a try / finally block. This follows the pattern already established by Stream, and I've spoken with Jan offline who has attempted to assuage my paranoia in this regard. :)

[GrabYourPitchforks]

/cc @pranavkm @rynowak since they were also asking for this

[jozkee]

Thanks for updating System.Net.Http.Json as well.