[QUIC] Adding CompleteWritesAsync

[ManickaP]

Implements async CompleteWritesAsync which will eventually replace sync CompleteWrites.

The behavior in this PR now mimics combination of CompleteWrites and WritesClosed. This PR also changes behavior of WritesAsync(..., completeWrites: true) to also wait for SEND_SHUTDOWN_COMPLETE (or abort). This PR brings few breaking behavioral changes:

• WritesAsync(..., completeWrites: true)
  • it's now not possible to separate when the data are buffered (SEND_COMPLETE) and when the completion is ACKed, making the "last write" potentially much longer
  • the "last write" now may start throwing in case of local Abort(write) (RESET_STREAM) or remote Abort(read) (STOP_SENDING)
• CompleteWritesAsync
  • as it's now async, the user doesn't have an easy way to fire-and-forget stream write completion

The behavior is up for discussion, some options:

• don't do anything and leave the QuicStream as it was before this change
• only report success in CompletWritesAsync and never throw (Exception is still part of CompleteWrites task)

Unfortunately, this effectively reverts #103902 as the WriteAsync now have to chain to async calls in a row...

Fixes #103434

[dotnet-issue-labeler]

Note regarding the new-api-needs-documentation label:

This serves as a reminder for when your PR is modifying a ref *.cs file and adding/modifying public APIs, please make sure the API implementation in the src *.cs file is documented with triple slash comments, so the PR reviewers can sign off that change.

[dotnet-issue-labeler]

Note regarding the new-api-needs-documentation label:

This serves as a reminder for when your PR is modifying a ref *.cs file and adding/modifying public APIs, please make sure the API implementation in the src *.cs file is documented with triple slash comments, so the PR reviewers can sign off that change.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

[MihaZupan]

Unfortunately, this effectively reverts #103902 as the WriteAsync now have to chain to async calls in a row...

But only for the one final write that has completeWrites set, right?
Could we avoid that by changing

await valueTask.ConfigureAwait(false);
if (completeWrites)
{
    await _sendTcs.GetFinalTask(this).ConfigureAwait(false);
}

to

return completeWrites
    ? WaitForTaskAndSendTcsAsync(valueTask)
    : valueTask;

async WaitForTaskAndSendTcsAsync(ValueTask valueTask)
{
    await valueTask.ConfigureAwait(false);
    await _sendTcs.GetFinalTask(this).ConfigureAwait(false);
}

?

We use similar tricks elsewhere in HTTP, e.g. in HttpConnection.WriteAsync.

---

Equivalent to using WriteAsync with completeWrites: true.

Are we exposing the CompleteWritesAsync method purely as a nicer-to-write helper compared to WriteAsync([], true)?

[ManickaP]

But only for the one final write that has completeWrites set, right?

Yes and the trick would work 👍

---

Are we exposing the CompleteWritesAsync method purely as a nicer-to-write helper compared to WriteAsync([], true)?

Basically yes.

[ManickaP]

Closing, decided not to go with the API.