Invoke-RestMethod/WebRequest: Support CTRL-C when reading data and connection hangs

[stevenebutler]

APIs that should be cancelled are passed the cancellation token so that CTRL-C can cancel even when processing long or timed-out response streams.

Switched use of synchronous stream APIs to asynchronous APIs that use the cancellation token.

PR Summary

This PR closes #16145

The root cause of these issues was that once the response headers are read, the cancellation token was not always being passed to APIs that deal with reading the response stream.

In some scenarios, CTRL-C worked (where the cancellation token was being passed down), but this was not occurring in all scenarios and the use of blocking read APIs that do not take cancellation tokens prevented the early cancellation when CTRL-C was used.

PR Context

This PR helps because it allows all I/O in the web requests cmdlets to be cancelled if the user uses CTRL-C to terminate processing.

A previous iteration of this PR added a timeout, but was incorrect as the desired behaviour is to cancel if the connection is stalled for 5 minutes, not after a fixed timeout.

PR Checklist

• PR has a meaningful title
  • Use the present tense and imperative mood when describing your changes
• Summarized changes
• Make sure all .h, .cpp, .cs, .ps1 and .psm1 files have the correct copyright header
• This PR is ready to merge and is not Work in Progress.
  • If the PR is work in progress, please add the prefix WIP: or [ WIP ] to the beginning of the title (the WIP bot will keep its status check at Pending while the prefix is present) and remove the prefix when the PR is ready.
• Breaking changes
  • None
  • OR
  • Experimental feature(s) needed
    • Experimental feature name(s):
• User-facing changes
  • Not Applicable
  • OR
  • Documentation needed
    • Issue filed:
• Testing - New and feature
  • N/A or can only be tested interactively
  • OR
  • Make sure you've added a new test if existing tests do not effectively test the code changed
• Tooling
  • I have considered the user experience from a tooling perspective and don't believe tooling will be impacted.
  • OR
  • I have considered the user experience from a tooling perspective and opened an issue in the relevant tool repository. This may include:
    • Impact on PowerShell Editor Services which is used in the PowerShell extension for VSCode
      (which runs in a different PS Host).
      • Issue filed:
    • Impact on Completions (both in the console and in editors) - one of PowerShell's most powerful features.
      • Issue filed:
    • Impact on PSScriptAnalyzer (which provides linting & formatting in the editor extensions).
      • Issue filed:
    • Impact on EditorSyntax (which provides syntax highlighting with in VSCode, GitHub, and many other editors).
      • Issue filed:

[stevenebutler]

@CarloToso and @iSazonov - I renamed the branch to better reflect what this is doing. This closed the PR #19315 I had open so I've opened this one instead for review. Changes are similar to what @CarloToso already reviewed, but updated against latest master changes and with all code factor issues in changed files resolved.

[adityapatwardhan]

I think we should add tests for this behavior. I was thinking of executing Invoke-WebRequest and Invoke-RestMethod under the PowerShell API and call a Stop() on it. That should be equivalent to sending a cancellation.

Example of using the API:

$ps = [PowerShell]::Create()
$ps.AddScript("Invoke-WebRequest .......")
$ps.Stop()

and then use Wait-UntilTrue to verify that the pipeline was stopped in a resonable time.

[adityapatwardhan]

We should add a test.

[stevenebutler]

I think we should add tests for this behavior. I was thinking of executing Invoke-WebRequest and Invoke-RestMethod under the PowerShell API and call a Stop() on it. That should be equivalent to sending a cancellation.

Example of using the API:

$ps = [PowerShell]::Create()
$ps.AddScript("Invoke-WebRequest .......")
$ps.Stop()

and then use Wait-UntilTrue to verify that the pipeline was stopped in a resonable time.

Ok, I'll look into it when I have some spare time. I assume there's a way to get the web server harness to delay writing, or should I modify it to permit that if it doesn't already?

[adityapatwardhan]

There is a Delay in the WebListener -

PowerShell/test/tools/WebListener/Controllers/DelayController.cs

Lines 17 to 29 in 4767c8c

	public class DelayController : Controller
	{
	public JsonResult Index(int seconds)
	{
	if (seconds > 0){
	int milliseconds = seconds * 1000;
	Thread.Sleep(milliseconds);
	}
	var getController = new GetController();
	getController.ControllerContext = this.ControllerContext;
	return getController.Index();
	}

[stevenebutler]

There is a Delay in the WebListener -

PowerShell/test/tools/WebListener/Controllers/DelayController.cs

Lines 17 to 29 in 4767c8c

	public class DelayController : Controller
	{
	public JsonResult Index(int seconds)
	{
	if (seconds > 0){
	int milliseconds = seconds * 1000;
	Thread.Sleep(milliseconds);
	}
	var getController = new GetController();
	getController.ControllerContext = this.ControllerContext;
	return getController.Index();
	}

The delay has to occur after headers are sent to test this change. That code looks like it may delay the response headers until after the delay. I can add something to delay after sending headers if that’s okay.

[adityapatwardhan]

Sure, please add more functionality to WebListener if needed.

[iSazonov]

The PR does not solve the problem completely as I pointed in related issue.

[stevenebutler]

The PR does not solve the problem completely as I pointed in related issue.

I reviewed your comment in the related issue and I don't agree with your conclusion. You claim that cancellation tokens don't always work with the asynchronous API but the issue you link to does not support your argument. In that issue the core team dispute that cancellation tokens can't cancel the async APIs and provided some sample code that would solve the issue.

It looks like you want to use synchronous APIs with HttpClient and have it honour timeouts set on the HttpClient but were advised that the synchronous APIs are incomplete and won't do what you want. To resolve this issue we should be using the async APIs instead since they are fully cancellable (and if they're not then that's a legitimate bug the core team would fix if we can provide them with a reproducible test case).

If you have a specific test case that this PR doesn't solve for hanging on CTRL-C then it would be an oversight on my part (e.g. missing converting one of the synchronous reads to asynchronous with cancellation token), and I will look into it.

[stevenebutler]

Sure, please add more functionality to WebListener if needed.

I have added a range of tests around the CTRL-C handling as suggested. I have also manually tried some of these cases using the /Stall endpoint in the WebListener and can confirm that once the download has started with existing PowerShell, the response cannot be cancelled if the stream has stalled.

With the changes, a stalled stream cancels immediately when CTRL-C is pressed.

I do have some concerns the tests may be a little timing sensitive since we are racing the CTRL-C with the response headers already having been received and there's no easy way to synchronise that. I've tried to keep the times really short so the tests don't take too long, but they could be increased if the tests are flaky in the CI system.

[iSazonov]

If you have a specific test case that this PR doesn't solve

You can see in related dotnet issue.

It looks like you want to use synchronous APIs with HttpClient

Web cmdlets are synchronous by design.
Asynchronous methods can be used for some reasons, but they do not make this code synchronous.
I already tried to do what you do and it is not a general solution and only complicates the code. All we need to do is stop the cmdlet and for this we don't need to implement CTS in the operations available to us because there are unavailable ones.

[stevenebutler]

If you have a specific test case that this PR doesn't solve

You can see in related dotnet issue.

It looks like you want to use synchronous APIs with HttpClient

Web cmdlets are synchronous by design. Asynchronous methods can be used for some reasons, but they do not make this code synchronous. I already tried to do what you do and it is not a general solution and only complicates the code. All we need to do is stop the cmdlet and for this we don't need to implement CTS in the operations available to us because there are unavailable ones.

There do not appear to be any async versions of the methods that don't accept a CTS, and they were changed from synchronousOp(...) to asyncOp(..., cts).GetAwaiter.GetResult() in the patch because WebCmdlets are synchronous.

The PR includes tests that test the web cmdlets will cancel immediately while the connection is stalled for a variety of response types, including download to file, processing ATOM responses and processing JSON responses which definitely don't work with PS 7.3.3 and work with the PR.

The patch only makes an exception for using CTS when reading from memorystream, since they do not require IO so should terminate within a reasonable time.

If there's a specific case I've missed please describe it so the PR can be improved to address it (or rejected if it cannot be fixed).

[stevenebutler]

@adityapatwardhan I'm not sure why it says I asked to remove the other review requests. I had requested you review the change again after I added unit tests for the feature as per your request. If those other people need to review as well, can you add them back to the PR reviewers?

[stevenebutler]

Hi @adityapatwardhan - could you please provide some guidance on whether this PR is going to be reviewed again soon, or whether I should just close it because it is unsuitable?

[adityapatwardhan]

@stevenebutler = Reviewing it today. sorry for the delay.

[stevenebutler]

Hi @adityapatwardhan , thank you for the review. I see the macos build has timed out and is blocking the merge. Is there something I should do to address this? It looks unrelated to my change.

Thanks for removing the additional tag.

[pull-request-quantifier-deprecated]

This PR has 285 quantified lines of changes. In general, a change size of upto 200 lines is ideal for the best PR experience!

---

Quantification details

Label      : Large
Size       : +232 -53
Percentile : 68.5%

Total files changed: 10

Change summary by file extension:
.cs : +147 -53
.ps1 : +81 -0
.psm1 : +4 -0

Change counts above are quantified counts, based on the PullRequestQuantifier customizations.

Why proper sizing of changes matters

Optimal pull request sizes drive a better predictable PR flow as they strike a
balance between between PR complexity and PR review overhead. PRs within the
optimal size (typical small, or medium sized PRs) mean:

• Fast and predictable releases to production:
  • Optimal size changes are more likely to be reviewed faster with fewer
    iterations.
  • Similarity in low PR complexity drives similar review times.
• Review quality is likely higher as complexity is lower:
  • Bugs are more likely to be detected.
  • Code inconsistencies are more likely to be detected.
• Knowledge sharing is improved within the participants:
  • Small portions can be assimilated better.
• Better engineering practices are exercised:
  • Solving big problems by dividing them in well contained, smaller problems.
  • Exercising separation of concerns within the code changes.

What can I do to optimize my changes

• Use the PullRequestQuantifier to quantify your PR accurately
  • Create a context profile for your repo using the context generator
  • Exclude files that are not necessary to be reviewed or do not increase the review complexity. Example: Autogenerated code, docs, project IDE setting files, binaries, etc. Check out the Excluded section from your prquantifier.yaml context profile.
  • Understand your typical change complexity, drive towards the desired complexity by adjusting the label mapping in your prquantifier.yaml context profile.
  • Only use the labels that matter to you, see context specification to customize your prquantifier.yaml context profile.
• Change your engineering behaviors
  • For PRs that fall outside of the desired spectrum, review the details and check if:
    • Your PR could be split in smaller, self-contained PRs instead
    • Your PR only solves one particular issue. (For example, don't refactor and code new features in the same PR).

How to interpret the change counts in git diff output

• One line was added: +1 -0
• One line was deleted: +0 -1
• One line was modified: +1 -1 (git diff doesn't know about modified, it will
  interpret that line like one addition plus one deletion)
• Change percentiles: Change characteristics (addition, deletion, modification)
  of this PR in relation to all other PRs within the repository.

---

Was this comment helpful? 👍  :ok_hand:  :thumbsdown: (Email)
Customize PullRequestQuantifier for this repository.

[adityapatwardhan]

@stevenebutler Thank you for your contribution!

[daxian-dbw]

@stevenebutler
Some of the Ctrl-C tests added in this PR are fragile and have been failing a lot in CIs, one example: https://github.com/PowerShell/PowerShell/pull/19529/checks?check_run_id=12840094279.

Invoke-WebRequest and Invoke-RestMethod support Cancellation through CTRL-C.Invoke-RestMethod: CTRL-C after stalled JSON download processes JSON response
Expected $true, but got $false.

Can you please take a look and make the tests more reliable? /cc @adityapatwardhan

[stevenebutler]

@daxian-dbw The tests rely on timeouts occurring within about a second of when you'd normally expect them to occur which might be a bit risky if running on heavily subscribed hardware.

The only way I can think of to make them more reliable is to give a larger window for the passing of time vs the expectation that the cancellation will take affect. What is generally considered a reasonable test duration when testing for timeouts?

[stevenebutler]

Have created #19532 which increases timeouts but it is failing on macos due to an unrelated test right now.

[ghost]

🎉v7.4.0-preview.3 has been released which incorporates this pull request.:tada:

Handy links:

• Release Notes