-
Notifications
You must be signed in to change notification settings - Fork 7.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Invoke-RestMethod/WebRequest: Support CTRL-C when reading data and connection hangs #19330
Conversation
@CarloToso and @iSazonov - I renamed the branch to better reflect what this is doing. This closed the PR #19315 I had open so I've opened this one instead for review. Changes are similar to what @CarloToso already reviewed, but updated against latest master changes and with all code factor issues in changed files resolved. |
I think we should add tests for this behavior. I was thinking of executing Invoke-WebRequest and Invoke-RestMethod under the PowerShell API and call a Example of using the API: $ps = [PowerShell]::Create()
$ps.AddScript("Invoke-WebRequest .......")
$ps.Stop() and then use |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should add a test.
Ok, I'll look into it when I have some spare time. I assume there's a way to get the web server harness to delay writing, or should I modify it to permit that if it doesn't already? |
There is a Delay in the WebListener - PowerShell/test/tools/WebListener/Controllers/DelayController.cs Lines 17 to 29 in 4767c8c
|
The delay has to occur after headers are sent to test this change. That code looks like it may delay the response headers until after the delay. I can add something to delay after sending headers if that’s okay. |
Sure, please add more functionality to WebListener if needed. |
The PR does not solve the problem completely as I pointed in related issue. |
I reviewed your comment in the related issue and I don't agree with your conclusion. You claim that cancellation tokens don't always work with the asynchronous API but the issue you link to does not support your argument. In that issue the core team dispute that cancellation tokens can't cancel the async APIs and provided some sample code that would solve the issue. It looks like you want to use synchronous APIs with HttpClient and have it honour timeouts set on the HttpClient but were advised that the synchronous APIs are incomplete and won't do what you want. To resolve this issue we should be using the async APIs instead since they are fully cancellable (and if they're not then that's a legitimate bug the core team would fix if we can provide them with a reproducible test case). If you have a specific test case that this PR doesn't solve for hanging on CTRL-C then it would be an oversight on my part (e.g. missing converting one of the synchronous reads to asynchronous with cancellation token), and I will look into it. |
I have added a range of tests around the CTRL-C handling as suggested. I have also manually tried some of these cases using the /Stall endpoint in the WebListener and can confirm that once the download has started with existing PowerShell, the response cannot be cancelled if the stream has stalled. With the changes, a stalled stream cancels immediately when CTRL-C is pressed. I do have some concerns the tests may be a little timing sensitive since we are racing the CTRL-C with the response headers already having been received and there's no easy way to synchronise that. I've tried to keep the times really short so the tests don't take too long, but they could be increased if the tests are flaky in the CI system. |
You can see in related dotnet issue.
Web cmdlets are synchronous by design. |
There do not appear to be any async versions of the methods that don't accept a CTS, and they were changed from synchronousOp(...) to asyncOp(..., cts).GetAwaiter.GetResult() in the patch because WebCmdlets are synchronous. The PR includes tests that test the web cmdlets will cancel immediately while the connection is stalled for a variety of response types, including download to file, processing ATOM responses and processing JSON responses which definitely don't work with PS 7.3.3 and work with the PR. The patch only makes an exception for using CTS when reading from memorystream, since they do not require IO so should terminate within a reasonable time. If there's a specific case I've missed please describe it so the PR can be improved to address it (or rejected if it cannot be fixed). |
5c48c2b
to
0235558
Compare
1cc514c
to
ef95109
Compare
@adityapatwardhan I'm not sure why it says I asked to remove the other review requests. I had requested you review the change again after I added unit tests for the feature as per your request. If those other people need to review as well, can you add them back to the PR reviewers? |
Uses async APIs with cancellation tokens to support cancellation throughout the download process. Includes tests for stopping stalled downloads with different response types, HTTP vs HTTPS and compression streams.
ef95109
to
8e21076
Compare
Hi @adityapatwardhan - could you please provide some guidance on whether this PR is going to be reviewed again soon, or whether I should just close it because it is unsuitable? |
@stevenebutler = Reviewing it today. sorry for the delay. |
test/powershell/Modules/Microsoft.PowerShell.Utility/WebCmdlets.Tests.ps1
Outdated
Show resolved
Hide resolved
Hi @adityapatwardhan , thank you for the review. I see the macos build has timed out and is blocking the merge. Is there something I should do to address this? It looks unrelated to my change. Thanks for removing the additional tag. |
…uest-and-invoke-restmethod-when-connection-hangs
This PR has Quantification details
Why proper sizing of changes matters
Optimal pull request sizes drive a better predictable PR flow as they strike a
What can I do to optimize my changes
How to interpret the change counts in git diff output
Was this comment helpful? 👍 :ok_hand: :thumbsdown: (Email) |
@stevenebutler Thank you for your contribution! |
* Remove FormObject.cs and FormObjectCollection.cs (PowerShell#19383) * Exclude redundant parameter aliases from completion results (PowerShell#19382) * Revert "Remove FormObject.cs and FormObjectCollection.cs (PowerShell#19383)" (PowerShell#19387) This reverts commit 190c99a. * Add the parameter `-RelativeBasePath` to `Resolve-Path` (PowerShell#19358) * Fix a crash in the type inference code (PowerShell#19400) * Remove GetResponseObject (PowerShell#19380) * Add `-Environment` parameter to `Start-Process` (PowerShell#19374) * Add `-Environment` parameter to `Start-Process` * address codefactor * fix test for Windows * handle case where value is $null to remove env var * change variables to make it more clear what the test is doing * Add PoolNames variable group to compliance pipeline (PowerShell#19408) * Improve package management acceptance tests by not going to the gallery (PowerShell#19412) * Skip VT100 tests on Windows Server 2012R2 as console does not support it (PowerShell#19413) * Update the `ICommandPredictor` interface to reduce boilerplate code from predictor implementation (PowerShell#19414) * Enable type conversion of `AutomationNull` to `$null` for assignment (PowerShell#19415) * Remove code related to `#requires -pssnapin` (PowerShell#19320) * Support CTRL-C when reading data and connection hangs for `Invoke-RestMethod` and `Invoke-WebRequest` (PowerShell#19330) * Update to the latest NOTICES file (PowerShell#19332) * Update the cgmanifest (PowerShell#19459) * WIP: Harden default command test. (PowerShell#19416)
@stevenebutler
Can you please take a look and make the tests more reliable? /cc @adityapatwardhan |
@daxian-dbw The tests rely on timeouts occurring within about a second of when you'd normally expect them to occur which might be a bit risky if running on heavily subscribed hardware. The only way I can think of to make them more reliable is to give a larger window for the passing of time vs the expectation that the cancellation will take affect. What is generally considered a reasonable test duration when testing for timeouts? |
Have created #19532 which increases timeouts but it is failing on macos due to an unrelated test right now. |
🎉 Handy links: |
APIs that should be cancelled are passed the cancellation token so that CTRL-C can cancel even when processing long or timed-out response streams.
Switched use of synchronous stream APIs to asynchronous APIs that use the cancellation token.
PR Summary
This PR closes #16145
The root cause of these issues was that once the response headers are read, the cancellation token was not always being passed to APIs that deal with reading the response stream.
In some scenarios, CTRL-C worked (where the cancellation token was being passed down), but this was not occurring in all scenarios and the use of blocking read APIs that do not take cancellation tokens prevented the early cancellation when CTRL-C was used.
PR Context
This PR helps because it allows all I/O in the web requests cmdlets to be cancelled if the user uses CTRL-C to terminate processing.
A previous iteration of this PR added a timeout, but was incorrect as the desired behaviour is to cancel if the connection is stalled for 5 minutes, not after a fixed timeout.
PR Checklist
.h
,.cpp
,.cs
,.ps1
and.psm1
files have the correct copyright headerWIP:
or[ WIP ]
to the beginning of the title (theWIP
bot will keep its status check atPending
while the prefix is present) and remove the prefix when the PR is ready.(which runs in a different PS Host).