CryptoStreamTests ReadAsync_CancelPendingTask_ThrowsCancellationException long running test on Ubuntu

[Jozkee]

Found in #66641

https://dev.azure.com/dnceng/public/_build/results?buildId=1663765&view=ms.vss-test-web.build-test-results-tab

https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-66641-merge-baf0875ff584492eb0/System.Security.Cryptography.Tests/1/console.e1f9ab07.log?sv=2019-07-07&se=2022-04-04T23%3A54%3A40Z&sr=c&sp=rl&sig=0Xj99sZqOKidsi7uCXqA%2BJiZVCyhU3wTTD4SRaRxxxA%3D

   System.Security.Cryptography.Tests: [Long Running Test] 'System.Security.Cryptography.Tests.CryptoStreamTests.ReadAsync_CancelPendingTask_ThrowsCancellationException', Elapsed: 00:03:15
   System.Security.Cryptography.Tests: [Long Running Test] 'System.Security.Cryptography.Tests.CryptoStreamTests.ReadAsync_CancelPendingTask_ThrowsCancellationException', Elapsed: 00:05:15
   System.Security.Cryptography.Tests: [Long Running Test] 'System.Security.Cryptography.Tests.CryptoStreamTests.ReadAsync_CancelPendingTask_ThrowsCancellationException', Elapsed: 00:07:15
   System.Security.Cryptography.Tests: [Long Running Test] 'System.Security.Cryptography.Tests.CryptoStreamTests.ReadAsync_CancelPendingTask_ThrowsCancellationException', Elapsed: 00:09:15
   System.Security.Cryptography.Tests: [Long Running Test] 'System.Security.Cryptography.Tests.CryptoStreamTests.ReadAsync_CancelPendingTask_ThrowsCancellationException', Elapsed: 00:11:15
   System.Security.Cryptography.Tests: [Long Running Test] 'System.Security.Cryptography.Tests.CryptoStreamTests.ReadAsync_CancelPendingTask_ThrowsCancellationException', Elapsed: 00:13:15
...
[EXECUTION TIMED OUT]
Exit Code:-3Executor timed out after 900 seconds and was killed
['System.Security.Cryptography.Tests' END OF WORK ITEM LOG: Command timed out, and was killed]

[ghost]

Tagging subscribers to this area: @dotnet/area-system-security, @vcsjones
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Found in #66641

https://dev.azure.com/dnceng/public/_build/results?buildId=1663765&view=ms.vss-test-web.build-test-results-tab

https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-66641-merge-baf0875ff584492eb0/System.Security.Cryptography.Tests/1/console.e1f9ab07.log?sv=2019-07-07&se=2022-04-04T23%3A54%3A40Z&sr=c&sp=rl&sig=0Xj99sZqOKidsi7uCXqA%2BJiZVCyhU3wTTD4SRaRxxxA%3D

   System.Security.Cryptography.Tests: [Long Running Test] 'System.Security.Cryptography.Tests.CryptoStreamTests.ReadAsync_CancelPendingTask_ThrowsCancellationException', Elapsed: 00:03:15
   System.Security.Cryptography.Tests: [Long Running Test] 'System.Security.Cryptography.Tests.CryptoStreamTests.ReadAsync_CancelPendingTask_ThrowsCancellationException', Elapsed: 00:05:15
   System.Security.Cryptography.Tests: [Long Running Test] 'System.Security.Cryptography.Tests.CryptoStreamTests.ReadAsync_CancelPendingTask_ThrowsCancellationException', Elapsed: 00:07:15
   System.Security.Cryptography.Tests: [Long Running Test] 'System.Security.Cryptography.Tests.CryptoStreamTests.ReadAsync_CancelPendingTask_ThrowsCancellationException', Elapsed: 00:09:15
   System.Security.Cryptography.Tests: [Long Running Test] 'System.Security.Cryptography.Tests.CryptoStreamTests.ReadAsync_CancelPendingTask_ThrowsCancellationException', Elapsed: 00:11:15
   System.Security.Cryptography.Tests: [Long Running Test] 'System.Security.Cryptography.Tests.CryptoStreamTests.ReadAsync_CancelPendingTask_ThrowsCancellationException', Elapsed: 00:13:15
...
[EXECUTION TIMED OUT]
Exit Code:-3Executor timed out after 900 seconds and was killed
['System.Security.Cryptography.Tests' END OF WORK ITEM LOG: Command timed out, and was killed]

Author:	Jozkee
Assignees:	-
Labels:	area-System.Security
Milestone:	-

[danmoseley]

Hit on Debian

Console log: 'System.Security.Cryptography.Tests' from job 190fd8e4-25bd-47c6-a867-e98046f2c32d (ubuntu.1804.amd64.open.rt) using docker image mcr.microsoft.com/dotnet-buildtools/prereqs:debian-10-helix-amd64-20210304164434-56c6673 on a00B4BO

[danmoseley]

and again
Console log: 'System.Security.Cryptography.Tests' from job edae4e06-4cc9-48f4-84fc-93f75ec695fe (ubuntu.1804.amd64.open.rt) using docker image mcr.microsoft.com/dotnet-buildtools/prereqs:debian-10-helix-amd64-20210304164434-56c6673 on a00B5DP

#67003

[danmoseley]

I tried to see how often this was happening, but there's ?another failure mode where it times out, but doesn't say long running test.

https://engsrvprod.kusto.windows.net/engineeringdata?query=H4sIAAAAAAAEAEWNsQrCQBAFe8F%2fWK6KEINgHRslYKEgCVjK4W7MorkLe3vEAz%2fexMb6vZm5enkelfqwXHxg7EgIKmFy%2bEpn2xOUJZg6helR1HSPwpqKvaRB%2fUPs0KWioaDBgHUItVqN4Yc03JOPav7WS6RICDtwfszW2w2u5o3eOrXgEMUqewcloFXSib4ht21mTuyiksmhYsehI8znjCjN%2fBf6f%2biQvQAAAA%3d%3d&web=0

WorkItems
| where FriendlyName == "System.Security.Cryptography.Tests" and Status == "Timeout"
| where Queued > now(-30d)
| extend Duration = datetime_diff("Minute", Finished, Started)

[danmoseley]

However, the query above shows 211 timeouts in the last month, which is a lot.

[joperezr]

Failed again in f35aacba-469a-400c-8bd7-ddab3aa23801 which was our rolling build so marking this issue as blocking-clean-ci https://dev.azure.com/dnceng/public/_build/results?buildId=1677596&view=logs&j=3f13e9dd-5fbe-5edd-198d-04c8537bca02&t=2dfef947-451a-5d13-09e9-614229052870

[bartonjs]

Any thoughts, @stephentoub? As far as I can tell, your ReadAsync_CancelPendingTask_ThrowsCancellationException test is pretty simple, and CryptoStream's ReadAsync passes the token down to deferred stream.ReadAsync calls. (It can complete synchronously even with the token set to cancelled, but that wouldn't cause a hang).

[stephentoub]

Is this only failing on mono interpreter legs?

[adamsitnik]

Failed with timeout for Libraries Test Run release mono_interpreter Linux x64 Release

For details please refer to console logs

[joperezr]

Failed again: https://dev.azure.com/dnceng/public/_build/results?buildId=1697303&view=logs&j=3f13e9dd-5fbe-5edd-198d-04c8537bca02&t=2dfef947-451a-5d13-09e9-614229052870 on a mono interpreter run

[danmoseley]

@BrzVlad any thoughts why this only happens on Mono interpreter?

[BrzVlad]

Fixed by #67663