Powershell running as a child process spawned by SSHD on linux crashes on progress tracking

[et1975]

Prerequisites

• Write a descriptive title.
• Make sure you are able to repro it on the latest released version
• Search the existing issues.
• Refer to the FAQ.
• Refer to Differences between Windows PowerShell 5.1 and PowerShell.

Steps to reproduce

Very esoteric setup, but I can consistently reproduce pwsh child process stared via Enter-PSSession over SSH crashing with SIGABRT when using Posh-SSH to upload a file over SFTP.
Alpine and Debian show the same problem.

Reproduction Steps
Use an mcr powershell container as the base, add SSHD and configure Powershell subsystem.
Something like

FROM mcr.microsoft.com/powershell:7.2.1-debian-bullseye-slim
RUN apt-get update \
    && apt-get -y install openssh-server \
    && rm -rf /var/lib/apt/lists/* \
    && mkdir -p /var/run/sshd \
    && ssh-keygen -A \
    && sed "s/Subsystem.*sftp.*/Subsystem powershell \/usr\/bin\/pwsh -sshs -NoLogo -NoProfile/" -i /etc/ssh/sshd_config
ENTRYPOINT ["/usr/sbin/sshd", "-D", "-e"]

Start remote session into such a container, initiate an upload with Set-SFTPItem - it crashes at random, the larger the file the more likely it is to happen.

"The SSH client session has ended with error message: The SSH transport process has abruptly terminated causing this remote session to break."

Expected behavior

Should not crash.

Actual behavior

Crashes with this stack trace:

* thread #39, name = '.NET ThreadPool', stop reason = signal SIGABRT
    frame #0: 0x00007fcd715fbce1 libc.so.6`raise + 321
libc.so.6`raise:
->  0x7fcd715fbce1 <+321>: movq   0x108(%rsp), %rax
    0x7fcd715fbce9 <+329>: subq   %fs:0x28, %rax
    0x7fcd715fbcf2 <+338>: jne    0x7fcd715fbd14            ; <+372>
    0x7fcd715fbcf4 <+340>: movl   %r8d, %eax
(lldb) clrstack
OS Thread Id: 0x2632 (39)
        Child SP               IP Call Site
00007FCB4F7FD680 00007fcd715fbce1 [HelperMethodFrame: 00007fcb4f7fd680] 
00007FCB4F7FD800 00007FCD01FB9158 System.ThrowHelper.ThrowArgumentOutOfRange_IndexException() [/_/src/libraries/System.Private.CoreLib/src/System/ThrowHelper.cs @ 101]
00007FCB4F7FD810 00007FCCFCEB10C1 System.Collections.Generic.List`1[[System.__Canon, System.Private.CoreLib]].get_Item(Int32) [/_/src/libraries/System.Private.CoreLib/src/System/Collections/Generic/List.cs @ 150]
00007FCB4F7FD820 00007FCCF876B8F1 System.Management.Automation.PSDataCollection`1[[System.__Canon, System.Private.CoreLib]].get_Item(Int32)
00007FCB4F7FD860 00007FCCF83BE6F8 System.Management.Automation.ServerPowerShellDriver.HandleProgressAdded(System.Object, System.Management.Automation.DataAddedEventArgs)
00007FCB4F7FD8A0 00007FCCF876CE5D System.Management.Automation.PSDataCollection`1[[System.__Canon, System.Private.CoreLib]].RaiseDataAddedEvent(System.Guid, Int32)
00007FCB4F7FD8E0 00007FCCF876CD44 System.Management.Automation.PSDataCollection`1[[System.__Canon, System.Private.CoreLib]].RaiseEvents(System.Guid, Int32)
00007FCB4F7FD940 00007FCCF876CF85 System.Management.Automation.PSDataCollection`1[[System.__Canon, System.Private.CoreLib]].InternalAdd(System.Guid, System.__Canon)
00007FCB4F7FD9A0 00007FCCF833CE4A System.Management.Automation.PSInformationalBuffers.AddProgress(System.Management.Automation.ProgressRecord)
00007FCB4F7FD9B0 00007FCCF85C2D32 System.Management.Automation.Internal.Host.InternalHostUserInterface.WriteProgress(Int64, System.Management.Automation.ProgressRecord)
00007FCB4F7FD9E0 00007FCD01FB6A1E SSH.OperationProgressHelper+<>c__DisplayClass9_0.<.ctor>b__1(UInt64)
00007FCB4F7FDA30 00007FCCFFBC2B14 System.Threading.QueueUserWorkItemCallback.Execute() [/_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs @ 907]
00007FCB4F7FDA60 00007FCCFF349F20 System.Threading.ThreadPoolWorkQueue.Dispatch()
00007FCB4F7FDAD0 00007FCCF74A3B35 System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart() [/_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.cs @ 63]
00007FCB4F7FDCF0 00007fcd71140747 [DebuggerU2MCatchHandlerFrame: 00007fcb4f7fdcf0] 

### Error details

_No response_

### Environment data

```powershell
PS /> $PSVersionTable

Name                           Value
----                           -----
PSVersion                      7.2.1
PSEdition                      Core
GitCommitId                    7.2.1
OS                             Linux 5.10.76-linuxkit #1 SMP Mon Nov 8 10:21:19 UTC 2021
Platform                       Unix
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

### Visuals

_No response_

[et1975]

Found a workaround: $ProgressPreference = "SilentlyContinue"

[PaulHigin]

@WG-Remoting
@et1975 Thanks for the report. It looks like the out-of-range exception is being generated here.

PowerShell/src/System.Management.Automation/engine/remoting/server/ServerPowerShellDriver.cs

Line 552 in 7cc9c87

if ((indexIntoDataSent == 0) && (!_datasent[indexIntoDataSent]))

This is very old code with what looks like a hack to adjust the data collection index. Needs further investigation.

[et1975]

The code still looks exactly as @PaulHigin linked.

[DarkLite1]

We have the same issue on Windows Server 2022 with PowerShell 7.4.0:

The background process reported an error with the following message: Unhandled exception. System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection. (Parameter 'index').

We'll try the suggested workaround:
$ProgressPreference = "SilentlyContinue"

Thanks for having another look a this, as it is really annoying when this happens.