Fix race condition for multiple very fast downloads

[HebaruSan]

Problem

See #3906 for details; to sum up, one of the new NetAsyncDownloader tests from #3904 was failing in GitHub workflows but nowhere else.

When downloading multiple URLs, the DownloadTarget.size values were 0 for some of the files when NetAsyncDownloader.DownloadAndWait completed.

Cause

This new test found a real, pre-existing problem for us! I'm really glad I didn't give in to the temptation to add [Category("FlakyNetwork")] and move on. 🎉

There's a race condition with multiple extremely fast downloads (the test is using file:// URLs to avoid spurious failures caused by server glitches, so the files are already on disk and possibly cached in memory). At the start, the main thread loops over the requested URLs and starts a thread for each one. This sets up the downloads list with the currently executing downloads, and the queuedDownloads list with the downloads that are waiting their turn. When a thread completes, the sizes of the downloads and queuedDownloads lists are compared to completed_downloads to determine whether there are any other downloads in progress or waiting.

If at any time the active download threads manage to finish their work before all of the downloads are started, then they will be comparing the completed download count to incomplete downloads and queuedDownloads lists! This results in the completion signal being sent early, and when the main thread returns to DownloadAndWait, it will skip waiting and signal completion with whatever was finished up to that point.

Changes

• Now the downloader's main thread holds the dlMutex lock (which the other threads acquire before they check for completion) while it starts all the downloads. This ensures that by the time any downloader thread is able to check for completion, the downloads and queuedDownloads lists will be fully populated, which will prevent the completion notification from being triggered prematurely.
• The test now uses 13 files instead of 4, which made the problem start happening in my Ubuntu VM where it hadn't before (there seems to be a sensitivity to the number of downloads, with more downloads making the problem more likely).
• Now the test performs two additional checks: comparing the real downloaded file sizes that it observes on disk to the target sizes and the sizes of the original files.
• Since that test was so helpful, I decided to try adding one for failed downloads. In the process, I discovered that the index values in the failed downloads exception can be incorrect, so some further changes are made to clean that up (specifically, never queueing file:// URLs and looking up the failed targets in the original input, which is now an IList instead of an ICollection because we need to call IndexOf on it).

Fixes #3906.

[techman83]

Oh awesome, this likely explains the errant failure reported here and there. Thanks @HebaruSan !