fix(registry): HttpRegistry block_until_ready returns early when work is still pending

[arlosi]

What does this PR try to resolve?

block_until_ready is returning early when there are still pending transfers. This leads to extra restarts of the resolver, impacting performance.

In the existing implementation:

• handle_completed_downloads runs and there are no completed downloads
• multi.perform() is called and completes new downloads, with no pending transfers remaining
• Since there are no items remaining_in_multi, the function returns early

This fixes the issue by reordering the calls to multi.perform(), handle_completed_downloads, then the completion check.

How should we test and review this PR?

A test is added that uses cargo's tracing to show the number of blocking calls. First commit shows existing behavior, second commit shows fix.

Originally based on the PR #14680 by @x-hgg-x. The ordering fix in this PR also avoids an additional block_until_ready call for retried failed downloads.

[rustbot]

r? @ehuss

rustbot has assigned @ehuss.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

[Eh2406]

does the code

cargo/src/cargo/core/package.rs

Line 960 in 8040c00

self.add_sleepers()?;

also need adjustment?

[weihanglo]

does the code

cargo/src/cargo/core/package.rs

Line 960 in 8040c00

self.add_sleepers()?;

also need adjustment?

Seem pretty likely, though this is downloading the package so the perf gain may not be as significant as registry index one.

[weihanglo]

👍🏾
Good either with or without #14694 (comment).

[arlosi]

does the code

cargo/src/cargo/core/package.rs

Line 960 in 8040c00

self.add_sleepers()?;

also need adjustment?

The current implementation looks correct to me. It doesn't have the same bug with early returning when there is still work to be done.

My notes on how it's working:

loop {
- wait()
  - wait_for_curl() 
     loop {
     - add_sleepers() [self.sleeping -> self.pending (may not move all, depending on timing)]
     - multi.perform() [self.pending -> self.results]
     }
  - self.pending.remove()
  - self.sleeping.push() [for failed downloads that need retry]
}

[arlosi]

@bors r=weihanglo

[bors]

📌 Commit 8db2192 has been approved by weihanglo

It is now in the queue for this repository.

[bors]

⌛ Testing commit 8db2192 with merge 79a491a...

[bors]

☀️ Test successful - checks-actions
Approved by: weihanglo
Pushing 79a491a to master...