Share the host-limited concurrent map

[ddeboer]

Problem Statement

The workspace has one good answer to “run many outbound tasks concurrently without
overwhelming any single host”, and exactly one package can use it.

mapHostLimited in @lde/distribution-probe applies two caps – a global one and a
per-host one – and, crucially, does so without head-of-line blocking: when a host is
saturated, its next item yields to a later item on a different host rather than stalling
the queue. Results still come back in input order. That combination is the whole value,
and it is the part that is easy to get wrong when re-implemented.

It is private to probe.ts. The public wrapper exposes probing, not scheduling. So the
next package that needs bounded outbound work either re-implements the scheduler,
reaches for a dependency, or – most likely – does neither and hammers one host, which is
a rude thing to do to the publishers whose endpoints this workspace exists to read.

There is no general-purpose concurrency helper anywhere in the workspace: no p-limit,
no p-queue, no bottleneck. Only p-retry, in two packages.

Solution

Promote the host-limited concurrent map to a shared module, unchanged in behaviour, and
have its current owner consume it from there.

User Stories

1. As a package author, I want a shared host-limited concurrent map, so that outbound
   work is bounded globally and per host without me re-implementing the scheduler.
2. As a package author, I want a saturated host not to stall work for other hosts, so
   that one slow origin does not set the pace for an entire run.
3. As a package author, I want results in input order, so that callers can pair results
   with their inputs positionally.
4. As an operator, I want every package doing outbound work to be a polite neighbour to
   the hosts it reads, so that a workspace-wide run does not look like an attack.

Implementation Decisions

Behaviour is preserved exactly. This is a move, not a redesign. The two caps, the
no-head-of-line-blocking property, input-order results, and the requirement that the
task never rejects all stay as they are. Any change to the algorithm belongs in its own
issue, argued on its own merits.

Placement. @lde/distribution-probe currently depends only on @lde/dataset,
rdf-parse and tslib, and it is itself depended on by @lde/pipeline,
@lde/distribution-monitor and @lde/distribution-health – so it sits low in the graph
and cannot take a dependency on a higher-level package. Two candidates:

• A small dedicated package. Depends on nothing, importable from anywhere including
  the lowest layers. Costs one more package to publish and version.
• An existing low-level package. Avoids a new publish, but every consumer then takes
  a dependency on something whose name promises more than a scheduler.

Recommendation: the dedicated package, because the whole point is that anything may use
it, and a scheduler that arrives attached to unrelated code will be avoided by exactly
the callers it is meant to serve.

Host key. Keep the current rule – the URL host, falling back to the full href when
there is no host – so non-HTTP schemes (urn:, file:) get their own budget rather
than sharing one bucket.

Naming. The exported name should say what it does without naming a domain: it maps
items to results under a global and a per-host cap. distribution must not appear in
it.

Testing Decisions

A good test here asserts the observable scheduling contract – how many tasks are in
flight, for which hosts, and in what order results come back – not the internals of the
queue. Concurrency is asserted by observing peak in-flight counts, never by sleeping and
hoping.

• Global and per-host caps both hold under a mix of hosts.
• A saturated host does not block items for other hosts: with one slow host and several
  fast ones, the fast hosts’ items complete without waiting for it.
• Results come back in input order regardless of completion order.
• Items whose key has no host (urn:, file:) get their own budget rather than sharing
  a single bucket.
• An empty input, a single item, and more items than either cap.

Prior art: the tests that exist today for it in its current home, which move with it.

Out of Scope

• Any change to the scheduling algorithm, the caps’ defaults, or the retry and backoff
  logic that sits beside it in its current home. Retry stays where it is.
• Adopting a third-party concurrency library in its place. That is a separate argument.
• New consumers. This issue makes the module available; wiring it into anything else is
  that consumer’s work.

Further Notes

Split out of #675, where it was bundled with an enrichment seam that has since been
rescoped. It was always separable, it is independently valuable, and it can land first.