feat(core): make inner option of ThrottlingRequestManager optional - #4037
feat(core): make inner option of ThrottlingRequestManager optional#4037harryautomazione wants to merge 7 commits into
Conversation
7b64e7b to
0b30bda
Compare
|
heads up: the |
|
Thanks for the rebase @B4nan, I've synced the branch with master, updated the public API reports and resolved a relative import issue. All CI checks are green now. |
…ingRequestManager
…nd format schema rules
6c4f5c7 to
2754745
Compare
|
Thanks @harryautomazione — and sorry, I've folded this into #4061 rather than asking you to rebase it. That branch replaced Your |
…anager to fix leaky abstraction of ThrottlingRequestManager (#4061) - closes #3999 - closes #4027 - supersedes #4037 - supersedes #4038 - contributes to #4039 Worth reviewing in order — each change stands alone, with the reasoning in its own message. 1. `checkReadiness()` replaces `isEmpty()` / `isFinished()` on `IRequestLoader` and `IRequestManager`. 2. `IRequestManager.recordPacingSignal()` replaces the `SupportsDomainThrottling` capability probe. 3. The crawler builds the `sameDomainDelaySecs` throttler in its constructor, which is what puts it *inside* the tandem. 4. `sameDomainDelaySecs` goes in through `recordPacingSignal()` as a floor covering every domain, so a manager that already paces takes it instead of getting a second pacer wrapped around it. 5. `ThrottlingRequestManager`'s `inner` is optional — omitted, it opens the default queue on first use. Same feature as #4037, which is a much smaller patch on top of the lazy `inner` factory this branch already has. On #3999 this takes the first option — wrappers forward — but the objection there was that it makes `RequestManagerTandem` know throttling exists. It doesn't: pacing signals are `IRequestManager` members, so the tandem forwards a `PacingSignal` it never looks inside, and learns nothing throttling-specific. That deletes the capability-discovery problem rather than solving it, which is why the delegation interface from #4038 isn't here. One capability check did survive that — the guard refusing `sameDomainDelaySecs` over a `ThrottlingRequestManager` — and it missed a throttler behind a tandem (#discussion_r3851833671). The fourth change removes it rather than deepening it: the floor is a `PacingSignal` like any other, so whatever paces takes it wherever it sits in a composition, and a manager pacing only *some* of its domains throws rather than under-applying it. No `instanceof` left anywhere. #4039 is the shape `sameDomainDelaySecs` was producing on its own: wrapping happened at first use, by which point the tandem for a `requestList` already existed, so the pacer went outside it and a list's requests reached the queue without passing a per-domain clock. The third commit builds the pacer in the constructor instead, so those requests are routed by domain like any other. The hand-built form in #4039's reproduction is untouched — `fetchNextRequest` still takes whatever `inner` offers without checking the clocks — so #4040 is still needed. Migration notes in `docs/upgrading/upgrading_v4.md`.
|
Thanks for the detailed explanation and for incorporating the opener signature changes into #4061, @janbuchar! Makes complete sense with the new |
Summary
Closes #4027
This PR makes the
inneroption ofThrottlingRequestManagerOptionsoptional (inner?: T). When omitted,ThrottlingRequestManagerautomatically opens and defaults to a standardRequestQueueviarequestManagerOpenerduring#ensureSubManagers().Changes
ThrottlingRequestManagerOptions& Schema (packages/core/src/storages/throttling_request_manager.ts):inner?: Toptional inThrottlingRequestManagerOptions.throttlingRequestManagerOptionsSchemawithinner: schemas.anyObject.optional().ThrottlingRequestManager:#innerin#ensureSubManagers()viaawait this.#requestManagerOpener(null, { configuration: this.config })ifoptions.innerwas omitted.isEmpty,purge,#forEachManager,#sumOverManagers,#everyManager).test/core/storages/throttling_request_manager.test.ts):ThrottlingRequestManagercan be instantiated withoutinnerand correctly routes and processes requests.Checklist