docs: Document LoadSignal customization for ConcurrencySystem - #3976
Conversation
janbuchar
commented
Aug 6, 2026
janbuchar
commented
- closes Document extending ConcurrencySystem with custom load signals #3567
janbuchar
added
the
t-tooling
barjin
left a comment
barjin
left a comment
There was a problem hiding this comment.
I'd love to see the proxy example shrink a little, but the other examples seem alright to me. Thanks!
| class ProxyHealthSignal implements LoadSignal { | ||
| readonly name = 'proxyHealth'; | ||
| readonly overloadedRatio = 0.3; | ||
|
|
||
| private readonly store = new SnapshotStore(); | ||
| private interval?: NodeJS.Timeout; | ||
|
|
||
| async start({ maxSampleWindowMillis }: LoadSignalStartContext): Promise<void> { | ||
| // Retain exactly the window we will be sampled over - a store never given one keeps everything. | ||
| this.store.useSampleWindow(maxSampleWindowMillis); | ||
| // This may be a restart, so drop anything measured before the downtime. | ||
| this.store.clear(); | ||
|
|
||
| this.interval = setInterval(async () => { | ||
| const createdAt = new Date(); | ||
| this.store.push({ createdAt, isOverloaded: await areProxiesStruggling() }, createdAt); | ||
| }, 1_000); | ||
| } | ||
|
|
||
| async stop(): Promise<void> { | ||
| clearInterval(this.interval); | ||
| } | ||
|
|
||
| getSample(sampleDurationMillis?: number): LoadSnapshot[] { | ||
| return this.store.getSample(sampleDurationMillis); | ||
| } | ||
| } |
There was a problem hiding this comment.
Is there any chance we could make this a bit more concise with SnapshotStore.fromInterval (source)?
The actual user-specific part is the (very clear and understandable) areProxiesStruggling(): Promise<boolean> indicator - we should, imo, advertise the simplest way to plug a function like this into Crawlee.
There was a problem hiding this comment.
nice! But aren't we still reimplementing parts of SnapshotStore.fromInterval here (namely, starting, owning and stopping the Interval)?
There was a problem hiding this comment.
What is this SnapshotStore.fromInterval you keep mentioning? https://github.com/apify/crawlee/pull/3917/changes#diff-4f9c73bec18c2b7246ab59bf2319956e2b1c59afad4e510efb004ebe0217d8cf
There was a problem hiding this comment.
Ah, I must have been checking master 💡
imho it's a shame the users cannot plug their fn(): boolean | Promise<boolean> function into the LoadSignal directly (or via a simple shim... like SnapshotStore.fromInterval, for example 😄 )
But we can always add that later on, no need to deal with this now 👍
There was a problem hiding this comment.
Yeah, it felt kinda gratuitious to me so I removed it in the name of minimizing the public API.
| // Our storage backend reports no rate-limit statistics, so stop polling it every second. | ||
| client: false, |
There was a problem hiding this comment.
Another spot for client -> (storage)Backend rename?
3 participants
