refactor!: Split off ConcurrencySystem from AutoscaledPool #3917

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

janbuchar merged 55 commits into v4 from injectable-autoscaled-pool

Jul 30, 2026

docs/guides/scaling_crawlers.mdx

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -10,7 +10,7 @@ import CodeBlock from '@theme/CodeBlock';
  
    import MinMaxConcurrencySource from '!!raw-loader!./scaling_crawlers_minMaxConcurrency.ts';

    import MaxRequestsPerMinuteSource from '!!raw-loader!./scaling_crawlers_maxRequestsPerMinute.ts';

    import AutoscaledPoolOptionsSource from '!!raw-loader!./scaling_crawlers_autoscaledPoolOptions.ts';

    import ConcurrencySystemSource from '!!raw-loader!./scaling_crawlers_concurrencySystem.ts';

    As we build our crawler, we might want to control how many requests we do to the website at a time. Crawlee provides several options to fine tune how many parallel requests should be made at any time, how many requests should be done per minute, and how should scaling work based on the available system resources.

    @@ -45,55 +45,53 @@ It's recommended to leave it at the default value that is provided and letting t
  
    ## Advanced options

    While the options above should be enough for most users, if we wanted to get super deep into the configuration of the autoscaling pool (the internal utility in Crawlee that helps us allow crawlers to scale up and down), we can do so through the <ApiLink to="cheerio-crawler/interface/CheerioCrawlerOptions#autoscaledPoolOptions">`autoscaledPoolOptions`</ApiLink> object available on crawler options.

    While the options above should be enough for most users, if we wanted to get super deep into the configuration of autoscaling (the internal machinery in Crawlee that helps us allow crawlers to scale up and down), we can do so by injecting a pre-configured <ApiLink to="core/class/ConcurrencySystem">`ConcurrencySystem`</ApiLink> through the <ApiLink to="cheerio-crawler/interface/CheerioCrawlerOptions#concurrencySystem">`concurrencySystem`</ApiLink> crawler option. All the fine-grained scaling configuration lives on that instance (see <ApiLink to="core/interface/ConcurrencySystemOptions">`ConcurrencySystemOptions`</ApiLink>).

    :::danger Complex options up ahead!

    This section is super advanced and, unless you test the changes extensively and know what you're doing, it's better to leave these options to their defaults, as they are most likely going to work fine without much fuss.

    :::

    With that warning aside, if we're feeling adventurous, this is how we would pass these options when using a crawler:

    With that warning aside, this is how we pass those options. One thing to watch: the `minConcurrency`/`maxConcurrency`/`maxRequestsPerMinute` shortcuts cannot be combined with an injected system — they configure the default one it replaces, so set those limits on the instance instead.

    <CodeBlock language="js">

    	{AutoscaledPoolOptionsSource}

    	{ConcurrencySystemSource}

    </CodeBlock>

    ### `desiredConcurrency`

    This option specifies the amount of requests that should be running in parallel at the start of the crawler, assuming there are so many available. It defaults to the same value as `minConcurrency`.

    :::tip Capping the combined concurrency of several crawlers

    ### `desiredConcurrencyRatio`

    Injecting the *same* `ConcurrencySystem` instance into several crawlers makes them share a single concurrency budget, capping their combined parallelism instead of letting each crawler scale independently.

    The minimum ratio of concurrency to reach before more scaling up is allowed (a number between `0` and `1`). By default, it is set to `0.95`.

    The budget is shared, not divided: slots go to whoever asks first, so a crawler that fills it early keeps refilling it as its own requests finish, and one that starts later may get a much smaller share of it. If a crawler needs a guaranteed slice, give it its own instance — or implement <ApiLink to="core/interface/IConcurrencySystem">`IConcurrencySystem`</ApiLink>, whose allocation methods are told which crawler is asking, and allocate however we see fit.

    We can think of this as the point where the autoscaling pool can attempt to scale up (or down), monitor if there's any changes, and correct them if necessary.

    :::

    ### `scaleUpStepRatio` and `scaleDownStepRatio`

    ### `desiredConcurrency`

    These values define the fractional amount of desired concurrency to be added or subtracted as the autoscaling pool scales up or down. Both of these values default to `0.05`.

    This option specifies the amount of requests that should be running in parallel at the start of the crawler, assuming there are so many available. It defaults to the same value as `minConcurrency`.

    Every time the autoscaled pool attempts to scale up or down, this value will be added or subtracted from the current concurrency, and, based on the [`desiredConcurrencyRatio`](#desiredconcurrencyratio) and [`maxConcurrency`](#minconcurrency-and-maxconcurrency), determines how many requests can run concurrently.

    ### `desiredConcurrencyRatio`

    ### `maybeRunIntervalSecs`

    The minimum ratio of concurrency to reach before more scaling up is allowed (a number between `0` and `1`). By default, it is set to `0.9`.

    Indicates how often the autoscaling pool should check if more requests can be started and, if that's true, starts a new request if there are any available. This value is represented in seconds, and defaults to `0.5`.

    We can think of this as the point where the concurrency system can attempt to scale up (or down), monitor if there's any changes, and correct them if necessary.

    :::info

    ### `scaleUpStepRatio` and `scaleDownStepRatio`

    Changing this has no effect for requests that are fired immediately after the previous ones are finished. However, it will influence how fast new requests will be started after the autoscaled pool scales up.

    These values define the fractional amount of desired concurrency to be added or subtracted as the concurrency system scales up or down. Both of these values default to `0.05`.

    :::

    Every time the concurrency system attempts to scale up or down, this value will be added or subtracted from the current concurrency, and, based on the [`desiredConcurrencyRatio`](#desiredconcurrencyratio) and [`maxConcurrency`](#minconcurrency-and-maxconcurrency), determines how many requests can run concurrently.

    ### `loggingIntervalSecs`

    This option lets us control how often the autoscaled pool should log its current state (the current concurrency ratio, desired ratios, if the system is overloaded and so on).

    This option lets us control how often the concurrency system should log its current state (the current concurrency ratio, desired ratios, if the system is overloaded and so on).

    We can disable logging altogether by setting this to `null`. By default, it is set to `60` seconds.

    ### `autoscaleIntervalSecs`

    This option lets us control how often the autoscaling pool should check if it can and should scale up or down. This value is represented in seconds, and defaults to `10`.

    This option lets us control how often the concurrency system should check if it can and should scale up or down. This value is represented in seconds, and defaults to `10`.

    :::tip

    @@ -118,3 +116,52 @@ This controls how many total requests can be made per minute. It counts the amou
  
    This option can be set by specifying [`maxRequestsPerMinute`](#maxrequestsperminute) in your crawler options too, as it is a shortcut for visibility and ease of access.

    :::

    ## Load signals

    Whether the machine counts as overloaded is decided by *load signals*. Four are built in — memory, event loop, CPU and the storage client's rate-limit errors — and each is configured by its own bag under <ApiLink to="core/interface/ConcurrencySystemOptions#loadSignals">`loadSignals`</ApiLink>, carrying both its limits and the `overloadedRatio` at which it fires. If any signal reports overload, the system is overloaded and concurrency is held down.

    A signal we don't want watched at all can be switched **off** with `false`, which stops it being collected as well as evaluated (its entry in the reported status then simply reads as not overloaded):

    ```ts

    const concurrencySystem = new ConcurrencySystem({

        loadSignals: {

            // Our storage backend reports no rate-limit statistics, so stop polling it every second.

            client: false,

            eventLoop: { maxBlockedMillis: 100 },

        },

    });

    ```

    We can also watch resources of our own by implementing <ApiLink to="core/interface/LoadSignal">`LoadSignal`</ApiLink> — navigation timeouts or proxy health, say — and passing them in `loadSignals.custom`. The <ApiLink to="core/class/SnapshotStore">`SnapshotStore`</ApiLink> helper does the time-windowed bookkeeping for us:

    ```ts

    const store = new SnapshotStore();

    const proxyHealth: LoadSignal = {

        name: 'proxyHealth',

        overloadedRatio: 0.3,

        async start({ maxSampleWindowMillis }) {

            // Retain exactly the window we will be sampled over, and drop anything measured before a restart.

            store.useSampleWindow(maxSampleWindowMillis);

            store.clear();

            // ...start measuring, calling store.push() per measurement

        },

        async stop() { /* ...stop measuring */ },

        getSample: (sampleDurationMillis) => store.getSample(sampleDurationMillis),

    };

    const concurrencySystem = new ConcurrencySystem({ loadSignals: { custom: [proxyHealth] } });

    ```

    Each built-in is a public class too — `MemoryLoadSignal`, `EventLoopLoadSignal`, `CpuLoadSignal`, `ClientLoadSignal` — taking exactly the bag its `loadSignals` key accepts. Constructing one directly is how we *wrap* a built-in rather than reimplement it (to hold its overload verdict for a while after the resource recovers, say): switch the original off with `cpu: false`, and pass a signal that delegates to the instance we built in `custom`.

    :::info Signal names are the keys of the reported status

    A signal's `name` is the key its verdict appears under in the reported status, so two signals cannot share one — a duplicate throws at construction time. That is why taking over a built-in name (`memInfo`, `eventLoopInfo`, `cpuInfo`, `clientInfo`) requires switching that built-in off.

    :::

    ### `snapshotHistorySecs` and `currentHistorySecs`

    Signals are not read as a single instantaneous measurement but averaged over a window, and there are two: `currentHistorySecs` (default `5`) is the short window that gates whether one more request may start, while `snapshotHistorySecs` (default `30`) is the longer window autoscaling decisions are based on. Dispatch therefore reacts to spikes quickly while scaling stays stable. Both apply to every signal alike, built-in or custom, and signals size their snapshot retention to the wider of the two — so raising `snapshotHistorySecs` is what costs memory.

docs/guides/scaling_crawlers_autoscaledPoolOptions.ts

This file was deleted.

docs/guides/scaling_crawlers_concurrencySystem.ts

-Original file line number
+Diff line change
@@ -0,0 +1,19 @@
+    import { CheerioCrawler, ConcurrencySystem } from 'crawlee';
+    // Advanced scaling options live on a pre-configured ConcurrencySystem
+    const concurrencySystem = new ConcurrencySystem({
+        // ...
+    });
+    const crawler = new CheerioCrawler({
+        concurrencySystem,
+        // ...
+    });
+    // An injected system's lifecycle is owned by us, not the crawler
+    await concurrencySystem.start();
+    try {
+        await crawler.run(['https://crawlee.dev']);
+    } finally {
+        await concurrencySystem.stop();
+    }

docs/public-api/crawlee-basic.api.md

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -9,7 +9,7 @@ import type { AddRequestsBatchedResult } from '@crawlee/core';
  
    import { AnyPredicate } from 'ow';

    import { ArrayPredicate } from 'ow';

    import { AutoscaledPool } from '@crawlee/core';

    import { AutoscaledPoolOptions } from '@crawlee/core';

    import type { AutoscaledPoolPredicateOptions } from '@crawlee/core';

    import type { Awaitable } from '@crawlee/types';

    import type { BaseHttpClient } from '@crawlee/types';

    import { BasePredicate } from 'ow';

    @@ -18,6 +18,8 @@ import { BooleanPredicate } from 'ow';
  
    import { Cheerio } from '@crawlee/utils';

    import { CheerioAPI } from '@crawlee/utils';

    import { CheerioRoot } from '@crawlee/utils';

    import { ConcurrencySystem } from '@crawlee/core';

    import { ConcurrencySystemOptions } from '@crawlee/core';

    import type { Configuration } from '@crawlee/core';

    import { ContextPipeline } from '@crawlee/core';

    import type { CrawleeLogger } from '@crawlee/core';

    @@ -30,6 +32,7 @@ import type { EnqueueLinksOptions } from '@crawlee/core';
  
    import type { EventManager } from '@crawlee/core';

    import type { FinalStatistics } from '@crawlee/core';

    import type { GetUserDataFromRequest } from '@crawlee/core';

    import { IConcurrencySystem } from '@crawlee/core';

    import { IProxyConfiguration } from '@crawlee/core';

    import { IRequestLoader } from '@crawlee/core';

    import { IRequestManager } from '@crawlee/core';

    @@ -78,6 +81,7 @@ export class BasicCrawler<Context extends CrawlingContext = CrawlingContext, Con
  
        get contextPipeline(): ContextPipeline<CrawlingContext, ExtendedContext>;

        // (undocumented)

        protected static readonly CRAWLEE_STATE_KEY = "CRAWLEE_STATE";

        protected createDefaultConcurrencySystem(options: ConcurrencySystemOptions): ConcurrencySystem;

        // (undocumented)

        protected readonly errorHandler?: ErrorHandler<CrawlingContext, ExtendedContext>;

        exportData<Data>(path: string, format?: 'json' | 'csv', options?: DatasetExportOptions): Promise<Data[]>;

    @@ -133,6 +137,7 @@ export class BasicCrawler<Context extends CrawlingContext = CrawlingContext, Con
  
            maxRequestsPerCrawl: NumberPredicate & BasePredicate<number | undefined>;

            maxCrawlDepth: NumberPredicate & BasePredicate<number | undefined>;

            autoscaledPoolOptions: ObjectPredicate<object> & BasePredicate<object | undefined>;

            concurrencySystem: ObjectPredicate<object> & BasePredicate<object | undefined>;

            sessionPool: ObjectPredicate<object> & BasePredicate<object | undefined>;

            proxyConfiguration: ObjectPredicate<object> & BasePredicate<object | undefined>;

            statusMessageLoggingInterval: NumberPredicate & BasePredicate<number | undefined>;

    @@ -181,8 +186,9 @@ export class BasicCrawler<Context extends CrawlingContext = CrawlingContext, Con
  
    // @public (undocumented)

    export interface BasicCrawlerOptions<Context extends CrawlingContext = CrawlingContext, ContextExtension = Dictionary<never>, ExtendedContext extends Context = Context & ContextExtension, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<Context['request']>>> {

        additionalHttpErrorStatusCodes?: number[];

        autoscaledPoolOptions?: AutoscaledPoolOptions;

        autoscaledPoolOptions?: AutoscaledPoolPredicateOptions;

        blockedStatusCodes?: number[];

        concurrencySystem?: IConcurrencySystem;

        configuration?: Configuration;

        contextPipelineBuilder?: () => ContextPipeline<CrawlingContext, Context>;

        errorHandler?: ErrorHandler<CrawlingContext, ExtendedContext>;

docs/public-api/crawlee-browser.api.md

-Original file line number
+Diff line change
@@ Expand Up @@
             maxRequestsPerCrawl: NumberPredicate & BasePredicate<number | undefined>;
             maxCrawlDepth: NumberPredicate & BasePredicate<number | undefined>;
             autoscaledPoolOptions: ObjectPredicate<object> & BasePredicate<object | undefined>;
+            concurrencySystem: ObjectPredicate<object> & BasePredicate<object | undefined>;
             sessionPool: ObjectPredicate<object> & BasePredicate<object | undefined>;
             statusMessageLoggingInterval: NumberPredicate & BasePredicate<number | undefined>;
             statusMessageCallback: Predicate<Function> & BasePredicate<Function | undefined>;
@@ Expand Down @@

Navigation Menu

Uh oh!

refactor!: Split off ConcurrencySystem from AutoscaledPool #3917

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!

Uh oh!

Uh oh!