feat: adaptive playwright crawler

[janbuchar]

This uses the newly added restricted crawling contexts to execute request handlers. This allows us to compare browser and http-only request handler runs for a request and switch to http-only crawling on sites that we predict to be static. More information will be added here.

The intended usage is as follows:

import { AdaptivePlaywrightCrawler } from 'crawlee';

const startUrls = [{url: 'https://warehouse-theme-metal.myshopify.com/collections', label: 'START'}];

const crawler = new AdaptivePlaywrightCrawler({
    requestHandler: async ({ request, enqueueLinks, pushData, querySelector }) => {
        console.log(`Processing: ${request.url} (${request.label})`);

        if (request.label === 'DETAIL') {
            const urlPart = request.url.split('/').slice(-1); // ['sennheiser-mke-440-professional-stereo-shotgun-microphone-mke-440']
            const manufacturer = urlPart[0].split('-')[0]; // 'sennheiser'

            const title = (await querySelector('.product-meta h1')).text();
            const sku = (await querySelector('span.product-meta__sku-number')).text();

            const $prices = await querySelector('span.price')
            const currentPriceString = $prices.filter(':contains("$")').first().text()

            const rawPrice = currentPriceString.split('$')[1];
            const price = Number(rawPrice.replaceAll(',', ''));

            const inStockElements = await querySelector('span.product-form__inventory')
            const inStock = inStockElements.filter(':contains("In stock")').length > 0;

            const results = {
                url: request.url,
                manufacturer,
                title,
                sku,
                currentPrice: price,
                availableInStock: inStock,
            };

            await pushData(results);
        } else if (request.label === 'CATEGORY') {
            await enqueueLinks({
                selector: '.product-item > a',
                label: 'DETAIL', // <= note the different label
            });

            await enqueueLinks({
                selector: 'a.pagination__next',
                label: 'CATEGORY', // <= note the same label
            });
        } else if (request.label === 'START') {
            await enqueueLinks({
                selector: '.collection-block-item',
                label: 'CATEGORY',
            });
        }
    },
    renderingTypeDetectionRatio: 0.1,
    maxRequestsPerCrawl: 100,
    maxRequestRetries: 0,
    minConcurrency: 1,
    maxConcurrency: 1,
    headless: true,
});

await crawler.run(startUrls);

When handling a request from the queue, the crawler

1. tries to predict the rendering type (static/client only) based on URL, label and potentially other criteria (using a logistic regression model that gets updated on the fly)
2. for static pages, a HTTP-only scrape is done and the request handler works with Cheerio-based portadom
3. for client only pages, a playwright scrape is done and the request handler receives a portadom instance that uses Playwright locators (hence it waits for content to appear implicitly)
4. for a configurable percentage of requests, a detection is done (also if we're not confident about the prediction) - both HTTP-only and playwright scrapes are done and the results are compared. If (and only if) the HTTP-only scrape behaves the same, we conclude the page is static and update our logistic regression model.

[github-actions]

⚠️ Pull Request Tookit has failed!

Pull request is neither linked to an issue or epic nor labeled as adhoc!

[github-actions]

⚠️ Pull Request Tookit has failed!

Pull request is neither linked to an issue or epic nor labeled as adhoc!

[github-actions]

⚠️ Pull Request Tookit has failed!

Pull request is neither linked to an issue or epic nor labeled as adhoc!

[github-actions]

⚠️ Pull Request Tookit has failed!

Pull request is neither linked to an issue or epic nor labeled as adhoc!

[github-actions]

⚠️ Pull Request Tookit has failed!

Pull request is neither linked to an issue or epic nor labeled as adhoc!

[github-actions]

⚠️ Pull Request Tookit has failed!

Pull request is neither linked to an issue or epic nor labeled as adhoc!

[github-actions]

⚠️ Pull Request Tookit has failed!

Pull request is neither linked to an issue or epic nor labeled as adhoc!

[barjin]

Damn, I haven't checked this in a while... good job!

Given we mark this as experimental, I have no problem merging + releasing it and maybe having the marketing team make some promo for this so we get some real-life feedback.

I'm just still a bit unsure about the portadom - IMO it's good for the experiment right now, but 6 month stale package with no other dependents than us... just scares me a bit 😄

[B4nan]

I'm just still a bit unsure about the portadom - IMO it's good for the experiment right now, but 6 month stale package with no other dependents than us... just scares me a bit 😄

Same

[janbuchar]

@barjin @B4nan agreed, do you think that the querySelector function added in the last couple commits is good enough? Would you do something else? Extend it somehow?

[barjin]

(I'm sure there was some reason, but it was too long ago, sorry) - why don't we use parseWithCheerio? imo the cheerio / jQuery syntax is better known among devs + we have this in all our crawlers already... We're not doing anything with the calls / returned values from this function right now, right?

[janbuchar]

(I'm sure there was some reason, but it was too long ago, sorry) - why don't we use parseWithCheerio? imo the cheerio / jQuery syntax is better known among devs + we have this in all our crawlers already... We're not doing anything with the calls / returned values from this function right now, right?

querySelector actually returns a Cheerio object. The difference to parseWithCheerio is that it also waits for the selector to appear, which is necessary if we want to use the same request handler for browser-based and http-only scraping... which is the entire point of this.

[barjin]

A-ha, I see now... clever! It's a bit shame that there is a new API now (I thought we could switch between crawlers just by changing the constructor name), but given the circumstances, this is imo a good compromise.

[B4nan]

i got just few nits and documentation notes, plus:

• lets remove portadom now if it's not tightly coupled to the implementation anyhow and we have a simple but good enough alternative
• we might want to move the querySelector to all the crawlers just like we have the parseWithCheerio (can happen later once we see this is the right direction)
• as you mentioned yourself, the jsdoc for the adaptive crawler needs some work before we merge
• ran the e2e tests and they are passing, would be nice to have one for the adaptive crawling too

[B4nan]

btw can you fix the lock file so the tests can run here?

[B4nan]

so did you modify the lock file by hand or what was it about? :D

[janbuchar]

so did you modify the lock file by hand or what was it about? :D

rebased on master, dropped the original yarn-updating commit, ran yarn 🤷