How do I package Cheerio crawler deps for deployment on AWS lambda? Seeing require.resolve issues.

[jo-sip]

Hi.

I'm currently trying to upload my crawler on lambda, using layers as indicated in the docs. I'm currently manually zipping jquery and jsdom in nodejs/node_modules and including that in my layer trying to resolve the following issues.

I'm seeing these issues when esbuild runs for bundling the lambda code:

▲ [WARNING] "./xhr-sync-worker.js" should be marked as external for use with "require.resolve" [require-resolve-not-external]

    node_modules/jsdom/lib/jsdom/living/xhr/XMLHttpRequest-impl.js:31:57:
      31 │ ...ve ? require.resolve("./xhr-sync-worker.js") : null;
         ╵                         ~~~~~~~~~~~~~~~~~~~~~~

▲ [WARNING] "jquery" should be marked as external for use with "require.resolve" [require-resolve-not-external]

    node_modules/@crawlee/puppeteer/internals/utils/puppeteer_utils.js:37:35:
      37 │ const jqueryPath = require.resolve('jquery');
         ╵                                    ~~~~~~~~

▲ [WARNING] "puppeteer/package.json" should be marked as external for use with "require.resolve" [require-resolve-not-external]

    node_modules/@crawlee/puppeteer/internals/utils/puppeteer_utils.js:190:37:
      190 │ ...onPath = require.resolve('puppeteer/package.json');

This uploads fine, but when I run the handler I get "Error: Cannot find module './xhr-sync-worker.js'. I resolved the jquery import with the layer, but jsdom doesn't resolve this issue, probably because it's a relative path. Basically this issue: https://stackoverflow.com/questions/44354700/meteor-app-not-finding-xhr-sync-worker-js

Some questions:

1. Does the cheerio crawler use these deps? Here I see puppeteer is referenced. Is esbuild not properly removing unused deps perhaps? I shouldn't really be seeing these issues to begin with.
2. Is there a good way to remove the deps from the crawler if they're not used.

Thanks.

EDIT: I changed the deps from importing from "crawlee" to "@crawlee/cheerio" for most things. This seems to have resolved most issues but now I'm encountering this issue:

{"errorType":"TypeError","errorMessage":"Cannot read properties of undefined (reading 'bind')","trace":["TypeError: Cannot read properties of undefined (reading 'bind')","    at StorageManager3._getStorageClientFactories (/var/task/index.js:65941:51)","    at StorageManager3._getOrCreateStorage (/var/task/index.js:65929:77)","    at StorageManager3.openStorage (/var/task/index.js:65906:44)","    at async RequestQueue3.open (/var/task/index.js:86085:23)","    at async CheerioCrawler2.getRequestQueue (/var/task/index.js:87069:51)","    at async CheerioCrawler2.addRequests (/var/task/index.js:87086:30)","    at async CheerioCrawler2.run (/var/task/index.js:87009:11)","    at async Runtime.handler (/var/task/index.js:106347:3)"]}

handler code: (virtually identical to https://crawlee.dev/docs/deployment/aws-cheerio)

 Configuration.set("persistStorage", false);

  const crawler = new CheerioCrawler(
    {
      // requestHandler: router,
      maxRequestsPerCrawl: 1,
      // minConcurrency: 5,
      // maxRequestsPerMinute: 60,
    },
    new Configuration({
      persistStorage: false,
    }),
  );

  await crawler.run(["..."]);

[jo-sip]

This has to do with the bundling process it seems. The offending line is createStorageClient: client[clientName].bind(client),, where client name can change from variable names changing in the bundling process. Changing esbuild to use options --keep-names seems also somewhat resolve this, but now there's an issue with TypeError: gotScraping2 is not a function.