What hook can I use to catch net::ERR_EMPTY_RESPONSE errors?

[dyihoon90]

Hi there,

I am using Apify, but the question applies to Crawlee as well.

Sometimes when we crawl using Puppeteer, we get the following errors from the server. The server does not return any HTTP response in these cases.
ERR_CONNECTION_CLOSED
ERR_CONNECTION_RESET
ERR_CONNECTION_REFUSED
ERR_CONNECTION_ABORTED
ERR_CONNECTION_FAILED
ERR_EMPTY_RESPONSE
ERR_ABORTED

exhaustive list of errors are from here: https://source.chromium.org/chromium/chromium/src/+/main:net/base/net_error_list.h

When we get such errors, we would like a hook to pause the crawler for a while and resume after a certain time.

I have tried adding preNavigationHooks like so but to no avail.

function delayOnErrorHook(sleepMs: number) {
  const hook: PuppeteerHook = async ({ page, crawler, request }) => {
    page.on('error', async (error) => {
      logger.warn('page error encountered', error, { url: request.url });
      await sleepCrawler(sleepMs, crawler);
    });
    page.on('pageerror', async (pageerror) => {
      logger.warn('page pageerror encountered', pageerror, { url: request.url });
      await sleepCrawler(sleepMs, crawler);
    });
    page.on('requestfailed', async (failedRequest) => {
      logger.warn('page requestfailed encountered', undefined, {
        url: request.url,
        errorText: failedRequest.failure()
      });
      await sleepCrawler(sleepMs, crawler);
    });
  };
  return hook;
}

Has anyone ran into similar issues on Apify/Crawlee? Does the errorHandler in crawlee work for these errors? Thanks in advance.

*edited to add ERR_ABORTED

[LeMoussel]

Do you test with failedRequestHandler?

[dyihoon90]

I was able to do it with this preNavigationHook:

/**
 * Hook to add handlers for page failures
 * @param sleepMs
 * @returns
 */
function pageFailureHandlerHook(sleepMs: number, queuedAndDisallowedUrls: Set<string>, options: ICrawlOptions) {
  // eslint-disable-next-line @typescript-eslint/require-await
  const hook: PuppeteerHook = async ({ page, crawler, request }) => {
    const { url } = request;
    page.on('requestfailed', async (failedRequest) => {
      const errorText = failedRequest.failure()?.errorText;
      if (
        errorText === REQUEST_ERRORS.ERR_CONNECTION_CLOSED ||
        errorText === REQUEST_ERRORS.ERR_CONNECTION_RESET ||
        errorText === REQUEST_ERRORS.ERR_CONNECTION_REFUSED ||
        errorText === REQUEST_ERRORS.ERR_CONNECTION_ABORTED ||
        errorText === REQUEST_ERRORS.ERR_CONNECTION_FAILED ||
        errorText === REQUEST_ERRORS.ERR_EMPTY_RESPONSE
      ) {
        logger.warn('page requestfailed connection error encountered.', undefined, { url, errorText });
        await sleepCrawler(sleepMs, crawler);
      }
      // ERR_ABORTED from Chrome could indicate a link to a file, can check with a axios request
      if (errorText === REQUEST_ERRORS.ERR_ABORTED) {
        logger.warn('page requestfailed aborted error encountered.', undefined, { url, errorText });
        try {
          // do sth
        } catch (e) {
          if (e instanceof Error && e.name === 'AxiosError') {
            logger.warn('axios request also failed in page requestfailed', e);
          }
        }
      }
    });
  };
  return hook;
}