Unable to download non-HTML content if served with HTTP cookies

[simonwiles]

I'm trying to crawl a site with many PDFs that are served with HTTP cookies. Since the introduction of

browsertrix-crawler/crawler.js

Line 758 in 5e5efda

return resp.status === 200 && !resp.headers.get("set-cookie");

in ab096cd, this is not possible.

Crawler.directFetchCapture() returns false, despite the fetching and recording having completed correctly and the .warc.gz having been written; this causes Crawler.loadPage() to attempt a browser-based capture, and puppeteer's page.goto() times out at

browsertrix-crawler/crawler.js

Lines 310 to 312 in 09082e8

	if (this.params.behaviorOpts) {
	await Promise.allSettled(page.frames().map(frame => evaluateWithCLI(frame, "self.__bx_behaviors.run();")));
	}

I was able to work around it by adding an additional check like this:

if (await this.isHTML(data.url) && this.params.behaviorOpts) {
  await Promise.allSettled(page.frames().map(frame => evaluateWithCLI(frame, "self.__bx_behaviors.run();")));
}

and also passing --wait-until domcontentloaded, but this is obviously not ideal.

What's the logic behind checking for http cookies and forcing the use of the browser for non-HTML resources? I've taken that restriction out in a local version to fetch the ~20k PDFs that are my immediate concern, but I don't understand everything that's going on here.

Thanks!

[ikreymer]

Thanks for taking a look at this! The direct fetch logic was altered initially to deal with redirects, which were not being fetched. The cookie check was added 'just in case', as the direct fetch did not use the cookies from the browser - which is possible to do as well, in case it resulted in a 'soft 404' or not valid capture. Erring on the side of caution here, it may have introduced the unnecessary timeouts.

In simple testing, I was getting the PDF-pages to load without timeouts, Would like to take a look at the example site with PDFs to see what can be done.

[simonwiles]

Can confirm that non-html-page-load branch fixes the problem in the examples I have available. Thanks!

[ikreymer]

Was fixed in earlier release!