Website-scraper on Windows 11 throws error while scraping an academic archive site.

[rrgerber]

Configuration

version: [result of npm ls website-scraper --depth 0 command]

`-- website-scraper@5.2.0

options: [provide your full options object]

var datadir = './scraped';
var url = 'https://www.isca-speech.org/archive';
  var options = {
    urls: [url],
    directory: datadir,
    defaultFilename: "index.html",
    ignoreErrors: false,
    recursive: true,
    maxRecursiveDepth: 2,
  };

scrape(options, function (err, t)
  {
  });

Description

In trying to scrape the site https://www.isca-speech.org/archive, an error in the dependency "request" is invoked pretty early on, and the program dies.

Expected behavior: [What you expect to happen]

I expect at least the top-level html files saved to my filesystem.

Actual behavior: [What actually happens]

Here is what actually happens:

$ node --max-old-space-size=24000 main.js

/home/rich/node_modules/request/request.js:1147
response.body = strings.join('')
^

RangeError: Invalid string length
at Array.join (native)
at Request. (/home/rich/node_modules/request/request.js:1147:31)
at Request.emit (events.js:198:13)
at IncomingMessage. (/home/rich/node_modules/request/request.js:1083:12)
at Object.onceWrapper (events.js:286:20)
at IncomingMessage.emit (events.js:203:15)
at endReadableNT (_stream_readable.js:1129:12)
at process._tickCallback (internal/process/next_tick.js:63:19)

Additional Information

[Any additional information, configuration or data that might be necessary to reproduce the issue]

The behavior is at least repeatable on Windows 11.

[phawxby]

@rrgerber can you provide a reproduction repo including config and importantly lockfile. request is not a dependency of this project, there is no reason it should be getting called.

paul@Pauls-MacBook-Pro node-website-scraper % npm ls request
website-scraper@5.2.0 /Users/paul/Documents/GitHub/node-website-scraper
└── (empty)

[s0ph1e]

Hi @rrgerber

In addition to @phawxby's suggestion you can try to take a look on logs.
Please also note that callbacks are not supported since version 5.0.0, please use async/await or promises, see usage examples

[rrgerber]

OK, I will check my code. Thanks to both of you for your responses.