throw new Error(Directory ${absoluteDirectoryPath} exists);

[obsidianart]

Configuration

version: [result of npm ls website-scraper --depth 0 command]
website-scraper@5.3.1

options: [provide your full options object]

import scrape from 'website-scraper'; // only as ESM, no CommonJS
const options = {
  urls: ['https://www.imagzle.com'],
  directory: './docs'
};

// with async/await
const result = await scrape(options);

// with promise
scrape(options).then((result) => {});

Description

The script is throwing an error. I did try to delete the folder but the error happens regardless after the script creates the folder

Expected behavior: [What you expect to happen]
No errors
Actual behavior: [What actually happens]
stefanosolinas@stefanos-MacBook-Pro-2 scraper % node index.js
file:///Users/stefanosolinas/HDD/GIT/ollama%20private/scraper/node_modules/website-scraper/lib/plugins/save-resource-to-fs-plugin.js:16
throw new Error(Directory ${absoluteDirectoryPath} exists);
^

Error: Directory /Users/stefanosolinas/HDD/GIT/ollama private/scraper/docs exists
at file:///Users/stefanosolinas/HDD/GIT/ollama%20private/scraper/node_modules/website-scraper/lib/plugins/save-resource-to-fs-plugin.js:16:11
at Scraper.runActions (file:///Users/stefanosolinas/HDD/GIT/ollama%20private/scraper/node_modules/website-scraper/lib/scraper.js:229:20)
at Scraper.scrape (file:///Users/stefanosolinas/HDD/GIT/ollama%20private/scraper/node_modules/website-scraper/lib/scraper.js:276:15)
at default (file:///Users/stefanosolinas/HDD/GIT/ollama%20private/scraper/node_modules/website-scraper/index.mjs:4:30)
at file:///Users/stefanosolinas/HDD/GIT/ollama%20private/scraper/index.js:11:1

Node.js v20.3.0

Additional Information

[Any additional information, configuration or data that might be necessary to reproduce the issue]

[aivus]

Hello @obsidianart 👋

Seems strange.

Could you please check that there are no other processes of the module running?

Try to remove directory /Users/stefanosolinas/HDD/GIT/ollama private/scraper/docs, verify that it's removed and not re-created and then run the script.

[obsidianart]

I restarted my machine, deleted docs, nothing else running, same error.

[aivus]

Try to use directory: './docs/' + new Date().getTime(),
It will create a directory inside the docs folder based on the time.

Please check and let me know do you have the same issue as above or not

[deweller]

I'm also having this same issue.

[tushar-rupani]

With the same options the files have been downloaded in the respected path you have mentioned for the first time with this code.

const result = await scrape(options);

and with the same options again you're using Promise .then() syntax. It will again try to upload files there - henceforth, it is causing that error. either we use async/await or promise.

[aivus]

Good catch @tushar-rupani !

I missed the fact that scraper called 2 times

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.