Charset on final file is not like expected

[TiagoGouvea]

Configuration

version: website-scraper@5.3.1

options: [provide your full options object]

const options = {
  urls: ['https://www.appmasters.io/pt/'],
  directory: '/Users/tiagogouvea/www/scrapper/test/app/',
  plugins: [ new SaveToExistingDirectoryPlugin() ]
};

Description

Trying to scrap a website that has <meta charSet="utf-8"/> results on a content without correct charset presentation.

Following is a screenshot showing the original content and the scrapped one.

Expected behavior: That the save filed was on the same charset.

Actual behavior: It failed to save on the current way.

Additional Information

Congratulations for this amazing project! 👏👏👏

[henq98]

Edit index.js in website-scraper-existing-directory module folder
from
await fs.outputFile(filename, text, { encoding: 'binary' });
to
await fs.outputFile(filename, text, { encoding: 'utf-8' });

[aivus]

@henq98 good catch! We will check this.

@s0ph1e looks like we need to replace

website-scraper-existing-directory/index.js

Line 19 in 09e8461

await fs.outputFile(filename, text, { encoding: 'binary' });

with

await fs.outputFile(filename, text, { encoding: resource.getEncoding() });

[aivus]

Issue moved to #23

[s0ph1e]

Hi all 👋

I've just released a bug fix in version website-scraper-existing-directory@1.0.1

Thank you for your collaboration 👍

[TiagoGouvea]

Great! Thank you all!