Ignore data:image URLs

[nikolay-borzov]

Configuration

version: 4.2.3

options:

{
  urls: [
   'http://example.com/product/abc'
  ],
    directory: path.resolve(__dirname, 'src'),
    urlFilter(url) {
      return url.indexOf('data:image') === -1;
    }
}

Description

The page contains image like:

<picture>
 <source media="(max-width: 559px)
   srcset="data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7 1x,
           data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7 2x">
  <source media="(min-width: 560px) and (max-width: 719px)"
    srcset="data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7 1x,
            data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7 2x">
  <source media="(min-width: 720px) and (max-width: 899px)"
    srcset="data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7 1x,
            data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7 2x">
  <source media="(min-width: 900px) and (max-width: 1199px)"
    srcset="data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7 1x,
            data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7 2x">
  <source media="(min-width: 1200px)"
    srcset="data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7 1x,
            data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7 2x">
  <img
    src="data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7"
    srcset="http://example.com/product/abc/media/521811121-392x351.jpg 2x">
</picture>

data:image URLs are not fully ignored despite setting urlFilter to filter out URLs containing 'data:image'. It looks like there are two URLs extracted from data:image URL - one with data:image/gif;base64, another with the rest part after , - R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7

website-scraper:debug filtering out { url: "data:image/gif;base64", filename: "undefined", depth: 3 } by url filter +0ms
website-scraper:debug found requested resource for { url: "http://example.com/product/abc/R0lGODlhAQABAIAAAAAAAP///R0lGODlhAQABAIAAAAAAAP///R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7", filename: "undefined", depth: 3 
} +0ms

R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7 is attached to base URL

Expected behavior: There should not be any downloads from URLs like Base URL + data:image URL

Actual behavior: HTML is downloaded as yH5BAEAAAAALAAAAAABAAEAAAIBRAA7_1.html

Additional Information

Downloaded HTML contains extra space between data:image/gif;base64, and R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7

<picture>
 <source media="(max-width: 559px)
   srcset="data:image/gif;base64, R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7 1x,
           data:image/gif;base64, R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7 2x">

I'm sorry I cannot provide the real URL. It's an internal URL on the customer side.

[s0ph1e]

Hi @nikolay-borzov 👋

Thank you for reporting this issue, I will try to reproduce it

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[aivus]

A space is adding because of the bug in srcset module (sindresorhus/srcset#1).
However this bug was fixed in v3.0.0.

Upgrade to this major version of srcset requires Node.js 10+, so we need to drop old versions here. I've created #438 for this.