fix: ignore invalid URLs in enqueueLinks in browser crawlers

[robertm-97]

Fixes: #1802

[robertm-97]

@mnmkng @vladfrangu @B4nan could anybody take a look at this? I'm happy with making code changes if needed.
Thanks in advance.

[mnmkng]

Thanks for the PR. I'm not sure if we want to silently ignore invalid URLs. Maybe this could be an option of enqueueLinks? throwOnInvalidURLs that users could optionally set to false ?

I would prefer not to ignore invalid URLs, because it usually points to some issues or inconsistencies in the crawlers. This way, we could be missing data and not even knowing about it.

[metalwarrior665]

@B4nan At least instead of just crashing with TypeError [ERR_INVALID_URL]: Invalid URL, we should expand the error message with the actual URL and context where it crashed (since the stack trace is sometimes tricky). There are more places in Crawlee and Apify SDK where this happens so we should do a bigger look.

[robertm-97]

I agree it should not be silently ignored. I can add a new option throwOnInvalidURLs that defaults to true, and also add better error handling when this happens.
Thanks

[B4nan]

Sounds good to me!

Btw it also feels weird that your PR changes only the browser crawler, we should ensure the other crawlers work the same - this should be probably implemented on lower level.

[robertm-97]

Other crawlers use the try catch approach which was also the inspiration for this PR.
But they also filter out invalid URLs from the array.
cheerio-crawler:

crawlee/packages/cheerio-crawler/src/internals/cheerio-crawler.ts

Lines 228 to 232 in cf64729

	try {
	return (new URL(href, baseUrl)).href;
	} catch {
	return undefined;
	}

jsdom-crawler:

crawlee/packages/jsdom-crawler/src/internals/jsdom-crawler.ts

Lines 268 to 273 in cf64729

	try {
	return (new URL(href, baseUrl)).href;
	} catch {
	return undefined;
	}

So if we would add the new throwOnInvalidURLs it needs to be added to the other crawlers as well.

[B4nan]

Right, I thought we do this somewhere already. In that case, we should align the behavior probably.

[robertm-97]

I aligned the enqueueLinks behavior across crawlers. Also added a helper function for this to avoid duplicated code.