Export uses a queue for every handle(url), so scaling in exponential concurrent promises

[bwbroersma]

Problem is that the queue is per handle(url):

sapper/src/api/export.ts

Lines 165 to 187 in 7be7e1e

	const q = yootils.queue(8);
	const base_match = /<base ([\s\S]+?)>/m.exec(cleaned);
	const base_href = base_match && get_href(base_match[1]);
	const base = resolve(url.href, base_href);
	let match;
	let pattern = /<a ([\s\S]+?)>/gm;
	while (match = pattern.exec(cleaned)) {
	const attrs = match[1];
	const href = get_href(attrs);
	if (href) {
	const url = resolve(base.href, href);
	if (url.protocol === protocol && url.host === host) {
	q.add(() => handle(url));
	}
	}
	}
	await q.close();

See https://github.com/bwbroersma/exponential-export-queue for the issue.

I changed the yootils queue behavior Rich-Harris/yootils#3 and have a simple sapper patch.

[bwbroersma]

See v0.25.0...bwbroersma:gh-604 for a fix for v0.25.0

[bwbroersma]

Also added an option concurrent for export, could also be named parallel or something. Might be useful not to hammer API's.

My API got hammered quite a bit (seeing FATAL: remaining connection slots are reserved for non-replication superuser connections on my postgres). It took me a time to realize what was happening since at first I was also experiencing the b5edf0e bug (and was switching between sapper 0.25 and alpha).

[bwbroersma]

BTW it's related to #369 and the current export that will try to crawl 100+ pages simultaneously (since I got API postgres errors) my case is a realistic website (: