feat(store): add ttl for scan

[rust84]

Description

Add a ttl to refresh scan store links every 5m. This is due to the founders card links being intermittently available. Currently the page is 404 but it was active yesterday when the cards went up. Credit to andrew on discord for the suggestion.

https://www.scan.co.uk/shops/nvidia/3090-founders-edition

Testing

Tested running snatcher locally with Scan enabled.

New dependencies

None.

[jef]

Would this override what's going out with PAGE_TIMEOUT?

[rust84]

The only place I can see that PAGE_TIMEOUT being used is in the lookup function when configuring the browser. The lookup function gets called after linksBuilder in the main loop so they shouldn't impact one another I think?

export async function tryLookupAndLoop(browser: Browser, store: Store) {
	if (store.linksBuilder) {
		const lastRunTime = linkBuilderLastRunTimes[store.name] ?? -1;
		const ttl = store.linksBuilder.ttl ?? Number.MAX_SAFE_INTEGER;  <-------------- LINKSBUILDER TTL
		if (lastRunTime === -1 || (Date.now() - lastRunTime) > ttl) {
			try {
				await fetchLinks(store, browser);
				linkBuilderLastRunTimes[store.name] = Date.now();
			} catch (error) {
				logger.error(error.message);
			}
		}
	}

	logger.debug(`[${store.name}] Starting lookup...`);
	try {
		await lookup(browser, store);       <------------- PAGE_TIMEOUT
	} catch (error) {
		logger.error(error);
	}

I've been running it on a branch for a couple of days and haven't had any issues thus far. The logs show the store links being updated every 5 mins which proves the ttl is being respected.

Oct 22 20:59:20 dev-shop-1 npm[3211]: [8:59:20 PM] debug :: ✖ [scan] [setup (3080)] :: DETECTING STORE LINKS
Oct 22 20:59:23 dev-shop-1 npm[3211]: [8:59:23 PM] debug :: [scan] Starting lookup...
...
Oct 22 21:04:43 dev-shop-1 npm[3211]: [9:04:43 PM] debug :: ✖ [scan] [setup (3080)] :: DETECTING STORE LINKS
Oct 22 21:04:46 dev-shop-1 npm[3211]: [9:04:46 PM] debug :: [scan] Starting lookup...

[jef]

Wow... I really misread this one. This was a huge misunderstanding on my part. I need to break 😅

For some reason I was confused because I only saw your one change, ttl and was curious where the other TTL code was. I thought you were changing the current implementation and was so confused. Especially with the store named Scan.

Thanks for putting this in!