Handle failures in pending jobs

[dalelane]

At the moment, all pending job processing is stopped when one fails.

We could improve on this, so that other jobs can still be attempted if there is a single blip.

[dalelane]

The code lives here:

taxinomitis/src/lib/pendingjobs/runner.ts

Lines 16 to 45 in 0f30eed

	try {
	const start = new Date();
	nextJob = await db.getNextPendingJob();
	while (nextJob) {
	await processor.processJob(nextJob);
	await db.deletePendingJob(nextJob);
	nextJob = await db.getNextPendingJob();
	}
	const end = new Date();
	const durationMs = end.getTime() - start.getTime();
	log.info({ durationMs }, 'No pending jobs remain');
	if (durationMs > constants.THREE_HOURS) {
	notifications.notify('Processing pending jobs took longer than THREE HOURS');
	}
	}
	catch (err) {
	log.error({ err, nextJob }, 'Pending job failure');
	notifications.notify('Critical failure in processing pending jobs: ' + err.message);
	if (nextJob) {
	db.recordUnsuccessfulPendingJobExecution(nextJob);
	}
	}

The try...catch is laid out so that if one task fails, it throws us out of the while (nextJob) loop.

[dalelane]

Closing - as not worth the effort. In over a year, it's never actually failed, so investing in handling failure seems like a waste of time. Worst case scenario, if there is a failure, I can manually remove the failing job to unblock the rest - and then revisit whether it's worth automating it.