Deprioritise ingestion from providers with successive error

[masih]

There seem to be a cascading error effect, where if a couple of providers repeatedly error during ad traversal they can slow down the ingestion for other providers.

The degree by which ingest is slowed down depends on the number of ingest workers and overall traffic on the indexer.

Indexers should deprioritise ingestion from providers that have errored numerously, with some exponential backoff. This then should allow indexers to make overall progress and come back to erroneous providers which should reduce cascading effects across the system.

[gammazero]

On every poll cycle, for each provider that is polled, we record the last poll sequence. In each subsequent poll cycle the providers to poll are sorted according to poll sequence, so those with older poll sequences are preferred. This avoids polling the same providers during each poll period if others should be polled.

Ingestion should only occur repeatedly for errored providers if they are the only ones that should be polled. We could add a backoff timer for providers that have errors across consecutive polls, but polling an errored provider should not affect performance significantly since that does not cause anything that was already synced to be resynced.