dpul:reindex:collections does not fully index collections

[hackartisan]

When the reindex is run in bulk via this rake task collections do not get fully indexed. For example, when run recently on dpul-staging the shakespeareandco collection had 548 items after all the indexing was done. On prod it has 787.

This bug has been observed now in both prod and staging.

eastasian also indexed with many fewer items: 247 out of 441. However, sae_sri_lanka_dissidents and piranesi indexed completely.

When run individually via the console, shakespeareandco indexed all records.

[kelea99]

@hackartisan Steve F. Just reported the Lapidus collection went down to only 336 items from over a thousand. It is the same issue I saw with Wills manuscripts collection. Something is up with reindex again. Can you have a look at what is failing?

[hackartisan]

According to figgy there should be 2542 complete open items in lapidus.

[hackartisan]

I ran Spotlight::ReindexJob.perform_later on lapidus via console and it's starting to fill up more -- currently at 590 and climbing.

[hackartisan]

In the new version of spotlight when an index job fails (and several other types of jobs are like this, too), it is not re-queued. Instead the failure is logged and then displayed on the exhibit dashboard. The error never gets to honeybadger and the dashboard shows much less information than honeybadger does.

I think we want to make the error raise still so it goes through honeybadger.