Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Better handling for records that fail to index #168

Closed
thatbudakguy opened this issue Feb 16, 2024 · 1 comment
Closed

Better handling for records that fail to index #168

thatbudakguy opened this issue Feb 16, 2024 · 1 comment
Assignees

Comments

@thatbudakguy
Copy link
Member

When a record fails to index, the cause could be one of a number of things:

  • the record is invalid, and solr rejects it for not matching the schema
  • the http request failed or was interrupted
  • we made too many requests to solr and got blocked or throttled

In all of these cases, it'd be better to log the failure and continue on – currently indexing just stops. In some of them, it might also be useful to retry the request (possibly after a delay) using faraday-retry or similar.

@thatbudakguy thatbudakguy self-assigned this Feb 23, 2024
thatbudakguy added a commit that referenced this issue Feb 23, 2024
- Gracefully handle indexing failures
- Use Blacklight's solr connection if available

Closes #168
Closes #166
thatbudakguy added a commit that referenced this issue Feb 23, 2024
- Gracefully handle indexing failures
- Use Blacklight's solr connection if available

Closes #168
Closes #166
thatbudakguy added a commit that referenced this issue Feb 23, 2024
- Gracefully handle indexing failures
- Use Blacklight's solr connection if available

Closes #168
Closes #166
@srappel
Copy link

srappel commented Mar 12, 2024

This made indexing much less frustrating and the info output is much easier to understand. Thanks @thatbudakguy !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants