Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve handling of failed updates #284

Closed
3 tasks done
fsteeg opened this issue Feb 5, 2021 · 3 comments · Fixed by #314
Closed
3 tasks done

Improve handling of failed updates #284

fsteeg opened this issue Feb 5, 2021 · 3 comments · Fixed by #314
Assignees
Labels

Comments

@fsteeg
Copy link
Member

fsteeg commented Feb 5, 2021

We need to improve handling of failing updates.

On 2021-02-03 we got a "GND updates fails" email. After a quick check in the the data/backup folder all updates seemed to be there. We have a mechanism that should include missed updates on the next day (GND-lastSuccessfulUpdate.txt, see https://github.com/hbz/lobid-gnd/blob/master/cron.sh). But thankfully @dr0i pushed me to investigate if this actually worked, and it didn't: the backup file is empty, so we either missed updates for that day or there were no updates(?).

What we should do:

  • Fix the mechanism based on GND-lastSuccessfulUpdate.txt
    - [ ] Don't copy empty update files to data/backup
  • Handle the case where the OAI-PMH service works fine, but there are no updates
  • Re-run updates for the missing day
@dr0i
Copy link
Member

dr0i commented Jun 3, 2022

Also experienced in #313.

@fsteeg
Copy link
Member Author

fsteeg commented Jun 3, 2022

Seems I lost track of this issue, and it might be a simple fix.

  • Fix the mechanism based on GND-lastSuccessfulUpdate.txt

Pushed 5d41b7b to only write the last successful update date after actual success.

  • Don't copy empty update files to data/backup

I think this one is actually nice the way it is, since we have a full record of what happened. When we look at the backups, we see the update happened, but the file was empty. If it's not there, we don't know if something was there and was later deleted etc. What do you think, @dr0i?

  • Re-run updates for the missing day

I think with the fix above, this is already handled: on the next day, we'll use the same older start day again, as on the day before, since we did not update the date when we got no data.

  • Handle the case where the OAI-PMH service works fine, but there are no updates

I think with the fix above, this should be fine too: if the response is not completely empty, the date will be updated, and on the next day we'll continue, as if there had been some updates.

@dr0i
Copy link
Member

dr0i commented Jun 3, 2022

+1 for keeping copy of empty update files.
I've crossed the todo out.

@dr0i dr0i assigned fsteeg and unassigned dr0i Jun 3, 2022
@fsteeg fsteeg closed this as completed in 991456d Jun 3, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants