Skip to content
This repository has been archived by the owner on Oct 24, 2022. It is now read-only.

Ensure Harvests don't try to run for future dates #240

Closed
JPrevost opened this issue Jun 29, 2015 · 1 comment · Fixed by #281
Closed

Ensure Harvests don't try to run for future dates #240

JPrevost opened this issue Jun 29, 2015 · 1 comment · Fixed by #281
Assignees
@JPrevost
Milestone
Launch Prep

Comments

@JPrevost
Copy link
Member

If a Harvest tries to run in which the capture date is for a date that has not yet happened, the Harvest should abort and log.

More specifically, we should only Harvest for dates up to and including yesterday. Never for today or future dates.

This could happen when either someone manually triggers harvests multiple times or when someone manually triggers a harvest prior to a scheduled Harvest being kicked off.
@JPrevost JPrevost added this to the Launch Prep milestone Jun 29, 2015
@JPrevost
Copy link
Member Author

Scenarios to try to better explain the potential problem:

Today is 4/15/2015
HarvestA last ran 4/13/2015
Cron kicks the job to run with a one day Harvest of 4/14/2015.
This should run.

Today is 4/15/2015
HarvestA last ran 4/14/2015
Cron (or an admin) kicks the job to run with a one day Harvest of 4/15/2015.
This should NOT run as 4/15 is not yet complete and thus more records could be added we'd miss.

Today is 4/15/2015
HarvestA last ran 4/15/2015 (this should not have been possible, but assume we somehow hit this error state we need to not perpetuate it)
Cron kicks the job to run with a one day Harvest of 4/16/2015.
This should NOT run.

@JPrevost JPrevost self-assigned this Jul 10, 2015
@JPrevost JPrevost added the ready label Jul 10, 2015
@JPrevost JPrevost mentioned this issue Jul 13, 2015
Merged
JPrevost added a commit that referenced this issue Jul 15, 2015
@JPrevost
Prevent batch harvests from requesting invalid dates
d6a5136 
This prevents batch harvests from requesting date that is older than
yesterday with the assumption that it’s safest to not request an OAI
feed for today as more data may be added later we’d miss. I’m not
entirely sure that’s valid logic, but this is safe. The harvester
increments the updated date hence the need to make sure it is less than
yesterday (i.e. updated needs to be two days ago to actually run which
would then get incremented by 1 to be a harvest for yesterday… clear as
mud).

Admin kicked off harvests and knips are not limited by date. Only the
batch starter.

closes #240
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@JPrevost JPrevost

Labels
None yet
Projects
None yet
Milestone
Launch Prep
Development

Successfully merging a pull request may close this issue.

Prevent batch harvests from requesting invalid dates MITLibraries/topichub
1 participant
@JPrevost