Added resumptionToken argument to allow imports to continue a previous import. #36
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hi,
Thanks for your awesome work. While working on a very large import it seemed nice to be able to allow batches of records in stead of importing everything at once. The applications keeps tracks of the resumptionToken and stores it locally. When doing the next batch, we start at the last known resumptionToken.
For this to work I've added the resumptionToken as an argument for ListRecords and ListIdentifiers (and off course to the RecordIterator). I've also added public methods to fetch the resumptionToken and the current batch, since we need the batch to determine when we want to store the new resumptionToken.
Would you consider adding this?
By the way, I've used this library to create a migration provider for Drupal (https://www.drupal.org/project/migrate_oaipmh). The Drupal migrate module allows for batch imports, so that was the main reason to start this. But I think it would make sense in other cases as well.