Job reporting fixes #107
Closed
Job reporting fixes #107
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…during last two harvests
…datasets to gather
| @@ -178,6 +178,8 @@ def _create_or_update_package(self, package_dict, harvest_object): | |||
|
|
|||
| else: | |||
| log.info('Package with GUID %s not updated, skipping...' % harvest_object.guid) | |||
| harvest_object.report_status = 'not modified' | |||
There was a problem hiding this comment.
I'd much rather report_status was only updated in queue.py, rather than introduce a new place where it potentially gets updated. Also, this is a deprecated method to be updating.
My preference would be for fetch_stage() to return True, False or "unchanged" to queue.py and set report_status there.
|
I modified it based on your comments, now import stage returns "unchanged" if it the dataset is unchanged and queue.py set report_status based on this. |
davidread
pushed a commit
that referenced
this pull request
Nov 3, 2015
* fix "existing_package_dict" which wasn't containing metadata_modified (because of the schema in the context) so you never skipped an object. * fix IntegrityError due to resource revision_id being harvested. No idea why this hasn't caused errors before now. * "unchanged" is now checked in base instead of ckanharvester - makes sense. Looking at other harvesters, it's normal to return from the import_stage with the value returned from base._create_or_update_package so I've continued with that. * "unchanged" response is now documented * better report_status tests in test_queue2.
|
I've taken these changes, added some tests and related fixes and created PR #177 as a replacement for this one. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Job reporting has some inconsistencies which causes confusing behavior.
If harvest job is executed and there are zero new revision, the harvest job never receives job finished timestamp.
If harvest job is executed again, it refresher all harvested datasets, but since it creates new HarvestObjects, they don't have report_status saved and the job report shows that all datasets are deleted although nothing actually changed.