Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Prevent deleted harvesters from running until purged (fix #2185) #2209

Merged
merged 4 commits into from Jun 25, 2019
Merged

Prevent deleted harvesters from running until purged (fix #2185) #2209

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
87909b0
Prevent deleted harvesters from running until purged (fix #2185)
noirbizarre Jun 21, 2019
44284e2
Fix missing deleted attribute in harvest source mask
noirbizarre Jun 25, 2019
9f1dac2
Delete jobs when purging harvest sources
noirbizarre Jun 25, 2019
7ec5da7
Updated changelog
noirbizarre Jun 25, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Expand Up @@ -16,6 +16,7 @@
- Remove diff from js dependencies to fix CVE [#2204](https://github.com/opendatateam/udata/pull/2204)
- Replace default sort label for better readability [#2206](https://github.com/opendatateam/udata/pull/2206)
- Add a condition to up-to-dateness of a dataset [#2208](https://github.com/opendatateam/udata/pull/2208)
- Prevent deleted harvesters from running until purged. Harvest jobs history is deleted too on purge. [#2209](https://github.com/opendatateam/udata/pull/2209)

## 1.6.11 (2019-05-29)

Expand Down
2 changes: 1 addition & 1 deletion js/views/harvester.vue
View file
Open in desktop
Expand Up @@ -45,7 +45,7 @@ import SourceWidget from 'components/harvest/source.vue';
import JobWidget from 'components/harvest/job.vue';
import Layout from 'components/layout.vue';

const MASK = ['id', 'name', 'url', 'owner', 'organization', 'backend', 'validation{state}', 'schedule'];
const MASK = ['id', 'name', 'url', 'owner', 'organization', 'backend', 'validation{state}', 'schedule', 'deleted'];

export default {
name: 'HarvestSourceView',
Expand Down
2 changes: 1 addition & 1 deletion udata/harvest/models.py
View file
Open in desktop
Expand Up @@ -159,7 +159,7 @@ class HarvestJob(db.Document):
default=DEFAULT_HARVEST_JOB_STATUS, required=True)
errors = db.ListField(db.EmbeddedDocumentField(HarvestError))
items = db.ListField(db.EmbeddedDocumentField(HarvestItem))
source = db.ReferenceField(HarvestSource, reverse_delete_rule=db.NULLIFY)
source = db.ReferenceField(HarvestSource, reverse_delete_rule=db.CASCADE)
data = db.DictField()

meta = {
Expand Down
6 changes: 4 additions & 2 deletions udata/harvest/tasks.py
View file
Open in desktop
Expand Up @@ -19,6 +19,8 @@ def harvest(self, ident):
log.info('Launching harvest job for source "%s"', ident)

source = HarvestSource.get(ident)
if source.deleted:
return # Ignore deleted sources
Backend = backends.get(current_app, source.backend)
backend = Backend(source)
items = backend.perform_initialization()
Expand Down Expand Up @@ -75,8 +77,8 @@ def harvest_finalize(results, source_id):
harvest_job_finalize(results, job.id)


@task(route='low.harvest')
def purge_harvest_sources():
@job('purge-harvesters', route='low.harvest')
def purge_harvest_sources(self):
log.info('Purging HarvestSources flagged as deleted')
from .actions import purge_sources
purge_sources()