New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Publish kafka message when resource is created or modified #2733
Conversation
…a into producerResourceEvents
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🌟 Nice PR!
I don't think I could make the kafka messages being produced yet.
If I'm correct, on update_resource API call, update_resource
method is not called, thus no signal on_resource_updated would be sent.
@@ -682,7 +682,9 @@ def add_resource(self, resource): | |||
} | |||
}) | |||
self.reload() | |||
self.on_resource_added.send(document=self, resource_id=resource.id) | |||
print('BEFORE SEND') | |||
self.on_resource_added.send(self.__class__, document=self, resource_id=resource.id) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need to explicitly send self.__class__
? It doesn't seem used?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The first parameter of the send
method is sender
, and we use to send the sender as the first param elsewhere in the software.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we can use the params we want, having document and resource_id only?
Co-authored-by: maudetes <maudet.estelle@gmail.com>
from udata.event.producer import produce | ||
|
||
|
||
def serialize_resource_for_event(resource): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we move dataset serialization in events as well? (same question for reuses and orgas) 🤔
@@ -682,7 +682,9 @@ def add_resource(self, resource): | |||
} | |||
}) | |||
self.reload() | |||
self.on_resource_added.send(document=self, resource_id=resource.id) | |||
print('BEFORE SEND') | |||
self.on_resource_added.send(self.__class__, document=self, resource_id=resource.id) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we can use the params we want, having document and resource_id only?
…a into producerResourceEvents
Co-authored-by: maudetes <maudet.estelle@gmail.com>
…a into producerResourceEvents
Co-authored-by: maudetes <maudet.estelle@gmail.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Almost good I think 😊
udata/core/dataset/tasks.py
Outdated
@@ -52,8 +52,7 @@ def purge_datasets(self): | |||
# Remove each dataset's resource's file | |||
storage = storages.resources | |||
for resource in dataset.resources: | |||
if resource.fs_filename is not None: | |||
storage.delete(resource.fs_filename) | |||
dataset.remove_resource(resource) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Iterating on a list while removing its elem seem like a bad idea (skip some iteration) sadly.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Put back the logic:
if resource.fs_filename is not None:
storage.delete(resource.fs_filename)
Dataset.on_resource_removed.send(Dataset, document=dataset, resource_id=resource.id)
without the dataset.remove_resource
, as the resources will be deleted with the dataset deletion and the important is to delete the file in the storage and to emit the signal.
…a into producerResourceEvents
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
✨
fix etalab/data.gouv.fr#825