Add DID data type to transfer and deletion events #4557
Comments
|
🤔 |
|
This came up as a request from the sites; they would like to be able to identify what kind of files are being deleted. On the DDM Dashboard, it appears to be useful to be able to group by data type. For example, |
|
I think it would be much easier if this is added to the monitoring/dashboard pipeline. Adding the |
|
What was the decision about this one ? |
|
I don't remember discussing this anymore. @dchristidis @cserf any info? |
|
I support the idea to add this to the monitoring pipeline, @dchristidis do you prefer to get this info on some dashboards or from rucio directly? |
|
Cedric and I just discussed this: It should be the datatype (link) field from the Let's evaluate where we can most efficiently get the did information pulled in. Join in an existing query? |
The commit takes quite a dirty approach of retrieving the `datatype` directly via a sql query when preparing the message. There is no easy way to improve that: - In all code path, there is a lot of logic between the moment when we retrieve the work queue from the database and the moment when we sent the message. Forwarding the datatype through all the call stack will make the code more complicated. - We cannot import from core.dids here, because it creates a circular import problem. So using existing get_metadata calls is not easily achievable to avoid a raw database call.
The commit takes quite a dirty approach of retrieving the `datatype` directly via a sql query when preparing the message. There is no easy way to improve that: - In all code path, there is a lot of logic between the moment when we retrieve the work queue from the database and the moment when we sent the message. Forwarding the datatype through all the call stack will make the code more complicated. - We cannot import from core.dids here, because it creates a circular import problem. So using existing get_metadata calls is not easily achievable to avoid a raw database call.
The commit takes quite a dirty approach of retrieving the `datatype` directly via a sql query when preparing the message. There is no easy way to improve that: - In all code path, there is a lot of logic between the moment when we retrieve the work queue from the database and the moment when we sent the message. Forwarding the datatype through all the call stack will make the code more complicated. - We cannot import from core.dids here, because it creates a circular import problem. So using existing get_metadata calls is not easily achievable to avoid a raw database call.
Motivation
This will improve our monitoring capabilities, allowing for better understanding of data flows.
Modification
Enrich the transfer and deletion events with the DID data type. It’s fine if it’s unset or some communities do not use it at all. We should carefully evaluate the performance cost.
The text was updated successfully, but these errors were encountered: