Implementation of data quality controls on EDC connectors

[abelmunozal]

Hello all,
I want to share with you a question that, I think, could be helpful for many people: how to implement data/assets quality controls in EDC. Imagine the following use case: a consumer connector that gathers data from more than one provider connector and joins it.

The idea is, after the join, to perform quality controls in order to leave out results with issues: relation mistakes, unrelated data, information mistakes, etc.

My first idea was to implement it in a data transport extension on the consumer connector; once the data is fully received.
After a study of the IDS RAM 4.0, I thought of developing the concept of AppStores in my EDC connector. Until I know, it could help me to implement more sophisticated (and heavyweight) data quality controls as an isolated process out of the connector code. But it could take a lot of time and effort.

What do you think about what is the best way? Could quality controls be implemented as an extension of EDC? What about heavyweight quality control processes: lineage, quarantine, ...? Is there any roadmap in order to implement support for the AppStore IDS concept in EDC?

I hope this question will be helpful in addressing quality control in EDC and give us as many ideas as possible.

Thanks in advance,

[jimmarino]

Hi,

Thanks for the write-up. How a consumer processes data is outside the scope of EDC, and this can be implemented in various ways. For example, with support for the new IDS Dataspace Protocol Specifications (work in progress), you could use the EDC event system to be notified when a transfer process completes, which could initiate quality control processing by some other system.

The EDC is focused on implementing the recently released IDS Dataspace Protocol specifications. These specifications do not include the concept of an AppStore, and the core EDC project would likely not implement such as concept as it is outlined in the current RAM. Other projects could do this using the core capabilities provided by EDC if there was a requirement to do so.

[abelmunozal]

Hi Jim. Yes, I assisted to the sessions.

What I'm not sure about is how to do that without the new IDS Dataspace Protocol specifications or IDS RAM 4.0. I'm not sure if they have attained the required maturity to be used in production environments.

On the other side, my idea is to create an open-source project, and I'm not sure if the licencing of IDS connectors really allows working in an open community. I think (maybe I'm wrong), this situation was the motivation behind the born of the Eclipse Dataspaces Community.

[jimmarino]

The motivation behind the EDC was to create an open source project where organizations and individuals collaborate on data sharing based on interoperable usage control policies. The project is governed by the Eclipse Foundation and its community guidelines and is distributed under the Apache 2.0 license.

You'll need to determine if the EDC license is acceptable and if the code fits your production requirements.