Replies: 4 comments 11 replies
-
|
Hi @hlange thanks! Can you add your observations here and we can create issues and figure out where to put the documentation based on that? |
Beta Was this translation helpful? Give feedback.
-
|
sure, when I have the text, I will post it here |
Beta Was this translation helpful? Give feedback.
-
GAIA-X4KI - Sending query data to the provider1. IntroductionAs part of the GAIA-X 4KI project funded by the federal ministry for economic affairs and climate action of Germany, prototypes were developed to learn more about the Eclipse Dataspace Connector (EDC). A challenge in the project was data transmission from a consumer to a provider. The consumer should send a data set to the provider, who responds with an appropriate answer. The connectors were developed using a version that existed between milestones 6 and 7. The reason for this was that the In the following sections, the experiences made during the development of such a connector are described. 2. PrototypesSeveral prototypes were developed to see how the EDC could be used for this use case. Necessary extensions were identified and examined for their practicability for the exchange of information from consumer to provider. This will be discussed in more detail in the next sections. 2.1 Data Plane Transfer ClientThe first transfer option to be implemented in a connector was the "Data Plane Transfer Client" because the examples in the repository use this and are therefore the most well-known. The following sections deal with the extensions and the corresponding places in the source code, as well as their configuration. It is also shown how the data exchange can be carried out with it. 2.1.1. SetupThe prototype had the following extensions included.
The associated configuration file for the provider was the following: web.http.port=8181
web.http.path=/api
web.http.data.port=8182
web.http.data.path=/api/v1/data
ids.webhook.address=http://127.0.0.1:8282
edc.ids.id=urn:connector:provider
edc.api.auth.key=password
edc.transfer.functions.enabled.protocols=httpHere is an example that shows how the provider was started: java -Dedc.fs.config=launchers/general_connector/config/provider.config.properties \
-Dedc.vault=launchers/general_connector/vault.properties \
-Dedc.keystore=launchers/general_connector/certs/cert.pfx -Dedc.keystore.password=123456 \
-jar launchers/general_connector/build/libs/connector.jar
2.1.2. Relevant Source CodeThe next sections introduce the relevant elements in the source code that were helpful to understand the functionality of the implementation. 2.1.2.1. DataRequestAfter a contract has been negotiated, a request can be sent to the provider, this request is transformed into a private Map<String, String> properties = new HashMap<>();2.1.2.2. Mapping of DataRequest on DataFlowRequestThe above private Map<String, String> properties = Map.of();The private DataFlowRequest createRequest(DataRequest dataRequest, DataAddress sourceAddress) {
return DataFlowRequest.Builder.newInstance()
.id(UUID.randomUUID().toString())
.processId(dataRequest.getProcessId())
.trackable(true)
.sourceDataAddress(sourceAddress)
.destinationType(dataRequest.getDestinationType())
.destinationDataAddress(dataRequest.getDataDestination())
.callbackAddress(callbackUrl != null ? callbackUrl.get() : null)
.properties(dataRequest.getProperties())
.build();2.1.2.3. HttpSourceRequestParamsSupplierAfter the objects have been converted, the data source can be requested using these parameters. However, for this part, it is important to know how the requests are created based on the parameters defined during asset creation. Supply BodyA body can only be transferred if the required property @Override
protected @Nullable String extractBody(HttpDataAddress address, DataFlowRequest request) {
return Boolean.parseBoolean(address.getProxyBody()) ? request.getProperties().get(BODY) : null;
}However, this not only affects the body, but also the possibility of setting the content type. @Override
protected @Nullable String extractContentType(HttpDataAddress address, DataFlowRequest request) {
return Boolean.parseBoolean(address.getProxyBody()) ? request.getProperties().get(MEDIA_TYPE) : null;
}Supply Method
@Override
protected @NotNull String extractMethod(HttpDataAddress address, DataFlowRequest request) {
if (Boolean.parseBoolean(address.getProxyMethod())) {
return Optional.ofNullable(request.getProperties().get(METHOD))
.orElseThrow(() -> new EdcException("Missing http method for request: " + request.getId()));
}
return DEFAULT_METHOD;
}2.1.3. UsageWith the knowledge from the previous sections, the connectors can be used. 2.1.3.1. ProviderFor the provider, the asset must be created with the {
"asset": {
"properties": {
"asset:prop:id": "asset-1",
"asset:prop:name": "asset name",
"asset:prop:contenttype": "application/json",
"asset:prop:policy-id": "use-eu"
}
},
"dataAddress": {
"properties": {
"baseUrl": "http://localhost:8000",
"name":"",
"type": "HttpData",
"proxyQueryParams":true,
"proxyMethod": true,
"proxyBody": true,
"proxyPath": true
}
}
}
2.1.3.2. ConsumerThis allows the consumer to pass on additional information to the data source via the {
"protocol": "ids-multipart",
"assetId": "asset-1",
"contractId": "2:30c2851a-c3a6-4f45-a40a-b60ff633832f",
"dataDestination": {
"properties": {
"baseUrl":"http://127.0.0.1:8001",
"type": "HttpData"
}
},
"properties" : {
"body" : "Here could be your data",
"method":"POST",
"mediaType":"text/plain"
},
"transferType": {
"contentType": "application/octet-stream",
"isFinite": true
},
"managedResources": false,
"connectorAddress": "http://127.0.0.1:8282/api/v1/ids/data",
"connectorId": "consumer"
}
2.1.4. ImpressionThis solution is much easier to implement than the solution in the next chapter, but it wasn't very obvious that this solution existed. This is partly due to the documentation being developed, but also due to the naming scheme when creating an asset. The use of Proxy as a prefix can easily be associated with the During development, no information or test cases were found that mentioned this transfer option. Furthermore, this implementation was only possible after problem #1972 was fixed. For this reason, it cannot be guaranteed that this was intended. However, apart from naming the asset attribute with the proxy prefix, this is a good solution for some use cases, especially when the interaction between participants does not require many interactions. 2.2. Data Plane Transfer SyncAs a second prototype, the "Data Plane Transfer Sync" was implemented as a transfer option. In this implementation, the Catena-X@Home repository was used, which required a similar structure for the connectors, but differed in part in the implementation. The following sections describe the implementation and its use in more detail. 2.2.1. SetupThe prototype included the following additional extensions compared to the previous one.
The associated configuration file for the provider was as follows: web.http.port=8181
web.http.path=/api
web.http.data.port=8182
web.http.data.path=/api/v1/data
web.http.ids.port=8183
web.http.ids.path=/api/v1/ids
web.http.validation.port=8184
web.http.validation.path=/validation
web.http.public.port=8185
web.http.public.path=/public
web.http.dataplane.port=8186
web.http.dataplane.path=/dataplane
web.http.control.port=8187
web.http.control.path=/control
edc.ids.id=urn:connector:provider
edc.ids.endpoint=http://localhost:8183/api/v1/ids
ids.webhook.address=http://localhost:8183
edc.transfer.proxy.endpoint=http://localhost:8185/public/
edc.dataplane.token.validation.endpoint=http://localhost:8184/validation/token
edc.api.auth.key=password
edc.receiver.http.endpoint=http://localhost:8000/
edc.transfer.functions.enabled.protocols=http
edc.transfer.proxy.token.signer.privatekey.alias=1
edc.transfer.proxy.token.verifier.publickey.alias=public-keyThe provider was started this way: java -Dedc.fs.config=launchers/sync_connector/config/provider.config.properties \
-Dedc.vault=launchers/sync_connector/vault.properties \
-Dedc.keystore=launchers/sync_connector/certs/cert.pfx -Dedc.keystore.password=123456 \
-jar launchers/sync_connector/build/libs/connector.jar2.2.2. Relevant Source CodeThe next sections present the relevant elements in the source code that contribute to understanding how this implementation works. The parts that were explained for the previous prototype are also relevant here, but are not mentioned again. 2.2.2.1. Data-Plane-SelectorThe first part to consider for this prototype is how the Data Plane Selector extension works. This is necessary even though the data plane is included in the connector in this implementation. There are two implementations for the data plane selector in the DataPlane Selector Client:
Which one is used by the connector depends on whether the This will select the data plane based on the configured options and the information received from the request. For one to be chosen, it must first be communicated to the connector, as there is currently no default data plane. This is done via a REST API where a An example is shown below, the {
"id":"dataplane1",
"edctype": "dataspaceconnector:dataplaneinstance",
"url": "http://localhost:8187/control/transfer",
"allowedSourceTypes":["HttpData", "HttpProvision"],
"allowedDestTypes": ["HttpData", "HttpProvision", "HttpProxy"],
"properties": {
"publicApiUrl":"http://localhost:8185/public/"
}
}2.2.3. UsageFor this type of data transmission to work via EDC, both connectors must be implemented and configured for the The procedure for this transmission option differs slightly from the previous prototype. The actual data exchange is not initialized by the request, but this generates a token. This token is passed to the interface defined in the configuration file ( {
"protocol": "ids-multipart",
"assetId": "asset-1",
"contractId": "2:915544f2-8434-47bd-8b78-da351e735ce3",
"dataDestination": {
"type": "HttpProxy"
},
"transferType": {
"contentType": "application/octet-stream",
"isFinite": true
},
"managedResources": false,
"connectorAddress": "http://localhost:8183/api/v1/ids/data",
"connectorId": "consumer"
}This results in the response below to the configured ( {
"id": "74cb8139-2a8c-4c3b-86a6-4e42a795ecd2",
"endpoint": "http://localhost:9195/public/",
"authKey": "Authorization",
"authCode": "eyJhbGciOiJSUzI1NiJ9.eyJkYWQiOiJ7XCJwcm9wZXJ0aWVzXCI6e1wiYXV0aEtleVwiOlwiQXV0aG9yaXphdGlvblwiLFwiYmFzZVVybFwiOlwiaHR0cDpcL1wvbG9jYWxob3N0OjgxODVcL3B1YmxpY1wvXCIsXCJhdXRoQ29kZVwiOlwiZXlKaGJHY2lPaUpTVXpJMU5pSjkuZXlKa1lXUWlPaUo3WENKd2NtOXdaWEowYVdWelhDSTZlMXdpWW1GelpWVnliRndpT2x3aWFIUjBjRHBjTDF3dmJHOWpZV3hvYjNOME9qZ3dNREJjSWl4Y0luQnliM2g1VVhWbGNubFFZWEpoYlhOY0lqcGNJblJ5ZFdWY0lpeGNJbkJ5YjNoNVRXVjBhRzlrWENJNlhDSjBjblZsWENJc1hDSndjbTk0ZVVKdlpIbGNJanBjSW5SeWRXVmNJaXhjSW5SNWNHVmNJanBjSWtoMGRIQkVZWFJoWENJc1hDSndjbTk0ZVZCaGRHaGNJanBjSW5SeWRXVmNJbjE5SWl3aVpYaHdJam94TmpZNE5EVTJOVGcxTENKamFXUWlPaUl5T2pFMFlUUTNNVGN6TFdObFpHTXROREZoTUMwNU5UVTNMV1l4Tm1KbVpHUm1OV05oTVNKOS52c0J3NV9iVnVLVDFkT2ZYMXZXQzZ5MWJkTG1NaG9nRWhmaWh4ckJKdElRcE94VzZ1ZVJwUUZSZy1ndWQ3R1pEX3hhYmhrYTJLUEZLcHN4ZVB2ajJjSmtERFhiamJYM3U4SXhwY2l4MFkzSUd2QW5ZUEFfWFpsWkdSSDRvbHJxaFd2T253UG9sVWNpb0t1d2JLM1d4dWp6QjhmLXBTcVpCWElIQmlXUFpmZGtlZzktVVZZTTlmT1dWS3BpeFE2Y1Z0enZWbzFWWmowUUxvOWVSMWFVU0RzZVpQU241cEhWUFAwcmNzUGZDSFp1RWFSR1ZxcEplbzMzaUZ5eGI2T2ZCcVlMVXVjV244ZEd6WjdKYWxzZFJvMkJiQjVieXRwbzc0VFdxbC0xUDVZVFRVb25sWWNZR1BCd0pNMUEyRF9PM1pOby1MVGQwRjdQUlJOYTJkQ2RiNmdcIixcInByb3h5TWV0aG9kXCI6XCJ0cnVlXCIsXCJwcm94eVF1ZXJ5UGFyYW1zXCI6XCJ0cnVlXCIsXCJwcm94eUJvZHlcIjpcInRydWVcIixcInR5cGVcIjpcIkh0dHBEYXRhXCIsXCJwcm94eVBhdGhcIjpcInRydWVcIn19IiwiZXhwIjoxNjY4NDU2NTg1LCJjaWQiOiIyOjE0YTQ3MTczLWNlZGMtNDFhMC05NTU3LWYxNmJmZGRmNWNhMSJ9.R8h6wKbe2S8QXuPqXzTusYGgCJBGryquX6wBnV9zwB28sCTbHOFI_kxnyxNxndzkDK6jms4QLOIt1OlA72KtnItzT1lQe8rjBLkY1RMcl3c5R8xYF1mZ6Qsdswxdagc2Xsay2ZZTQcotFua2Uwy7wsDCa2tc99w30occPM_I4_6gFShA2iv0xjr9lYIyuV_V-H9EQQ_dV5mqaf7T5gxNKYAxVXD2WiQGpSkjfwDDpbYzEnqBKTMN65dSa7bonQ0mmB7ULd_FBJyl3mmS5Nn20abZZ073jZM2u4R_3Pq_8aN-HwTKhDPwmgeFiYbJsag36c8vnGCBCON1F9XcsqMMKg",
"properties": {
"cid": "2:14a47173-cedc-41a0-9557-f16bfddf5ca1"
}
}A request to the public API of the consumer with a token like the above would look like the cURL request below. curl -X POST \
'http://localhost:9195/public/' \
--header 'Accept: */*' \
--header 'User-Agent: Thunder Client (https://www.thunderclient.com)' \
--header 'Content-Type: application/json' \
--header 'Authorization: eyJhbGciOiJSUzI1NiJ9.eyJkYWQiOiJ7XCJwcm9wZXJ0aWVzXCI6e1wiYXV0aEtleVwiOlwiQXV0aG9yaXphdGlvblwiLFwiYmFzZVVybFwiOlwiaHR0cDpcL1wvbG9jYWxob3N0OjgxODVcL3B1YmxpY1wvXCIsXCJhdXRoQ29kZVwiOlwiZXlKaGJHY2lPaUpTVXpJMU5pSjkuZXlKa1lXUWlPaUo3WENKd2NtOXdaWEowYVdWelhDSTZlMXdpWW1GelpWVnliRndpT2x3aWFIUjBjRHBjTDF3dmJHOWpZV3hvYjNOME9qZ3dNREJjSWl4Y0luQnliM2g1VVhWbGNubFFZWEpoYlhOY0lqcGNJblJ5ZFdWY0lpeGNJbkJ5YjNoNVRXVjBhRzlrWENJNlhDSjBjblZsWENJc1hDSndjbTk0ZVVKdlpIbGNJanBjSW5SeWRXVmNJaXhjSW5SNWNHVmNJanBjSWtoMGRIQkVZWFJoWENJc1hDSndjbTk0ZVZCaGRHaGNJanBjSW5SeWRXVmNJbjE5SWl3aVpYaHdJam94TmpZNE5EVTJOVGcxTENKamFXUWlPaUl5T2pFMFlUUTNNVGN6TFdObFpHTXROREZoTUMwNU5UVTNMV1l4Tm1KbVpHUm1OV05oTVNKOS52c0J3NV9iVnVLVDFkT2ZYMXZXQzZ5MWJkTG1NaG9nRWhmaWh4ckJKdElRcE94VzZ1ZVJwUUZSZy1ndWQ3R1pEX3hhYmhrYTJLUEZLcHN4ZVB2ajJjSmtERFhiamJYM3U4SXhwY2l4MFkzSUd2QW5ZUEFfWFpsWkdSSDRvbHJxaFd2T253UG9sVWNpb0t1d2JLM1d4dWp6QjhmLXBTcVpCWElIQmlXUFpmZGtlZzktVVZZTTlmT1dWS3BpeFE2Y1Z0enZWbzFWWmowUUxvOWVSMWFVU0RzZVpQU241cEhWUFAwcmNzUGZDSFp1RWFSR1ZxcEplbzMzaUZ5eGI2T2ZCcVlMVXVjV244ZEd6WjdKYWxzZFJvMkJiQjVieXRwbzc0VFdxbC0xUDVZVFRVb25sWWNZR1BCd0pNMUEyRF9PM1pOby1MVGQwRjdQUlJOYTJkQ2RiNmdcIixcInByb3h5TWV0aG9kXCI6XCJ0cnVlXCIsXCJwcm94eVF1ZXJ5UGFyYW1zXCI6XCJ0cnVlXCIsXCJwcm94eUJvZHlcIjpcInRydWVcIixcInR5cGVcIjpcIkh0dHBEYXRhXCIsXCJwcm94eVBhdGhcIjpcInRydWVcIn19IiwiZXhwIjoxNjY4NDU2NTg1LCJjaWQiOiIyOjE0YTQ3MTczLWNlZGMtNDFhMC05NTU3LWYxNmJmZGRmNWNhMSJ9.R8h6wKbe2S8QXuPqXzTusYGgCJBGryquX6wBnV9zwB28sCTbHOFI_kxnyxNxndzkDK6jms4QLOIt1OlA72KtnItzT1lQe8rjBLkY1RMcl3c5R8xYF1mZ6Qsdswxdagc2Xsay2ZZTQcotFua2Uwy7wsDCa2tc99w30occPM_I4_6gFShA2iv0xjr9lYIyuV_V-H9EQQ_dV5mqaf7T5gxNKYAxVXD2WiQGpSkjfwDDpbYzEnqBKTMN65dSa7bonQ0mmB7ULd_FBJyl3mmS5Nn20abZZ073jZM2u4R_3Pq_8aN-HwTKhDPwmgeFiYbJsag36c8vnGCBCON1F9XcsqMMKg' \
--header 'X-Api-Key: password' \
--data-raw '{"example":"test123"}'The token is then validated by the consumer, and if all is well, the request is forwarded to the provider's public API, which validates it again and then queries the data source with the transmitted data. The data source's response is then returned via the connectors' public APIs. The following graphic shows the process of such a request.
It should also be noted that the token is currently only valid for 10 minutes; this should be able to be adjusted in the future, but is currently an active issue. When the token expires, a new token must be requested from the control plane, as at the beginning. 2.2.4. ImpressionDuring development, it was often not clear which extension was missing, as the desired result was not achieved, but no error message was thrown that a package was missing. An example of this is the This implementation is good for frequently requesting an asset with different information. The process for obtaining a new token is currently not automated and must be requested again and again from the control plane if the token is only valid for a very short time. This could negate the benefit that should be gained from this type of data exchange. Furthermore, this implementation has the advantage that the provider does not have to be explicitly informed about the data sink in which the data is to be stored, since communication only takes place via the public API. 3. ObservationDuring the development of the connectors, things came up that were not unique to one implementation. These observations are discussed in more detail in this section. 3.1. Information for provider data sourceNeither implementation currently provides any automatic information about the basis on which the query of the data source came about. To check whether information can be used to match a request to a contract, a server was implemented with FastAPI and the request was printed to the console. The following code snippet shows what information was printed to the console. print( await request.body())
print( request.headers.values())
print( request.query_params.values())
print( request.path_params.values())And that was what was shown in the console. b'{"example":"test123"}'
['application/json', 'chunked', 'localhost:8000', 'Keep-Alive', 'gzip', 'okhttp/4.10.0']
dict_values([])
dict_values([])As can be seen, there is currently no property sent by the implementation of the EDC that allows the request to be associated with a contract. The only option currently is that this information must be provided manually when making the request. However, this can lead to errors and it would therefore be more secure if the information already contained in the request is also automatically transmitted to the data source. 4. SummaryThe Eclipse Dataspace Connector (EDC) already provides the necessary interfaces to transfer data to the interfaces of a provider to process the necessary tasks. For this purpose, the EDC provides However, work needs to be done to securely track who is accessing resources and on what basis. ContactIf you have any questions regarding the GAIA-X4KI project, please contact Dr. David Mischnick (david.mischnick@dlr.de) If you have any questions regarding the implementation, please write me (Henrik.Lange@dlr.de) an email or contact me at https://www.linkedin.com/in/henriklangebs/ |
Beta Was this translation helpful? Give feedback.
-
|
a sample regarding the consumer pull had already been created: https://github.com/eclipse-edc/Samples/tree/main/transfer/transfer-06-consumer-pull-http |
Beta Was this translation helpful? Give feedback.
-
|
I know the sample transfer-06-consumer-pull-http, unfortunately, it does not contain any custom query data. |
Beta Was this translation helpful? Give feedback.
-
|
it can easily modified accordingly, could be a good opportunity for a contribution ;) |
Beta Was this translation helpful? Give feedback.
-
|
Might be easy for you, not for a newbie. I do not know where to start. Could you give a hint? |
Beta Was this translation helpful? Give feedback.
-
|
@Madeo19 you could starting opening an issue, that's also considered as "contribution" |
Beta Was this translation helpful? Give feedback.
-
Now if we want to use the custom request for our resource. How |
Beta Was this translation helpful? Give feedback.
-
|
If you want to specify the query string that need to be statically passed to the data source on the provider side, you can set the |
Beta Was this translation helpful? Give feedback.
-
Thank you for explaining that. I just wanted to ask an additional question. I have a header parameter that I want to pass in this I have tried to pass using the information on the documentation. Kindly let me know if that is the correct format. |
Beta Was this translation helpful? Give feedback.
-
|
Headers cannot be passed mainly for security reasons, if you want to have some specific headers passed, you could register a custom |
Beta Was this translation helpful? Give feedback.
-
Hello everybody,
I invested last week some time to implement the Data Plane Transfer Sync prototypically on my development machine for the project Gaia-X4KI. I thought it would be quite nice, if I write down what steps were necessary and what problems I met to have a running example.
My question is where (which folder or repo) and in what format (pdf, md) should I publish it?
Best regards,
Henrik
Beta Was this translation helpful? Give feedback.
All reactions