Provide Workflow-API with EDC Provisioning to manage Compute Jobs in Private Cloud (code2data, code2compute)

[reisman234]

In the latest Q&A-Session, I introduced our plans to use an edc connector as a Gatekeeper to our compute infrastructure and trigger Jobs and return the produced result to the Consumer. Our first attempt was to use a http-data-asset, which will be trigger the backend with an incoming transfer request, and process a specific execution workflow. For short jobs this workflow succeeded, but for longer jobs timeouts occur in the edc and produces errors in the backend.

To face this problem, I got an introduction the In the Q&A-Session about the http-provision extension, which faces such long-running workflows to prepare data before the actual transfer.

Because of the missing documentation of that extension (only system-tests used as resource), it required some time to implement a connector alongside a proper backend. I have here a small summary which indicate all points that must be considered. I would love to see that committed/added in the main repo :)

I have now a running prototype which is able to deploy jobs into a kubernetes system and wait for that termination. With the callback, the edc can be informed of the finished process and provide a dataAddress from which the data can be loaded and transferred to the consumer.

IMHO, the current implementation of the http-provisioning extension is very limited and very strict. Please correct me if I'm wrong with the following points.

1. only static configuration of http-provisoning through the properties file of a connector. I think registering backends via API call would be a nice feature, but I also think it could be compensated by allowing forwarding the asset properties (see 2.).
2. With the implemented test case in system-tests for provisioning
   

   Connector/system-tests/e2e-transfer-test/runner/src/test/java/org/eclipse/edc/test/e2e/AbstractEndToEndTransfer.java

   Lines 84 to 89 in bcf7c87

   	createResourcesOnProvider(assetId, noConstraintPolicy(), UUID.randomUUID().toString(), Map.of(
   	"name", "transfer-test",
   	"baseUrl", PROVIDER.backendService() + "/api/provider/data",
   	"type", "HttpProvision",
   	"proxyQueryParams", "true"
   	));

   
   I would expect that I can modify the baseUrl or provide queryParams, but from the real implementation of a HttpProviderProvisioner.java, only the "type", "HttpProvision" property is considered to call the proper backend-api, which is configured in the properties file.
   

   Connector/extensions/control-plane/provision/provision-http/src/main/java/org/eclipse/edc/connector/provision/http/impl/HttpProviderProvisioner.java

   Lines 76 to 79 in bcf7c87

   	@Override
   	public boolean canProvision(ResourceDefinition resourceDefinition) {
   	return resourceDefinition instanceof HttpProviderResourceDefinition && dataAddressType.equals(((HttpProviderResourceDefinition) resourceDefinition).getDataAddressType());
   	}

3. At some point, it should also be possible for a user/consumer to send additional data, how it is discussed in How to send additional information via HttpData? #1386

[jimmarino]

I'm having difficulty understanding what the ask is. Can you please briefly outline what you are trying to do in a couple of sentences?

As a rule of thumb, HttpProvisionerRequest does not need to convey additional properties. The assetId, transferProcessId and policy properties already provide enough identifiers for routing requests to backends and correlating additional data.

[reisman234]

In general, we want to connect our on-premise compute infrastructure (currently, it is just a Kubernetes cluster) with the edc to provide compute resources to a dataspace for different services to run them e.g. a tensorflow, PyTorch or something else but for now it is just Carla and ROS as a data-generator.

At the moment, we're just using the edc as it is with the extensions you provide.

With the http-provision extension and our basic middlelayer-api, I was able to implement a first prototype, which is able to schedule a static data-generator job in Kubernetes, and after it's execution the generated file will be transferred to the data-consumer.

Now, what I want to do, is to provide services assets, where a service-consumer can or is required to add additional input. E.g. For a PyTorch service, a consumer wants to choose its own scripts or training data, or use that from other participants from the dataspace. This "Additional Information was also discussed in #1386 but not for http-provision"

[jimmarino]

Based on the description you provided, I would consider revising the approach. It appears you are trying to include client "artifacts" (e.g., scripts, metadata, code, containers, etc.) as part of the connector message exchanges. This will scale poorly and will likely result in a very complicated implementation.

Here's a brief outline of how I would look at this problem. Take it with a grain of salt since I don't have visibility into your specific requirements.

As a design principle, client "artifacts" should never be associated with the control-plane flow; instead, they should be contributed out-of-band in the same way the control-plane/data-plane is architected. In fact, client artifacts are part of the data-plane flow. I would therefore create a custom data-plane:

1. The client submits a request for an "asset."
2. The provider uses the provisioner to create some type of object storage (I'm using object storage for this example, another technology could be used such as a specialized K8S cluster with an HTTPS ingress point). This will be your custom data-plane.
3. When provisioning is complete, notify the client and provide credentials for uploading its "artifacts"
4. The client receives the credentials and uploads its data.
5. The object storage (or another ingress point) has a trigger to start processing after the last artifact is received (this could be a marker file)
6. The results can either be delivered to the client or the client can be notified when processing is complete.

This approach does not require specialized processing or tunneling artifacts in the control plane. It will also scale much better since the reliability and performance of object storage (or another system) can be leveraged to transfer very large artifacts.

This also has the advantage that the data-plane could be provisioned on a trusted third party so that the provider never has access to the client's artifacts. In this case, the provision step could call the API of a third party.

[reisman234]

Thank you for picturing your view to this compute-/server-provider use case! To be sure that I did you understand right, I have created a concept with some parties.

• compute-/service-provider
• data-provider
• service-consumer

The numbering of the workflow faces almost your points, but to have them clear, here is my explanation:

1. request asset
2. trigger provision
   2.1 provision of storage
   2.2 share credentials
3. http-backend gets information and triggers file transfers from data-providers or uploads files from own storage
4. Negotiation of asset from third party/dataspace participant
   4.1 transfer file
5. Trigger to start processing
6. execute and observe service/job
   • state of executions (callback or polled by data-consumer)

note: provisioner in this picture could be our middlelayer-api, which I mentioned above, to enable point 5 and 6.

[jimmarino]

This scenario is different than the one we discussed (or I misunderstood the requirements):

1. This diagram introduces another party (the compute/service provider). All EDC-based data sharing is by design bipartite, hence only between two connectors. The design should be based on a two-party contract agreement (this is very important).

2. The provisioner is owned by the data provider. The infrastructure owned by the compute/service provider should be opaque to the data provider and consumer. The data provider provisioner should call some API. Also, no external systems should connect directly to the provisioner for security reasons since it will need elevated write privileges with backend infrastructure. The provisioner should either provision compute infrastructure owned by the provider or call an external API owned by a third-party provider. Limited write credentials should be obtained and passed back to the consumer.

3. The consumer uploads its artifacts using the limited write credentials, and data processing is started when the last artifact is received.

Importantly, the EDC data plane is not needed in this scenario.

[reisman234]

Sorry for the confusion, I was thinking a little ahead in the scenario I created. The following picture reflects more the post above, and has only two participants.

1. request (service-)asset
2. trigger provision
   1. provision consumer exclusive backend-api & configure proxy
   2. share access credentials
3. upload required service-input
4. Trigger processing
5. execute and observe service/job
   • state of executions (callback or polled by data-consumer)

[jimmarino]

At a high level, that seems reasonable, but with the caveat that I don't know the requirement details. Also, line 2.2 should not exist as connectors only interact with one another (the credentials should be part of the transfer protocol).

[reisman234]

Thank you for your positive opinion!

Line 2.2 shows the flow of the created result by the provisioner (credentials). So after the provisioner has its jobs finished, it uses the callback with the information for the provider-connector where it can retrieve the result and send it to the DateDestination (Line 2.2). This is how I understand the provider-push behavior. The DataDestination was specified in the body of the /transferprocess call to the consumer. Thus, as far as I know, there is no exchange of result data between the connectors.

This is the call:

curl --location --request POST 'http://192.168.205.20:9192/api/v1/management/transferprocess' \
--header 'x-api-key: password' \
--header 'Content-Type: application/json' \
--data-raw '{
  "protocol": "ids-multipart",
  "assetId": "demo-asset-1",
  "contractId": "demo-con-def-1:367ad08c-03ad-4d8c-8254-070d00dbae41",
"dataDestination": {
    "properties": {
        "baseUrl":"http://proto-backend:8001/large/",
        "type": "HttpData"
    }
  },
  "transferType": {
    "contentType": "application/octet-stream",
    "isFinite": true
  },
  "managedResources": false,
  "connectorAddress": "http://192.168.205.10:8282/api/v1/ids/data",
  "connectorId": "urn:connector:edc"
}

[ma3u]

@reisman234 : I also created a related discussion thread for compute-to-data capabilities based on the functional requirements of the new IDSA rulebook v2. I would suggest to work together.