Recieved org.eclipse.dataspaceconnector.spi.EdcException during Data Transfer

[Its-suLav-D]

Hey there, when I initiate a data transfer request, I get this error from my Data Plane `

"org.eclipse.dataspaceconnector.spi.EdcException: Received code transferring HTTP data for request a5e4480c-2685-41b4-b5e1-83f6734056da: 500 - . {\"errors\":[\"Error processing data transfer request\"]}"

Here is my transfer_request_payload

TRANSFER_PAYLOAD = {
    "id": DATA_TRANSFER_ID,
    "assetId": EDC_ASSET,
    "contractId": AGR_ID,
    "connectorId": "urn:connector:provider-control-plane",
    "connectorAddress": TEST_PROVIDER + '/api/v1/ids/data',
    "protocol": NEG_PROTOCOL,
    "dataDestination": {
        "type": "HttpProxy"

    },
    "managedResources": False,
        "transferType": {
        "contentType": "application/json",
        "isFinite": True
    } 
}

Trace

"log":"[2022-12-01 14:54:52] [SEVERE ] Error processing data transfer request","stream":"stderr",
log: "org.eclipse.dataspaceconnector.spi.EdcException: Received code transferring HTTP data for request 9a4a5af8-dafc-417c-a987-cfb575e8ccba: 500 - . {\"errors\":[\"Error processing data transfer request\"]}"
log: "\tat org.eclipse.dataspaceconnector.dataplane.http.pipeline.HttpDataSource.getPart(HttpDataSource.java:54)"
log: "\tat org.eclipse.dataspaceconnector.dataplane.http.pipeline.HttpDataSource.openPartStream(HttpDataSource.java:41)"
log: "\tat org.eclipse.dataspaceconnector.dataplane.spi.pipeline.OutputStreamDataSink.transfer(OutputStreamDataSink.java:46)"
log: \tat org.eclipse.dataspaceconnector.dataplane.framework.pipeline.PipelineServiceImpl.transfer(PipelineServiceImpl.java:109)
log: \tat org.eclipse.dataspaceconnector.dataplane.framework.manager.DataPlaneManagerImpl.transfer(DataPlaneManagerImpl.java:112)
log: \tat org.eclipse.dataspaceconnector.dataplane.api.controller.DataPlanePublicApiController.handle(DataPlanePublicApiController.java:146)
log: \tat org.eclipse.dataspaceconnector.dataplane.api.controller.DataPlanePublicApiController.get(DataPlanePublicApiController.java:72)
log: "\tat jdk.internal.reflect.GeneratedMethodAccessor33.invoke(Unknown Source)
log: \tat java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43), "stream": stderr
log: \tat java.base/java.lang.reflect.Method.invoke(Method.java:566)
log: \tat org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
log: \tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:134)
log: "\tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:177)"
log: tat org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$VoidOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:159
log: tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:81)
log: \tat org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478)"
log: \tat org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400)
log: \tat org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
log: \tat org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:255)
log: log":"\tat org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)","stream":"stderr"
log":"\tat org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)","stream":"stderr",
"log":"\tat org.glassfish.jersey.internal.Errors.process(Errors.java:274)","stream":"stderr"
"log":"\tat org.glassfish.jersey.internal.Errors.process(Errors.java:244)","stream":"stderr"
"log":"\tat org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)","stream":"stderr"
"log":"\tat org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234)","stream":"stderr"
"log":"\tat org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684)","stream":"stderr"
log":"\tat org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)","stream":"stderr"
"log":"\tat org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)","stream":"stderr"
"log":"\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:358)","stream":"stderr"
"log":"\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:311)","stream":"stderr"
"log":"\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)","stream":"stderr"
"log":"\tat org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764)","stream":"stderr"
"log":"\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:529)","stream":"stderr"
"log":"\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)","stream":"stderr"
"log":"\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1380)","stream":"stderr"
"log":"\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)","stream":"stderr"
"log":"\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)","stream":"stderr"
"log":"\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)","stream":"stderr"
"log":"\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1302)","stream":"stderr"
"log":"\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)","stream":"stderr",
"log":"\tat org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:192)","stream":"stderr"
"log":"\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)","stream":"stderr",
"log":"\tat org.eclipse.jetty.server.Server.handle(Server.java:563)","stream":"stderr",
"log":"\tat org.eclipse.jetty.server.HttpChannel.lambda$handle$0(HttpChannel.java:505)","stream":"stderr"
"log":"\tat org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:762)","stream":"stderr",
"log":"\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:497)","stream":"stderr",
"log":"\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:282)","stream":"stderr",
"log":"\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)","stream":"stderr",
"log":"\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)","stream":"stderr",
"log":"\tat org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)","stream":"stderr"
"log":"\tat org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421)","stream":"stderr"
"log":"\tat org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390)","stream":"stderr"
"log":"\tat org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277)","stream":"stderr"
"log":"\tat org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.lambda$new$0(AdaptiveExecutionStrategy.java:139)","stream":"stderr"
log":"\tat org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411)","stream":"stderr"
"log":"\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:933)","stream":"stderr",
"log":"\tat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1077)","stream":"stderr",
"log":"\tat java.base/java.lang.Thread.run(Thread.java:829)","stream":"stderr",

The weird part is its irregular. Sometimes, I can fetch the data and see the content but sometimes I get this exception. Not sure why it is behaving like this. It would be super great if anyone could help me understand this behavior. Thanks a lot guys.

[jimmarino]

It looks like the source HTTP endpoint for the data being transferred you are using is returning code 500 to the data plane instance. Best bet is to look there for the problem.