Appropriate asset size for HTTP dataplane

[reisman234]

What asset size can the HTTP data plane handle in one transfer?

I currently try to transfer a binary file with a size of 1G, but after the file is in-memory, the connector throws the following exception.

test-connector    | SEVERE 2022-12-12T14:29:28.240849382 Error processing data transfer request: 1c9c0bd5-818f-4e9e-9c3c-26141dc8c35d                                                          
test-connector    | java.lang.NegativeArraySizeException: -1958614768                                                                                                                          
test-connector    |     at java.base/java.lang.String.<init>(String.java:569)                                                                                                                  
test-connector    |     at java.base/java.lang.String.<init>(String.java:1382)                                                                                                                 
test-connector    |     at okio.Buffer.readString(Buffer.kt:313)                                                                                                                               
test-connector    |     at okio.Buffer.readString(Buffer.kt:302)   
test-connector    |     at okio.RealBufferedSource.readString(RealBufferedSource.kt:96)
test-connector    |     at okhttp3.ResponseBody.string(ResponseBody.kt:187)                                                                                                                    
test-connector    |     at org.eclipse.dataspaceconnector.dataplane.http.pipeline.HttpDataSource.getPart(HttpDataSource.java:47)
test-connector    |     at org.eclipse.dataspaceconnector.dataplane.http.pipeline.HttpDataSource.openPartStream(HttpDataSource.java:41)
test-connector    |     at org.eclipse.dataspaceconnector.dataplane.spi.pipeline.ParallelSink.transfer(ParallelSink.java:50)
test-connector    |     at org.eclipse.dataspaceconnector.dataplane.framework.pipeline.PipelineServiceImpl.transfer(PipelineServiceImpl.java:89)
test-connector    |     at org.eclipse.dataspaceconnector.dataplane.framework.pipeline.PipelineServiceTransferServiceImpl.transfer(PipelineServiceTransferServiceImpl.java:50)
test-connector    |     at org.eclipse.dataspaceconnector.dataplane.framework.manager.DataPlaneManagerImpl.processDataFlowRequest(DataPlaneManagerImpl.java:154)
test-connector    |     at org.eclipse.dataspaceconnector.spi.telemetry.Telemetry.lambda$contextPropagationMiddleware$1(Telemetry.java:87)
test-connector    |     at org.eclipse.dataspaceconnector.dataplane.framework.manager.DataPlaneManagerImpl.run(DataPlaneManagerImpl.java:129)
test-connector    |     at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
test-connector    |     at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
test-connector    |     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
test-connector    |     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
test-connector    |     at java.base/java.lang.Thread.run(Thread.java:833)

[jimmarino]

That depends on a number of things, including JVM configuration, available memory, etc so it is impossible to give a specific answer.

However, as guidance:

1. The HTTP source and sink implementation are designed for small (e.g. kilobyte) messages
2. If you require large message transfers, use a more appropriate protocol such as S3 or FTP (that latter would be a good addition)
3. Don't run the data plane in the same instance as the control plane - there's plenty of documentation explaining why

[reisman234]

Thanks for the first guidance, I will come back to that!