Streaming incompatibility with Dubbo

[DatPhanTien]

• I have searched the issues of this repository and believe that this is not a duplicate.
• I have checked the FAQ of this repository and believe that this is not a duplicate.

Environment

• Dubbo version: 2.7.1
• Operating System version: Centos 7.4
• Java version: 1.8.202

Steps to reproduce this issue

We are having an interface that scan records from HBase and return a stream of records.
Stream<CustomRecord> scan(String startRow, String stopRow);

We have learnt that the Duddo protocol does not support Stream, but it can handle the stream of objects when it is converted into java.io.InputStream.

However, when using ByteArrayInputStream as the replacement of the Stream<CustomRecord> for our interface, we got an OutOfMemory exception, since the stream grows too large and exceeds the heap size (ByteArrayInputStream keeps the byte[] in memory).

Our question

Our question is: What would be the best approach for this issue? Do you have an alternative InputStream implementation for this use case? Or do you know of any already-used solution in Dubbo community?

[DatPhanTien]

One of the solution that we can think of is to write down the stream of objects to a local file. Then the interface returns a FileInputStream of that local file.

However, this solution really kills the beauty of java.util.stream.

We bet that the Dubbo community, with years of experience working with remote method invocation, would have a more efficient and adequate solution for this issue.

[chickenlj]

https://github.com/apache/incubator-dubbo/tree/3.x-dev/dubbo-rpc/dubbo-rpc-rsocket

I think you need a Stream pipeline between the Client and the Server. This feature in 3.0.0 may meet your requirement.

[DatPhanTien]

Dear @chickenlj, Thanks you for your exciting information.
After quickly checking the branch, I have several questions:

1. Do you have a short documentation about Dubbo 3.x? We just wonder if this new release it an API breaker with major difference compared to 2.7 APIs or not.
2. Do you have any documentation regarding this new protocol? Or they are not available yet?
3. For Stream pipeline, AFAIU the serialization of the stream is handle by the protocol. Is that right?
4. Do you know when will be the expectation of the the 3.x Dubbo release?

[uglycow]

the stream support is basing on rsocket , you can find doc for rsocket at http://rsocket.io/

for now , dubbo-rpc-rsocket is just draft code for demo.

[DatPhanTien]

@uglycow, thanks for your information.
Regarding the dubbo-rpc-rsocket, do you know if there exists any roadmap for the release? As I understand, this module will be added with the release of version 3.x.

[icodening]

The triple protocol in dubbo3 support bidirectional streaming, which may meet your needs