-
Notifications
You must be signed in to change notification settings - Fork 295
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Streaming RPC #16
Comments
Why do you believe a streaming RPC is necessary? What use case would it solve? And how would you expect it to look like? |
A typical use-case is retrieving 100s or 1000s of records of data from a database using SQL, in a loop. We may not know how many rows are going to be retrieved nor we would like to store all the records in memory and do the marshalling of the whole data, which may not be memory efficient. It would be desirable to retrieve each row and marshall it out immediately in a loop, to emit data as a continuous stream While data can be streamed out in a loop, provision to emit a header at the beginning and a footer at the end of stream is desirable. Example: |
This was why I asked. You can already do this. So I'd have to see a real use-case that shows that this isn't possible. In some C++ libraries, they have different requirements, because they try to read the data, store it into a memory container (ie byte array, etc), and they decode that. So they need a separate streaming API, and separate ASYNC api, etc. However, Go library allows you decode directly from any inputstream (anything implementing io.Reader), and async is implemented under-the-hood and exposed via a sync API. In theory, you can have a stream with 200 entities encoded in sequence, and create an encoder, and keep decoding from the stream until the stream is finished. That's one reason why none of the Go encoding libraries (JSON, Gob, Msgpack, etc) have a separate streaming API, or net/rpc has a streaming API. If you look at the msgpack rpc spec, there's no streaming API built into the spec. I don't see a way to even build it. I don't see a clear API that is needed to enable a use-case. If you still think this is necessary, please provide a use-case which is not currently supported by the current API, and what API you believe would be necessary to support it. That will help me see what you are alluding to. |
Another use-case is to receive a request over RPC to transmit data (of file(s), which could be 1 or 2 GiB size). Obviously the data can not be sent in one response packet. So, we end up opening the network connection directly and pump out the contents. While doing so, RPC package is expected not to interfere, which is not perfect. Or we end up avoiding RPC in such scenarios. For RPC, the receiver methods can possibly take io.Reader/Writer or chan as input argument |
Hi @tejorupan it seems your comment was cut and didn't transmit completely. Can you retype it? I would love to see where you were going with it. Thanks. |
Another use-case is to receive a request over RPC to transmit data (of file(s), which could be 1 or 2 GiB size). Obviously the data can not be sent in one response packet. So, we end up opening the network connection directly and pump out the contents. While doing so, RPC package is expected not to interfere, which is not perfect. Or we end up avoiding RPC in such scenarios. For RPC, the receiver methods can possibly take io.Reader/Writer or chan as input argument, to deal with the streaming. The exact naming of API we will get into later, once you are convinced. |
A "suggested" naming of the API and how it would be used to solve a use-case more elegantly than the currently provided constructs will help make your case. My RPC support just provides a codec binding for binc/msgpack that integrates with net/rpc. There is no way to do anything that is not natively supported by net/rpc. Beyond that, the use-case you provided involves bundling RPC communication and a file for download on the same stream. I don't think that would be a good architecture for solving this problem. Most folks that request streaming APIs for RPC, etc typically want the ability to stream an array of results, processing them one at a time instead of waiting for the whole array to be populated first. The RPC functionality currently doesn't allow that. And other codecs with non-dependent RPC support (e.g. avro, thrift) have traditionally had a very hard time justifying the need, designing an API, and implementing support, for streaming RPC. See: |
I did some more research to find out about any thought put into streaming RPC in the net/rpc library. See: The summary of both is that streaming support for net/rpc was considered, but Rob thought it is very tricky to get right and hasn't seen it done right before. No support has been added. I think you should consider opening up an issue on the Go issue tracker, because that is where streaming support should be. My codec library just creates a codec for binc/msgpack that integrates with net/rpc. The full usage model is controlled from the net/rpc package. |
RawExt works like encoding/json.RawMessage. It represents raw unprocessed extension data. Users can have a RawExt in their struct, and the codec will just put the extension data (Tag byte and []byte data) as-is for potential further processing later. RawExt is also returned if decoding into a nil interface{} and no extension function has been registered. RpcCodecBuffered interface allows access to the Buffered Reader and Writer used internally by the rpc codec. This accomodates use-cases where the connection should be used by rpc and non-rpc functions, e.g. streaming a file after sending an rpc response. Both GoRpc and MsgpackSpecRpc return codecs that implement RpcCodecBuffered. In addition, some minor re-org and fixes were done: - decoding numbers into a nil interface{} for Binc always uses 64-bit values (int64, uint64) Updates issue #16 .
The latest commit 4f88b73 contains some code to alleviate the problem. You can build a custom streaming It's not a full RPC streaming solution, but allows you construct a custom solution for your specific use-case. |
You may find the following two interesting from the streaming RPC perspective |
I looked at the vitess code. They forked net/rpc and added streaming support to it. That's not the job of the codec library. Its the job of the layer that calls the codec library. You may want to file an issue with the go net/RPC library to push them to add streaming support. |
Thanks for your feedback. |
I opened a new issue: 6569 ( https://code.google.com/p/go/issues/detail?id=6569 ) for this. |
Please provide a mechanism to do Steaming RPC as an extension.
I know net/rpc lacks that feature currently.
The text was updated successfully, but these errors were encountered: