Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature Request: Add support for In-Process transport #906

Open
Merovius opened this issue Sep 22, 2016 · 41 comments

Comments

@Merovius
Copy link

commented Sep 22, 2016

Say I want, for example

  • Developing a frontend and a backend as separate logical services, but deploy them in one binary. For traditional systems, having both run in a single binary will just make deployment, installation and running it much simpler. But I also want to have the option to easily split them apart for more complex usecases.
  • Use an embedded database (like leveldb), but I want, for debugging, to be able to peek and poke at it too. I can't just open it twice (it uses locking to prevent that). So instead I want to expose it as a grpc service for debugging. To reuse all my application-level code to interpret the raw bytes in the db, however, I also want that to only use the exposed grpc service. The service can then just open a leveldb, wrap it in a service and use that to talk to the database, whereas the debugger can connect to the service and use that client.

So, I want to be able to implement a FooServer and then connect to it from the same process. Of course, I could just listen on localhost and connect to that or something like that, but then I'd pay the penalty of serializing and deserializing everything and running the bytes through the kernel (which is significant when it's in the path of talking to your database, for example).

Instead, it would be cool if grpc allowed me to get a "local" connection, like func LocalPair() (*grpc.Client, *grpc.Server), which doesn't use a network at all and just directly passes the proto.Messages around.

I'd be willing to try to implement that myself, but first I wanted to ask if this is a use case you'd be willing to support.

@menghanl

This comment has been minimized.

Copy link
Contributor

commented Sep 23, 2016

This can be done by providing a custom dialer (WithDialer) to the client and a custom listener to the server.
In this way, you can specify the net.Conn connecting client and server to be a wrapper of something in memory.

@Merovius

This comment has been minimized.

Copy link
Author

commented Sep 23, 2016

Thanks for replying. However, even a loopback-connection will still require the serialization and de-serialization part of all the requests and metadata. That's kind of a waste if we already have the correctly typed things at either end.

So this feature request is specifically about bypassing all of that. I haven't yet found a way to achieve that. For example, even if I were to try to wrap it myself and use a codegen, I can't actually inspect the passed CallOptions (and thus any sent metadata, which makes it kind of a no-starter) due to the way it's set up. So, from what I can tell, support for this would need to come from the grpc package itself.

@iamqizhao

This comment has been minimized.

Copy link
Contributor

commented Sep 26, 2016

To bypass all of these overhead, we need an in-process-transport impl (in addition to http2Client and http2Server transport) in the transport package and the corresponding custom codec, listener, etc.. We did think about this previously but this is not a trivial change and we do not have enough hands covering this now unfortunately.

As an alternative, I think if you can add a wrapper on the top of the grpc generated code to switch on in-process and normal case you should probably achieve this without any changes in grpc library (I have not thought through all the detail and could be wrong though).

@mwitkow

This comment has been minimized.

Copy link
Contributor

commented Oct 25, 2016

This would be really useful for @yugui and grpc-gateway use case.

@hsaliak

This comment has been minimized.

Copy link

commented May 16, 2017

Added an enhancement label to this, but the work is not prioritized at the moment.
To flesh this out further, it may be best to submit a proposal that discusses a few implementation options as a language specific gRFC

@dennisdoomen

This comment has been minimized.

Copy link

commented Jun 27, 2017

In the .NET world, this is something to be expected. OWIN does it really well and allows you to build HTTP-enabled components and host them anywhere. If you host it in-process, all the requests happen completely in-memory.

@jhump

This comment has been minimized.

Copy link
Member

commented Aug 30, 2017

@iamqizhao wrote:

As an alternative, I think if you can add a wrapper on the top of the grpc generated code to switch on in-process and normal case you should probably achieve this without any changes in grpc library

Unfortunately, this is not necessarily the case. The main issue for an in-process implementation is that the CallOption stuff is totally opaque. So, even were there a way to communicate headers and trailers in-process, there is no way code can actually interact with these options -- at least not without forking grpc-go and adding said code to that package so it can interact with these un-exported types.

I brought this up on the mailing list some time ago:
https://groups.google.com/d/msg/grpc-io/NOfh5ESgnyc/RgDJe5g0EgAJ

I'm now re-visisting this issue because I've hit it again with something else I'm trying to do: a client interceptor to do client-side retries that can support "hedging". With hedging, the interceptor may issue a retry before the prior attempt completes -- triggered by a timeout vs. a failure. But header/trailer metadata for only the last/successful attempt should be used for the header and trailer call options. Since these types are totally opaque, it isn't possible for an interceptor to do what it needs to do. And if it just passes along the options, without any changes, to multiple concurrent attempts, it is both non-deterministic as well as unsafe/racy regarding how the client-provided metadata addresses will get set.

@dfawley

This comment has been minimized.

Copy link
Contributor

commented Aug 31, 2017

FWIW, retries and hedging are coming to gRPC-Go natively in a month or two. Relevant gRFC.

Regarding the initial issue, we have since created the bufconn package to at least bypass the network stack (message are still [de]serialized and everything goes through the http2 transport, but this should help with overhead somewhat). Otherwise, the team still doesn't have the bandwidth to take on an in-process transport any time soon.

@jhump

This comment has been minimized.

Copy link
Member

commented Aug 31, 2017

Otherwise, the team still doesn't have the bandwidth to take on an in-process transport any time soon.

It would be nice if the API were at least amenable to a 3rd party library supplying this. Unfortunately, it is currently not due to #1495.

@Random-Liu

This comment has been minimized.

Copy link

commented Mar 2, 2018

Any updates on this?

/cc @stevvooe @crosbymichael I don't think bufconn is enough for us. We can only wrap all the grpc services now.

@jhump

This comment has been minimized.

Copy link
Member

commented Mar 2, 2018

There are still a couple of issues that prevent a 3rd-party package from providing an in-process channel: #1495 and #1802,

However, that hasn't stopped me! Take a look at these:
https://godoc.org/github.com/fullstorydev/grpchan
https://godoc.org/github.com/fullstorydev/grpchan/inprocgrpc

The above issues do represent shortcomings though. They basically mean that call options are ignored by custom channel implementations (which means you can't get response headers and trailers from a unary RPC). And there is a similar issue on the server side: you cannot set response headers or trailers from a unary RPC implementation.

@Random-Liu, I don't really understand why you can't use bufconn though. It is similar enough in concept and construction that if you can't use it, you may not be able to use inprocgrpc either. The main advantage of inprocgrpc is that it doesn't incur serialization/de-serialization overhead or deal with HTTP/2 framing to wire the client up to the server. It uses a channel with a limited buffer size to achieve back-pressure, instead of using flow control windows in the HTTP/2 protocol.

@dfawley dfawley added Status: Help Wanted P2 and removed P3 labels Mar 2, 2018

@dfawley

This comment has been minimized.

Copy link
Contributor

commented Mar 2, 2018

@Random-Liu,

Any updates on this?

We were literally talking about this yesterday, so we haven't forgotten about it. However, it's still not at the top of our priority list. It will probably be another 6 months before we can get to it. Our other priorities right now are channelz, performance, and retry. It would be a fairly meaty project if an outside contributor wanted to take it on, but we would be able to advise.

The main advantage of inprocgrpc is that it doesn't incur serialization/de-serialization overhead

Skipping serialization/deserialization seems dangerous in Go, because proto messages are mutable. This means that if the server modifies the request message, then the client would see the result of that modification. (The same is true for streaming in either direction.) This shouldn't typically be happening, but it's a notable difference between a real server and inprocgrpc.

@jhump

This comment has been minimized.

Copy link
Member

commented Mar 2, 2018

This means that if the server modifies the request message, then the client would see the result of that modification

No, this does not happen. The library avoids serialization/de-serialization, but it does copy.

@Random-Liu

This comment has been minimized.

Copy link

commented Mar 2, 2018

I don't really understand why you can't use bufconn though.

@jhump Because we may also want to get rid of the serialization/de-serialization, :)

With https://godoc.org/github.com/fullstorydev/grpchan/inprocgrpc, can I make the grpc server serve both remote requests and inproc requests at the same time?

@dfawley

This comment has been minimized.

Copy link
Contributor

commented Mar 2, 2018

The library avoids serialization/de-serialization, but it does copy.

Aha, I see now. So you do a proto.Clone() for proto messages, and a best-effort shallow copy otherwise (ref). FWIW, it may be better (if possible) to serialize and deserialize using the configured codec in the fallback case, in order to avoid this potential problem. But I also agree with the comment that says the fallback path should basically never be exercised, as most people use proto with grpc.

@Daniel-B-Smith

This comment has been minimized.

Copy link

commented Jul 30, 2018

Would this unblock gRPC over HTTP 1.1? Really, what we want is grpc-web for golang clients. Please let me know if there is a separate space where I should bring this up.

@dfawley

This comment has been minimized.

Copy link
Contributor

commented Jul 30, 2018

@Daniel-B-Smith yes, I believe it would unblock any efforts to implement that (with the caveat that the core grpc-go team is not committed to implementing it). I'll have an update on the progress of this issue shortly...

@dfawley

This comment has been minimized.

Copy link
Contributor

commented Jul 31, 2018

I am far enough along with a prototype that I'm ready to share some details on this effort. Please feel free to review and provide feedback.

First, some of my starting goals:

  1. Should encapsulate gRPC semantics (hide HTTP/2 semantics and grpc wire protocol semantics).
  2. Transports should be message-based, not byte-based, to facilitate an in-process transport that uses proto.Clone() for efficiency.
    • Note that retry support requires byte-based support from all transports, because we need to cache multiple sent messages, which our streaming API does not make possible without serialization.
  3. Unexpected asynchronous events should be delivered by callbacks, not channels.
    • Channels may be more idiomatic for Go, but require a goroutine to monitor them, which leads to worse performance (in the case of streams) and more complex code (see #2219 which refactors our transport monitoring).
    • Sending and receiving data should remain synchronous to mirror grpc's external API.
  4. Dialers (Client) and Listeners (Server) should return gRPC transports, not net.Conns. How to connect (e.g. net.Dial) is an implementation detail enacapsulated in the transport. Likewise, handshaking happens internally to Dialers and Listeners.

On to the API. Disclaimer: I have PoC-implemented only the client-side of this interface so far. However, a client is fully implemented and passes all tests, so this API is sufficient for all our current needs.

Package structure (under google.golang.org/grpc/):
transport: Definition of types shared between client and server
transport/client: Definition of client-side transport and stream
transport/server: Definition of server-side transport and stream

package transport

// OutgoingMessage is a message to be sent by gRPC.
type OutgoingMessage interface {
	// Marshal marshals to a byte buffer and returns information about the
	// encoding, or an error.  Repeated calls to Marshal must always return the
	// same values.
	Marshal() ([]byte, *MessageInfo, error)
}

// OutgoingMessage is a message to be received by gRPC.
type IncomingMessage interface {
	// Unmarshal unmarshals from the scatter-gather byte buffer given.
	Unmarshal([][]byte, *MessageInfo) error
}

// MessageInfo contains details about how a message is encoded.
type MessageInfo struct {
	// Encoding is the message's content-type encoding.
	Encoding string
	// Compressor is the compressor's name or the empty string if compression
	// is disabled.
	Compressor string
}
package client

// TransportBuilder constructs Transports connected to addresses.
type TransportBuilder interface {
	// Build begins connecting to the address.  It must return a Transport that
	// is ready to accept new streams or an error.
	Build(context.Context, resolver.Address, TransportMonitor, TransportBuildOptions) (Transport, error)

    // TransportBuildOptions, elided, is a struct with an []interface{} for opaque,
    // custom-transport options and possibly declared fields with known-common options.
}

// A Transport is a client-side gRPC transport.  It begins in a
// "half-connected" state where the client may opportunistically start new
// streams by calling NewStream.  Some clients will wait until the
// TransportMonitor's Connected method is called.
type Transport interface {
	// NewStream begins a new Stream on the Transport.  Blocks until sufficient
	// stream quota is available, if applicable.  If the Transport is closed,
	// returns an error.
	NewStream(context.Context, Header) (Stream, error)

	// GracefulClose closes the Transport.  Outstanding and pending Streams
	// created by NewStream continue uninterrupted and this function blocks
	// until the Streams are finished and the transport has closed.
    // Close may be called concurrently.
	GracefulClose()

	// Close closes the Transport.  Outstanding and pending Streams created by
	// NewStream are canceled.
	Close()

	// Info returns information about the transport's current state.
	Info() TransportInfo
}

// TransportInfo contains information about the transport's current state.  All
// information is optional.
type TransportInfo struct {
	// RemoteAddr is the address of the server (typically an IP/port).
	RemoteAddr net.Addr
	IsSecure   bool // if set, WithInsecure is not required and Per-RPC Credentials are allowed.
	AuthInfo   credentials.AuthInfo
}

// A TransportMonitor is a monitor for client-side transports.
type TransportMonitor interface {
	// Connected reports that the Transport is fully connected - i.e. the
	// remote server has confirmed it is a gRPC server.
	//
	// May only be called once.
	Connected()
	// OnError reports that the Transport has closed due to the error provided.
	// Existing streams may or may not continue, but new streams may not be
	// created.  When all existing streams have completed, the Transport will
	// fully close.
	//
	// Once called, no further calls in the TransportMonitor are valid.
	OnError(error)
}

// Stream is a client-side streaming RPC
type Stream interface {
	// SendMsg queues the message m to be sent by the Stream and returns true
	// unless the Stream terminates before m can be queued.  May not wait for m
	// to be sent.  May not be called simultaneously from multiple goroutines.
	SendMsg(m transport.OutgoingMessage, opts StreamSendMsgOptions) bool

	// RecvHeader blocks until the Stream receives the server's header and then
	// returns it.  Returns nil if the Stream terminated without a valid
	// header.  Repeated calls will return the same header.
	RecvHeader() *ServerHeader

	// RecvMsg receives the next message on the Stream into m and returns true
	// unless the Stream terminates before a full message is received.  May not
	// be called simultaneously from multiple goroutines.
	RecvMsg(m transport.IncomingMessage) bool

	// RecvTrailer blocks until the Stream receives the server's trailer and
	// then returns it.  Returns a synthesized trailer containing an
	// appropriate status if the RPC terminates before receiving a trailer from
	// the server.  Repeated calls will return the same trailer.
	//
	// If all messages have not been retrieved from the stream before calling
	// RecvTrailer, subsequent calls to RecvMsg should immediately fail.  This
	// is to prevent the status of the RPC from changing as a result of parsing
	// the messages.
	RecvTrailer() Trailer

	// Cancel unconditionally cancels the RPC.  Any pending SendMsg and RecvMsg
	// calls should become unblocked.  Queued messages may not be sent.  If the
	// stream does not already have a status, the one provided (which must be
	// non-nil) is used.
	Cancel(*status.Status)

	// Info returns information about the stream's current state.
	Info() StreamInfo
}


// ServerHeader contains header data sent by the server.
type ServerHeader struct {
	Metadata metadata.MD
}

// StreamInfo contains information about the stream's current state.
//
// All information is optional.  Fields not supported by a Stream returning
// this struct should be nil.  If a transport does not support all features,
// some grpc features (e.g. transparent retry or grpclb load reporting) may not
// work properly.
type StreamInfo struct {
	// BytesReceived is true iff the client received any data for this stream
	// from the server (e.g. partial header bytes), false if the stream ended
	// without receiving any data, or nil if data may still be received.
	BytesReceived *bool

	// Unprocessed is true if the server has confirmed the application did not
	// process this stream*, false if the server sent a response indicating the
	// application may have processed the stream, or nil if it is uncertain.
	//
	// *: In HTTP/2, this is true if a RST_STREAM with REFUSED_STREAM is
	//    received or if a GOAWAY including this stream's ID is received.
	Unprocessed *bool
}
package server

type TransportListener interface {
	// Accept blocks until a new Stream is created.  grpc calls this until
	// error != nil.
	Accept() (Stream, error)

	// GracefulClose causes the TransportListener to stop accepting new
	// incoming streams, and returns when all outstanding streams have
	// completed.
	GracefulClose()

	// Close immediately closes all outstanding connections and streams.
	Close()
}

type Header struct {
	MethodName string
}

type Trailer struct {
	Status   status.Status
	Metadata metadata.MD
}

type Stream interface {
  // Implementation details TBD; expected to be similar in structure to client.Stream.
}

/cc @johanbrandhorst @jhump

@dfawley

This comment has been minimized.

Copy link
Contributor

commented Jul 31, 2018

An important detail I left out is how to inject these custom transports.

Server-side, the story is simple: instead of giving grpc a net.Conn-based Listener, you would pass it a server.TransportListener that spits out server.Streams instead of net.Conns. That TransportListener would be responsible for managing the underlying connections.

Client-side, it's a bit more complicated: either the resolvers or just the schemes (TBD) will indirectly determine which transport to use. I.e. the "dns" scheme/resolver would output "tcp" addresses that by default use our default transport layer, whereas the "inproc" scheme/resolver (not yet implemented) would output "inproc" addresses that would use our inproc transport. You could register (per-ClientConn or globally, also TBD) a map from scheme/address type to client.TransportBuilder that determines which transport to use for those addresses.

@annismckenzie

This comment has been minimized.

Copy link

commented Jul 31, 2018

What timeframe are you thinking of for implementing this? It looks comprehensive and while it does add another 2 layers for the transports and resolvers I really do like to see this done.

@dfawley

This comment has been minimized.

Copy link
Contributor

commented Jul 31, 2018

@annismckenzie I'm targeting something in the next month or so. I am taking a couple weeks off soon, but hopefully I can wrap things up pretty quickly when I return since I already have a working prototype.

I'm not sure what you mean about adding "another 2 layers". This is basically just drawing a more consistent line between our existing, hard-coded HTTP/2 transport and the grpc package, and making it pluggable.

@annismckenzie

This comment has been minimized.

Copy link

commented Jul 31, 2018

@dfawley

This comment has been minimized.

Copy link
Contributor

commented Aug 3, 2018

@annismckenzie no worries, I am just trying to understand your concerns with the design, if you have any. I'm hopeful a few folks here will at least briefly look it over to let me know if there are any problems with it or improvements that can be made. Thanks!

@veqryn

This comment has been minimized.

Copy link

commented Aug 21, 2018

I would like to be able to create a GRPC Client and have it talk to my existing GRPC Server within the same process, without sending network traffic through loopback/localhost (without dialing).

I want to do this because I am hosting a GRPC Server and a GRPC Gateway in the same process (https://github.com/grpc-ecosystem/grpc-gateway).
The gateway requires a GRPC Client, which means that the GRPC Gateway uses the GRPC Client to send traffic on localhost loopback to the GRPC Server.

For example, I would like something roughly like this:

// GRPC Server
grpcServer := grpc.NewServer()
pb.RegisterPixelServer(grpcServer, myService)
// Serve and listen in a goroutine...

// GRPC Gateway
mux := runtime.NewServeMux()
grpcClient := pb.NewClientAdaptor(grpcServer, myService)
err := gw.RegisterPixelHandlerClient(ctx, mux, grpcClient)
// Serve up the gateway mux in a goroutine...

I realize that it is possible to manually make myService satisfy both MyServiceServer and MyServiceClient interfaces on a generated golang proto file.
However doing so would mean that grpc.CallOption's would get ignored, and I would also lose out on grpc's interceptors and stats handling, and other things that the grpc client and server are doing that would be bypassed by handling it directly.

@jhump

This comment has been minimized.

Copy link
Member

commented Aug 21, 2018

@veqryn, until this is provided in the core library, try out this:
https://godoc.org/github.com/fullstorydev/grpchan
https://godoc.org/github.com/fullstorydev/grpchan/inprocgrpc

// collect handlers in this map
handlers := grpchan.HandlerMap{}
// Pass --grpchan_out parameter to protoc after installing protoc-gen-grpchan
// in order to get this extra registration function:
pb.RegisterHandlerPixel(handlers, myService)

// Now we can wire up the handlers to both a gRPC server and an in-proc one
grpcServer := grpc.NewServer()
inproc := &inprocgrpc.Channel{}
handlers.ForEach(grpcServer.RegisterService)
handlers.ForEach(inproc.RegisterService)

// GRPC Gateway
mux := runtime.NewServeMux()
// protoc --grpchan_out will also generate this alternate client factory:
inprocClient := pb.NewPixelChannelClient(inproc)
err := gw.RegisterPixelHandlerClient(ctx, mux, inprocClient)
@jellevandenhooff

This comment has been minimized.

Copy link

commented Sep 9, 2018

I wish the Transport interface let implementors take care of service discovery and connection management. I suspect that both an in-memory transport and an HTTP/1.1 transport would want to do their own connection maintenance and service discovery and replace the default grpc.ClientConn implementation.

One way to do that would be to add a target string to the NewStream method. That would make the Transport interface much closer to http.RoundTripper (and make it easy to swap in eg. a http2.Transport-based implementation for eg #2112.) That would also make it possible to move the default connection maintenance and service discovery to a Transport out of the ClientConn, making the package quite a bit more flexible.

I wonder if what this issue is really asking for is for grpc.ClientConn to be an interface as in #1287; the Transport.NewStream method above looks super similar to ClientConn.NewStream.

@jhump

This comment has been minimized.

Copy link
Member

commented Sep 9, 2018

@jellevandenhooff, that is not the responsibility of a transport; that is the responsibility of a channel. In the layers of a gRPC client, the transport is the lowest layer and just implements a single connection. The channel (in the Go package, *grpc.ClientConn) is basically a "transport manager", which handles connection life cycle, pooling, pluggable service discovery, and pluggable load balancing strategies).

To implement custom service discovery and connection management, implement your own channel. While the current protoc-gen-go plugin does not work with custom channel implementations, you can use https://github.com/fullstorydev/grpchan for that. The channel currently must provide a separate method for unary RPCs vs. streaming RPCs (since that mirrors the API of *grpc.ClientConn, and the code generated by protoc-gen-grpchan is intentionally identical to that produced by protoc-gen-go, accept for taking a grpchan.Channel instead of a *grpc.ClientConn).

@johanbrandhorst

This comment has been minimized.

Copy link
Contributor

commented Jan 22, 2019

It's been almost 6 months since the latest status update from @dfawley, what is the latest status of this effort?

@dfawley

This comment has been minimized.

Copy link
Contributor

commented Jan 22, 2019

Sorry for the delay! Priorities shift and there is a lot going on, but I have not lost sight of this. This is old news, but there is a gRFC and also a proof of concept / prototype in my fork for the client-side work. I will be working on this with the aim to finish it up this quarter.

@alexmt

This comment has been minimized.

Copy link

commented Mar 13, 2019

One more use-case for custom transport is to support HTTP1.1 + gRPC-Web.

The project which needs it is https://github.com/argoproj/argo-cd. We have CLI client written in Golang which talks to gRPC API server. In most cases API server is running behind proxy and our users are struggling to configure the proxy to serve gRPC/HTTP/2 and terminate SSL at the same time.

The workaround was to run gRPC revese-proxy within CLI which converts gRPC/HTTP2 into gRPC-Web and send to API server over HTTP/1 . It would be great to just configure client to use gRPC-Web/HTTP1 transport.

@achew22

This comment has been minimized.

Copy link

commented Apr 11, 2019

@dfawley, friendliest of pings on this.

@dfawley

This comment has been minimized.

Copy link
Contributor

commented Apr 24, 2019

@achew22 Sad to say this was at the end of my list last quarter, I never got to it, and it didn't make the list this quarter (Apr-Jun). I still really want to see this happen; if I can't sneak it in this quarter somehow then I'll make sure it gets attention next quarter (Jul-Oct). This is a pretty big effort, but most of the work is essentially done, particularly for client-side.

@jared2501

This comment has been minimized.

Copy link

commented Jul 21, 2019

Quick ping on this again - some code I'm writing wishes to invoke an in-process gRPC server. It's a pain to have it invoke itself on on a TCP port since this pathway requires authorization.

@dfawley

This comment has been minimized.

Copy link
Contributor

commented Jul 22, 2019

I expect to start up on this again by ~mid-August.

If you are on linux you should be able to use UDS without worrying about authorization. Another option is an in-memory pipe like https://godoc.org/google.golang.org/grpc/test/bufconn.

@stale

This comment has been minimized.

Copy link

commented Sep 6, 2019

This issue is labeled as requiring an update from the reporter, and no update has been received after 7 days. If no update is provided in the next 7 days, this issue will be automatically closed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.