gRPC Meets Serverless

[Disturbing]

Hello Fission Community! First post so do point me to the correct location if I'm in the wrong here. :).

I'm working on a Microservice-first platform and decided that gRPC backend is the way I'd like to go. The backend itself will support both gRPC and RESTFUL endpoints (using restful transcoding). Later we will support more endpoints like UDP for games which I'm working proposing to the gRPC community to work on.

What I would like to propose is the support of building gRPC services that can have a 100Ms load time on demand. These services would simply be gRPC Bindable Services (a collection of APIs defined by and added to a gRPC Server dynamically. After some time, it will be removed from the gRPC Server.

The downside is that you will have slightly larger files to load into memory which will slow things down. But the upside is that if you have a pool of gRPC services with premade connections to other gRPC services if the node has them loaded already. Think of it as:

Node 1 has Service A -> Service B.
Node 1 has Service C -> Service B

With gRPC and the concept of 'shared channels', you will not have to create a new connection to service B if both Service A and Service C was loaded into the warm container. Another great benefit of this would be able to directly access local instances of services if they were loaded versus going across the wire. (This concept could be done with fission alone as well to call local warm functions)

I'd love to hear the community's thoughts on this and see if this is something people are interested in. Otherwise, it may make more sense to do it as a separate open source initiative.

[Disturbing]

I just ran some early unoptimized tests (of course not on a server running a massive amount of services at the same time), but 1000 lined compiled service (9Kb) 3rd party jar /w no dependencies takes ~20 Ms to cold start with gRPC + Java.

[erwinvaneyk]

Hi @Disturbing! I am not sure if I fully understand your proposal. From your explanation I infer that you are proposing to fully integrate gRPC into Fission, such that that every function is a separate gRPC service, correct?

Although, in my opinion, gRPC/Protobuf is interesting as a communication protocol to use in Fission, instead of less unstructured JSON, I do not see the immediate benefits of enforcing gRPC every where. As you mention in your own post, this would restrict the communication possibilities, excluding UDP or other (streaming) protocols. Wouldn't this be better to leave up to the user to implement on top of Fission?

So, maybe I am misunderstanding your proposal, but could you clarify the proposal a bit more in the context of Fission?

[soamvasani]

What I would like to propose is the support of building gRPC services that can have a 100Ms load time on demand.

Hey @Disturbing! We're definitely interested in gRPC invocation of functions. It would need support in the language environments and in the router. We should sketch out a design and implementation plan.

The ideas around shared channels should perhaps be tackled as a next step after that.

[Disturbing]

@soamvasani - Great, I just was speaking to Vishal @ Infracloud and mentioned you both are well connected. He's advising my next startup so let's def connect over this - would hate to make it a second project and my team will have a fair amount of time + contribution in the next 60 days for this.

@erwinvaneyk - It's not just about the protobuf (the protocol itself), but about supporting gRPC calls with protobuf. I'm thinking of this as more of a project standard versus hack together a single function while supporting HTTP/2 in fission.

Right now language environments take 'fission functions' to build/deploy/route. My ask is that we take gRPC services to build/deploy/route. A gRPC service can be defined like below:

// helloworldservice.proto
service HelloWorldService {
  rpc SayHello (ExampleMessage) returns (ExampleResponse) {}
}
message ExampleMessage {
	string name = 1;
}
message ExampleResponse {
	bool coolName = 1;
}

If you use this file to generate the java version if it and implements the class, this is what it looks like:

public class HelloWorldService extends HelloWorldServiceImplBase {
	private final Logger logger = Logger.getLogger(StarterService.class.getName());
			
	@Override
	public void sayHello(ExampleMessage message, StreamObserver<ExampleResponse> responseObserver)
	{
                // Example context data
		long appId = CommonContext.APP_ID.get();
		AppConfig configData = CommonContext.CONFIG.get();
		logger.log(Level.INFO, String.format("Hello world message received! %s", message.getName()));
		responseObserver.onNext(ExampleResponse.newBuilder().setCoolName(true).build());
		responseObserver.onCompleted();
	}
}

I propose two new inputs to fission:

fission function create --name hello --env grpc-java --code HelloWorldService.java
OR
fission function create --name hello --env grpc-java --jar HelloWorldService.jar

Each gRPC environment would be quite simple to make. Using MutableTypeHandlers, you can add and remove Services. Every gRPC Service generated implements BindableService which can be hot-swapped in and out of a service.

What this means is that we can have a single environment process running, on a single port, supporting an unlimited amount of services.

What are the benefits here?

• Ability to generate rpc+API on other languages + swagger def (gRPC can support restful as well)
• We can used shared channels (in the future) to highly increase performance for inter-communication between microservices
• Introduce the HTTP/2 protocol which has loads of benefits
• Introduce Contexts using Metadata handlers for gRPC. With frameworks like istio and linkerd, you can put some magic in the headers and throw them into contexts to easily have one function support multiple apps/clients like true microservices should
• I have ideas around 'SharedServices' which I'd like to introduce later. Imagine needing access to a shared mongodb instance and having a shared provider where multiple functions or services can get a warm connection pool to the database for faster queries + zero initialization.
• Be able to create compilable projects with multi-files and multi-function support.
• Create an environment library to do local debugging within IDE.

With that said, my interest is in creating entire gRPC "BindableServices" which can be registered to fission and would love to start working on the .NET Core and Java versions of them asap :).

[erwinvaneyk]

Thanks for the clarification @Disturbing, it really helps with understanding your proposal.

From how I read it, there are 3 main aspects of your proposal, namely:

• Incorporating gRPC into the Fission functions.
• Deploying multiple functions in the same environment.
• Introducing 'SharedServices' to pool resources, such as DB connections.

I definitely like the idea of integrating gRPC into the functions. Especially as it, like you mentioned, allows for typing (and all the benefits that brings). However, I don't think we should force this upon users, as I can imagine that there many cases in which you, as a Fission user, just want to hack to together a couple of functions without wanting to deal with types or any other boilerplate. Likewise, there are only a handful of languages that gRPC supports. So what about making the gRPC definitions optional? For things like type checking or autocompletion for example, you can just assume those functions to have a Any input/output message.

For the hotswapping using a single RPC environment, I get the benefits (performance, shared connections) you advocate. However there are also some new concerns that arise from allowing a single environment to host multiple functions. From the top of my head, a main issue is autoscaling. How do you ensure that a single resource-intensive function does not swamp all other functions in the same environment? Same as with the previous aspect, I can imagine definitely improves the performance, but it might not hold for all use cases. So it is definitely a good option to have, but IMO it should not be the only option. If we can properly define environments, the hotswapping could just be an implementation concern of the gRPC-enabled environment. For example, the environment could have an option to indicate whether it accepts more than one specializations. After specializing an environment, it could just return a status, which indicates whether it accepts more specializations. This would also allow an RPC environment to limit the number of specializations/functions it will control, limiting the risk of the resource swamping.

SharedServices is an interesting concept, that is approached in all sorts of different ways. For example, the data bindings in Azure Functions delegates the connections to the infrastructure, which can share connections and other performance improvements. Is that comparable to your SharedServices proposal? Personally, I am not entirely convinced that a separate entity (as in besides regular Fission functions) is needed for these types of issues. Anyhow, reading your proposal this is more something for in the future, so let's not go into too much detail here.

tl;dr What are the benefits of forcing this for all functions over supporting both the typed/hotswappable/etc gRPC functions and the 'regular', untyped, HTTP functions?

All in all, it is a very interesting idea, and Fission would definitely be the right place for this. It sounds like your end goal is to have source code to directly compile to a distributed system/program, which is also something I find really exciting. 😄

[Disturbing]

Great points and glad you like the idea. Thanks for taking the time to review this. And yes, directly compile to a distributed system/program is just the beginning ;) Feel free to reach out in slack (@Disturbing) and can share more on my end goals for this.

Moving forward - let's focus on Incorporating gRPC + Multiple "services" in a single environment.

Benefits for supporting both typed/hot-swappable gRPC functions and regular untyped http functions

Benefits of forcing this for all functions is performance and standardizations. gRPC can support both REST + HTTP/2, so this is not a problem. But I'm positive that gRPC's performance of executing code and handling a high amount of QoS on HTTP/2 is epic alongside the security and tools out of the box gRPC comes with.

The problem is, setting up a gRPC service from scratch takes more time than quickly writing a function, but you get a clean model, optimized protocol and more production ready code out of the box. To speed up the process with using gRPC, I'd suggest 'starter packs' where you simply fork or pull as a ready-to-go example to modify and implement a service. One java starter pack may just have a clean hello world service + proto file in a compilable maven project. Another may be GuiceJavaStarterPack which comes with sweet Dependency Injection tool to automatically inject dependencies into your service that you call for.

At the end of the day, taking away 'fission functions' should not occur here and having grpc as an addon environment would be a good way for people coming to serverless for the first time to have a service/project oriented aside from moving their traditional monolithic gRPC's to serverless quickly.

Incorporating gRPC

I agree that gRPC should not be 'the standard' of fission, but an amazing option. Would hate to see it as a fork, but simply incorporating as an env would be best. Hacking together functions quickly in a few lines of code + commands is essential. Following the idea of being able to set something up in less than 5-10 minutes should be the ultimate goal in building tools.

gRPC currently officially supports 10 languages which IMO are the top 10 languages, I think it's a great thing to have.

From the c#/c++/java version that I have reviewed, they all support dynamic registration of services which will make it easy to build a core that hosts services on the fly. We would need to confirm the rest.

What I would need to learn about to move forward: (EDITED)

I just read: https://github.com/fission/fission/blob/master/Documentation/wip/env-api.md

• I need to team up with someone that knows go to write a gRPC HTTP/2 Router support.
• I need to team up with someone to registering a package environment of a folder for multiple files.
• I would suggest skipping the FunctionLoadRequest functionality for a grpc env. I can custom load services on the fly as long as a connection is forwarded to the environment correctly. This will save RTT on the response. But it requires understanding how / where packages are located...
• In order to load a service, I would need to understand how the storage is working. If I can get access to the root path of where all packages would be located, that's great. If root paths are dynamic, then there may be an issue. Also packages would have to be name the same as the service name gRPC generates in order to do the unique mapping.

I'd love to discuss the above via slack if anyone is interested. I'm all game to work on this - this week.

Multi-service support

I would suggest a --dedicated option of true or false. Scaling should not be a problem for most apps. There are major benefits to sharing memory and CPU benefits of sharing services on a single application instance versus not. Java, for instance, has about 40 Mb requirement by default per instance, doing a separate instance per would not sit well.

Using smart load balancers like istio and linkerd as the core router (or at least take strategies from them) would aid this. If a particular container is overloaded, you do not need to forward further calls to that box and instead call another grpc environment which will auto load the service of need. Once warm environment containers begin to run out, spawn more. If warm env containers are running out fast, spawn two more at a time. The load balancing portion is usually a user-level and app-level decision, but I'm happy to see these smart load balancers use the 'response time' of calls in mix with CPU+Memory limits to scaling out.

Thoughts on stateful services

I would like to explore stateful services at some point in being able to register a function or service that does not go down after N duration. Real time game rooms like www.eatme.io and league of legends cannot be stateless, but spawning multiple instances of a room would be awesome. No need to discuss today, but going into 'service land' I'm curious if it Fission would be interested in the following

fission service create --name hello --env grpc-java --code HelloWorldService.java
Successfully created hello at {uri}:{port} with id 1
fission service delete --name hello --id 1

Shared service would be amazing and being able to spawn / shutdown them on the fly to easily duplicate game rooms like League of legends or Eat me would save a lot of time in developing managers that manage stateful services.

The reason for shared services is that these game rooms do not take much memory (64Mb is min docker memory requirement). Creating a unique app per game room would utilize a base 40Mb in java for instance per room when each room only takes ~64Kb in memory. Supporting this out of the box would be a huge benefit aside from stateless services with grpc :)

[vjpai]

Hello from the gRPC world! @soamvasani : it was a pleasure talking with you at Kubecon. I think this is a topic that requires investigation. One of the topics that is listed above that merits clarification: although all gRPC language APIs suppose multiple services registered per server, none of them currently support new services being registered to an already running server, and none of them currently support services being de-registered from a running server. I think that both of these are needed in order for gRPC to conveniently support serverless since functions will be dynamically created and sharded.

An alternative approach is to use gRPC with generic byte buffers rather than protobufs. All languages support this form of communication (for example, the GenericStub in C++ or the use of an empty codec in Go). Then the gRPC server would see the raw method name and could decide whether or not to invoke a function handler for it and how to pass it the data as well as manually call protobuf ser/deser (or whatever other ser/deser library you want to use).

I'll create a matching issue in the grpc repo and cross-reference.

[vjpai]

The story about gRPC isn't quite as grim as I depicted. At least in java, there is experimental API already in place to allow registration and deregistration of services. It's not well-documented but it exists: https://grpc.io/grpc-java/javadoc/io/grpc/HandlerRegistry.html .

[soamvasani]

Thanks for following up, Vijay! Great talking at Kubecon! :)

I'm now thinking, maybe we can get some of way there without needing dynamic service loading in gRPC. We could abandon the cold-start optimization in this case, and just start a new pod the conventional way using fission's newDeploy executor to create a k8s Deployment and Service. (The discussion above predates fission's ability to create new k8s deployments on function call.)

Users of fission gRPC functions would have to choose between having a pod running at all times, vs. paying the latency cost of starting a new pod. They won't have fast 0-to-1 scaling but they'll have other advantages.

We'll need to investigate if the router needs changes. It looks like gRPC already supports HTTP proxies, we need to make sure our proxy works correctly with CONNECT.

[plaisted]

I've been playing around with a similar setup for "gRPC as a service" and really like what fission has to offer. Unfortunately I don't think gRPC plays nice with traditional HTTP proxies (Envoy is only one I know of that supports it) so I think this would be difficult to get implemented in fission.

My general use cases would be deploying a gRPC "function" and then accessing it by...

Using gRPC end-to-end:

gRPC --> $FISSION_ROUTER/grpc/hello --> gRPC method

or using HTTP that gets proxied to gRPC stub:

http JSON payload --> http://$FISSION_ROUTER/hello --> HTTP  to gRPC proxy --> gRPC method

Having a k8s service / container for each function would work here fine but the routing of the gRPC would be difficult if current infrastructure doesn't support HTTP/2 and gRPC. Would there potentially need to be a standard fission router and then an optional Envoy based GRPC_ROUTER for gRPC use cases?

Apologies if this is off base, my knowledge of the fission ecosystem is very limited, just stumbled on this while doing some research for my home grown solution.

[Disturbing]

That or linkerd is good for discovery. You are spot on with the challenge. Unfortunately the ticket is low prio (or seems to be) Major features like autoscale and others make sense to go first.

…

On Sat, Jan 6, 2018 at 2:34 AM Michael Plaisted ***@***.***> wrote: I've been playing around with a similar setup for "gRPC as a service" and really like what fission has to offer. Unfortunately I don't think gRPC plays nice with traditional HTTP proxies (Envoy is only one I know of that supports it) so I think this would be difficult to get implemented in fission. My general use cases would be deploying a gRPC "function" and then accessing it by... Using gRPC end-to-end: gRPC --> $FISSION_ROUTER/grpc/hello --> gRPC method or using HTTP that gets proxied to gRPC stub: http JSON payload --> http://$FISSION_ROUTER/hello --> HTTP to gRPC proxy --> gRPC method Having a k8s service / container for each function would work here fine but the routing of the gRPC would be difficult if current infrastructure doesn't support HTTP/2 and gRPC. Would there potentially need to be a standard fission router and then an optional Envoy based GRPC_ROUTER for gRPC use cases? Apologies if this is off base, my knowledge of the fission ecosystem is very limited, just stumbled on this while doing some research for my home grown solution. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#250 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACQev2MvB7aLpM-Za5QSFWSPD2hGZx5fks5tHs20gaJpZM4OPdU2> .

[sanketsudake]

We can also explore https://github.com/bufbuild/connect-go

[wenerme]

Tried connect-go recentlly, works as promised, really looking forward fission support connect-go semantic.