Scalable Compute needs Scalable State
Bringing stateful microservices, and the power of reactive technologies to the Cloud Native ecosystem breaks down the final impediment standing in the way of a Serverless platform for general-purpose application development, true elastic scalability, and global deployment in the Kubernetes ecosystem. The marriage of Knative and Akka Cluster on Kubernetes allows applications to not only scale efficiently, but to manage distributed state reliably at scale while maintaining its global or local level of data consistency, opening up for a whole range of new addressable use-cases.
We are actively working on a Proof of Concept for CloudState event sourcing. This document describes the PoC, how to run, test and develop it. This document will likely change frequently as the PoC progresses.
The operator should work on any Kubernetes distribution, we have tested on GKE with Kubernetes 1.11 and 1.12.
Running on GKE
Create a GKE cluster. We recommend at least 6 vCPUs (ie, a node pool of 3
n1-standard-2nodes). Also ensure that the current user is a cluster admin. Detailed instructions for creating the GKE cluster can be found in the Knative documentation, follow all the steps up to (but not including) installing Knative.
Install Cassandra. This can be done from the Google Marketplace, by visiting the Cassandra Cluster, selecting configure, selecting your GCloud project, and then installing it in the Kubernetes cluster you just created. The defaults should be good enough, in our examples we called the app instance name
Install the CloudState operator:
kubectl apply -f https://raw.githubusercontent.com/lightbend/stateful-serverless/v0.3/src/operator/stateful-serverless.yaml
You are now ready to install an event sourced function. We have a shopping cart example in the
samples/js-shopping-cart directory of this project. This can be installed by following these instructions:
Configure a Cassandra journal. If you called your Cassandra deployment
cassandraand deployed it to the default namespace, this can be installed by running:
kubectl apply -f https://raw.githubusercontent.com/lightbend/stateful-serverless/v0.3/src/samples/js-shopping-cart/journal.yaml
Otherwise, download the above file and update the
serviceparameter to match the first node of your Cassandra stateful set.
Install the shopping cart, this can be done by running:
kubectl apply -f https://raw.githubusercontent.com/lightbend/stateful-serverless/v0.3/src/samples/js-shopping-cart/eventsourced.yaml
The operator will install a service that will expose this on an external IP address, watch the service to find out what the IP address is (it may take a minute or so to provision):
kubectl get svc shopping-cart -w
To test, instantiate a gRPC client in your favourite language for this descriptor. You may need to also download the
google/protobuf/empty.protodescriptors to compile it in your language. The shopping cart descriptor is deployed with debug on, so try getting the logs of the
shopping-cartcontainer in each of the deployed pods to see what's happening when commands are sent.
Points of interest
We'll start with the user function, which can be found in
samples/js-shopping-cart. The following files are interesting:
shoppingcart.proto- This is the gRPC interface that is exposed to the rest of the world. The user function doesn't implement this directly, it passes it to the Akka backend, that implements it, and then proxies all requests to the user function through an event sourcing specific protocol. Note the use of the
cloudstate.entity_keyfield option extension, this is used to indicate which field(s) form the entity key, which the Akka backend will use to identify entities and shard them.
domain.proto- These are the protobuf message definitions for the domain events and state. They are used to serialize and deserialize events stored in the journal, as well as being used to store the current state which gets serialized and deserialized as snapshots when snapshotting is used.
cloudstate-event-sourcingNode.js module to actually implement the event sourcing protocol.
cloudstate-event-sourcing Node module, which can be found in
entity.proto- This is the protocol that is implemented by the library, and invoked by the Akka backend. Commands, replies, events and snapshots are serialized into
google.protobuf.Any- the command payloads and reply payloads are the gRPC input and output messages, while the event and snapshot payloads are what gets stored to persistence. The
readyrpc method on the
Entityservice is used by the Akka backend to ask the user function for the gRPC protobuf descriptor it wishes to be served, this uses
google.protobuf.FileDescriptorPrototo serialize the descriptor.
entity.js- This is the implementation of the protocol, which adapts the protocol to the API used by the user function.
Next we'll take a look at the Akka proxy, which can be found in
Serve.scala- This provides the dynamically implemented gRPC interface as specified by the user function. Requests are forwarded as commands to the cluster sharded persistent entities.
StateManager.scala- This is an Akka persistent actor that talks to the user function via the event sourcing gRPC protocol.
CloudStateProxyMain.scala- This pulls everything together, starting the Akka gRPC server, cluster sharding, and persistence.
HttpApi.scala- This reads google.api.HttpRule annotations to generate HTTP/1.1 + JSON endpoints for the gRPC service methods.
The TCK makes it possible to verify that combinations of backends and frontends behaves as expected. In order to make a frontend eligible for testing in the TCK a sample application implementing a simple Shopping Cart (here showcased with the Node.js frontend) is required.