Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stream driven Functions #1159

Open
scothis opened this Issue Mar 15, 2019 · 1 comment

Comments

Projects
None yet
4 participants
@scothis
Copy link
Member

commented Mar 15, 2019

In riff 0.0.x we had support for streaming messages between Functions mediated by Topics backed by Kafka. With the transition to Knative, we lost the ability to stream messages from a Channel to a function and were limited to item-at-a-time request-reply semantics.

We still care deeply about the potential of stateful stream processing in a polyglot, FaaS environment.

Knative serving is no longer restricted to http/1.1, which unlocks support for streaming protocols like h2, WebSockets, gRPC and RSocket. Knative eventing has had no movement towards supporting any protocol other than http/1.1.

High level things to do:

  • bless a baseline protocol for channels and invokers to support
    • define additional stream semantics as needed
  • create stream enabled channels
  • update all invokers to support streams
  • ensure proper autoscaling for stream based requests
  • bridge http/1.1 into a stream based function
@markfisher

This comment has been minimized.

Copy link
Contributor

commented Apr 8, 2019

We have started working on a PoC that includes the following components:

  1. Stream CRD with "name" and "provider" (other properties will be added, e.g. "partitions")
  2. Processor CRD with "inputs", "outputs", and "function"
  3. Stream Gateway providing a generic API that can be backed by different event-log based messaging systems (such as Kafka or Kinesis); initially exploring liiklus
  4. Processor will connect to the Stream Gateway(s) for the input/output Streams and interact with the target Function (with which it's colocated in a Pod) via its Invoker layer using gRPC for bidirectional streaming

scothis added a commit to scothis/riff that referenced this issue Apr 8, 2019

Drop Knative Eventing
We'll replace the capability with a new streamming model.

This also upgrade Knative Serving and Build to 0.5 which was previously
blocked by also depending on Knative Eventing 0.4.

Refs: projectriff#1159

scothis added a commit that referenced this issue Apr 12, 2019

Drop Knative Eventing (#1237)
We'll replace the capability with a new streamming model.

This also upgrade Knative Serving and Build to 0.5 which was previously
blocked by also depending on Knative Eventing 0.4.

Refs: #1159
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.