Skip to content
No description, website, or topics provided.
Java Shell
Branch: master
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
ci Fix deploy script Aug 20, 2019
src Properly ACKing received messages Jul 12, 2019
.gitignore Inception Jun 14, 2019
.travis.yml Add slack build notifications Sep 4, 2019
README.adoc Update riff command Aug 20, 2019
pom.xml Properly ACKing received messages Jul 12, 2019
riff-streaming-concepts.png Inception Jun 14, 2019

README.adoc

riff Function Streaming Processor

Build Status

This repository contains the sources of the riff function streaming processor, a component used when dealing with streaming functions.

Purpose

When dealing with streaming functions, one creates a processor in riff like so:

riff streaming processor create example --function-ref some-existing-fn \
    --input  input-stream \
    --input  maybe-another-input \
    --output some-output-stream \
    --output maybe-another \
    --output even-a-third

This creates a pod made of two containers. This component will run in the function pod, alongside the function container, and will

  1. continuously pump data from any of the function input streams (input-stream and maybe-another-input in the example above) out of the stream gateway(s)

  2. extract the message payloads from the serialized form used in the broker(s) (see riff-serialization.proto)

  3. decide how to craft windows of function invocation (currently hardcoded to one minute of wallclock time)

  4. invoke the function over RPC, multiplexing the many input streams over the single stream allowed by the RPC spec (see riff-rpc.proto)

  5. upon reception of result frames, de-mux messages and serialize them back to the appropriate output streams (some-output-stream, maybe-another and even-a-third in the example above).

Notice that the number of input and output streams can be different and depends entirely on how the actual function is implemented. Likewise, the rate at which messages flow in and out is entirely dictated by the function implementation (one piece of input may trigger the emission of N output messages, in quick succession or over time).

riff streaming concepts

Building

This component is written in Java and leverages the following technologies:

  • Spring Boot as the overall framework, in particular for crafting an executable jar,

  • Liiklus as the stream gateway,

  • gRPC as both the RPC protocol to interact with liiklus and the riff RPC invocation protocol,

  • Reactor and reactor-grpc as the main event loop mechanism, preserving back-pressure from output to input.

To build the project, assuming a fresh checkout of this repository as the current directory:

mvn package

To build and create (and push) a docker image:

mvn package com.google.cloud.tools:jib-maven-plugin:1.3.0:build -Dimage=<MY IMAGE>

Running

When run, the processor expects the following environment variables to be set:

  • INPUTS: a comma separated list of N input stream coordinates, in the form gatewayHost:port/topicName,

  • OUTPUTS: a comma separated list of M output stream coordinates, in the form gatewayHost:port/topicName,

  • GROUP: a string identifier that will be used as the consumer group for the processor. Should be unique per deployment of the processor (not per function),

  • FUNCTION: location of the function RPC server, in the form host:port,

  • OUTPUT_CONTENT_TYPES: a json encoding of the content types expected for the M output streams, in the form

    [
        {"outputIndex": 0, "contentType": "text/plain"},
        {"outputIndex": 1, "contentType": "application/json"}
    ]
You can’t perform that action at this time.