Fn Flow Server
Clone or download
CI
Latest commit 60607e1 Sep 13, 2018
Permalink
Failed to load latest commit information.
.circleci adding govalidators to build Sep 10, 2018
actor Update to use dep, fix validation on fn IDs Sep 10, 2018
blobs Keep each test database in a different path; clean up afterwards Nov 30, 2017
cluster rebase and vet Nov 24, 2017
docs fix references to /r in docs Sep 11, 2018
graph Turn the logs down low, remove viper, make ports configurable Nov 29, 2017
infra Make cluster node port 8081, and configurable, update compose and kub… Nov 6, 2017
model updated to latest go-bin Sep 10, 2018
persistence add range check to inmemprovider Dec 2, 2017
protocol adjusting types, removing old proto code Nov 23, 2017
server fixes log format Jan 18, 2018
setup Update to use dep, fix validation on fn IDs Sep 10, 2018
sharding s/graph_id/flow_id/ golintyvety Nov 21, 2017
vendor Actually update vendor and dep files Sep 10, 2018
version flow: 0.1.85 release [skip ci] Sep 13, 2018
.gitignore Add barebones prometheus handler to server Oct 26, 2017
CONTRIBUTING.md initial rename changes Oct 24, 2017
Dockerfile initial rename changes Oct 24, 2017
Gopkg.lock Actually update vendor and dep files Sep 10, 2018
Gopkg.toml Actually update vendor and dep files Sep 10, 2018
LICENSE License Sep 22, 2017
Makefile updated to latest go-bin Sep 10, 2018
README.md fix references to /r in docs Sep 11, 2018
logo.jpg Add Readme, Update API Header naming, Add Old REST API section Aug 10, 2017
main.go basic blob server Nov 21, 2017
release.sh renaming project Oct 24, 2017
test.sh Add the go tools to the test path Nov 30, 2017
wercker.yml Add completion graph (#10) Aug 14, 2017

README.md

Fn Flow

Fn Project

CircleCI GoDoc Go Report Card

Fn Flow lets you build long-running, reliable and scalable functions using Fn that only consume compute resources when they have work to do and are written purely in code. Flow supports building complex parallel processes that are readable, testable (including via unit testing) using standard programming tools. Flow empowers you to build workflows as distributed programs that are as complex as you need them to be and supports a rich set of concurrency primitives including fork-join, chaining, delays and error handling.

How Fn Flow is different from other workflow tools

Most workflow systems strongly separate the coordination of a process from its execution, typically using declarative workflow programming languages like state machines to execute the workflow in one system and the units of work (e.g. steps, stages) in another.

This creates several new problems that Fn Flow aims to overcome:

  • Two languages, two sets of tools : As a software engineer you are already likely to be skilled at solving problems using existing programming languages and tools; requiring you to spontaneously move to a different paradigm and language just to distribute your process is hard work and slows you down. We think this context-switching is harmful and Flow is designed to let you stick to the languages and tools that you already know to quickly add features.
  • Restricted Semantics : While workflow languages are programming languages like any other, they typically do not have the same semantics as the languages you would use for other tasks, forcing you to reduce your process to something that fits in the semantic model of the workflow system. For state machine type systems this is often problematic as processes grow in complexity. The number of links between states can result in more programming errors and programs that are hard to read and reason about. Flow allows you to code in existing languages (like Java) and use existing code to implement your process.
  • Error handling : In the cloud, processes often need to handle and recover from failures in other systems, this recovery is frequently more complex than just trying again a few more times. Flow exposes these errors in a similar way to exceptions allowing you to handle, recover and take action as you see fit - including running more code to resolve the issue.
  • Glue code and context : Workflow systems are frequently used to orchestrate independent, heterogeneous tasks together. On paper this sounds simple, just drag and drop the tasks into a graph and run the code, right? In practice one task is unlikely to to take exactly the same input as another tasks output (if they did they would not be be very independent!). Tasks may also need context data from other tasks that executed before the previous step. In current workflow languages this often results in you having to add new tasks to translate data from one task to another and introduce complex data handling to pass context around. Flow allows you to use the language features of an existing programming language - glue code can be expressed using (e.g.) Java methods and expressions - without sacrificing type-safety.

For example...

The following Java Fn Flow function coordinates five other functions to:

  1. Scrape results for an image search (using the scraper function)
  2. In parallel, detect number plates in each image in the results (using detect-plates),
  3. For each detected plate, draw the location of the plate on the image (draw) then send an alert containing the rendered image (alert)
  4. Wait for the process to finish and then send a completion message (complete)
public void handleRequest(ScrapeReq input) throws Exception {
   FlowFuture<ScrapeResp> scrapes = currentFlow().invokeFunction("./scraper", input, ScrapeResp.class);
   scrapes.thenCompose(resp -> {
      List<FlowFuture<?>> pendingTasks = resp.result
         .stream()
         .map(scrapeResult -> {
            String id = scrapeResult.id;
            return currentFlow()
                  .invokeFunction("./detect-plates", 
                              new DetectPlateReq(scrapeResult.image_url, "us"), DetectPlateResp.class)
                  .thenCompose((plateResp) -> {
                     if (!plateResp.got_plate) { 
                         return currentFlow().completedValue(null);
                     }
                     return currentFlow()
                           .invokeFunction("./draw", 
                                       new DrawReq(id, scrapeResult.image_url, plateResp.rectangles,"300x300"), DrawResp.class)
                           .thenCompose((drawResp) -> currentFlow()
                                     .invokeFunction("./alert", new AlertReq(plateResp.plate, drawResp.image_url)));
                  });
         }).collect(Collectors.toList());
         
      return currentFlow()
            .allOf(pendingTasks.toArray(new FlowFuture[pendingTasks.size()]))
            .whenComplete((v, throwable) -> {
                  if (throwable != null) {
                     log.info("Scraping completed with at least one error", throwable);
                  } else {
                     currentFlow()
                        .invokeFunction("./complete",CompleteResult("Scraped " + pendingTasks.size() + " Images")));
                  }
               });;
   });
}

While the above program can be written and reasoned about as a single method, (and tested using a JUnit rule) it is in fact executed by braking each stage of the computation down into Fn Functions, each of which runs as an independent Fn call - for instance the results of 'detect-plates' may be processed on one or more different containers. Functions like 'detect-plates' may take a while to run and when they are running none of the surrounding code blocks are consuming resources in the platform.

Writing Flow functions

Flow Functions are currently supported in Java but the platform is not language specific and new language bindings can be build by implementing the function side of the Flow API. We are working on adding support for JavaScript, Python and Go.

To find out how to use Fn Flow in Java read the user guide.

Debugging flows

The Flow Service retains a trace of each Flow's execution and can publish the real time state of an ongoing flow. This can be used to diagnose errors in flows (in a similar way to using a stack trace in a debugger) - We have developed an experimental UI that shows this data in real time and lets you backtrack to the source of problems.

Flow UI

Running the Flow Service

Make sure the functions server is running

$ fn start                                                                                                                                                 ....
time="2017-09-16T22:04:49Z" level=info msg="available memory" ram=1590210560

      ______
     / ____/___
    / /_  / __ \
   / __/ / / / /
  /_/   /_/ /_/

time="2017-09-16T22:04:49Z" level=info msg="Serving Functions API on address `:8080`"

Set FNSERVER_IP to the IP address of the Fn Server:

FNSERVER_IP=$(docker inspect --type container -f '{{.NetworkSettings.IPAddress}}' fnserver)

Then run the Flow Service:

docker run --rm -d \
      -p 8081:8081 \
      -e API_URL="http://$FNSERVER_IP:8080/invoke" \
      -e no_proxy=$FNSERVER_IP \
      --name flowserver \
      fnproject/flow:latest

Configure via the environment

Env Default Usage
API_URL http://localhost:8080 sets the FN API endpoint for outbound invocations
DB_URL sqlite3://./data/flow.db DB url, you may also use "inmem:/" for in memory storage
LISTEN :8081 listen host/port (overrides PORT)

Also see our Flow UI

Get help

Contributing

Please see CONTRIBUTING.md.