go-data-routing

The library provides a pipeline engine for stream processing of data. It is based on the concept of EIP (enterprise integration patterns).

The motivation for this project is simple: to get an easy and clear way of coding ETL-like programs for parallel processing of data. In my case it was a BFS crawler tuned for extraction of specific metadata, (see a basic version in example folder).

Features

The library provides the following primitives:

• route (chain of nodes processing messages)
• node:
  • filter -- filter a stream of exchanges
  • processor -- processes a stream of exchanges (tasks) in parallel
  • wire tap: sends a copy of msg to another route (referenced by name)
  • to: enrich msg on another route (request-reply / enrichment pattern)

All the primitives are accessible through DSL.

Route builder DSL (methods):

Method	Signature	Args
Source	f func(n *Node)	function f, used for generation of exchanges
Filter	f func(e Exchange, n *Node)	function f, intercepts exchanges
Process	workers int	Number of workers, beeing used to process exchanges
To	route string	Name of the route, where to redirect an exchange for execution of request-reply
WireTap	route string	Name of the route, where to copy an exchange
Sink	f func(e Exchange) error	Function f, used for consumption exchanges

Examples:

• a simplistic bfs crawler: go run examples/crawler/main.go

Design of node:

• each node is connected with the next one (if exists) only with 1 channel
• node owns an input channel
• output is just a reference to the input of next node
• node does not close the output channel, instead it just sends a Stop msg to a next node
• if a node is the last in a chain than an output message being sent is discarded unless it's not a RequestReply

Code coverage

make code-coverage and see the coverage.html for details.

github.com/Zensey/go-data-routing/node.go (100.0%)
github.com/Zensey/go-data-routing/nodetype_string.go (75.0%)
github.com/Zensey/go-data-routing/pool.go (100.0%)
github.com/Zensey/go-data-routing/route_builder.go (71.8%)
github.com/Zensey/go-data-routing/router_context.go (100.0%)
github.com/Zensey/go-data-routing/worker.go (81.8%)

Related links

• A Stream Processing API for Go
• https://github.com/zensey/go-pool-of-workers
• https://github.com/chuntaojun/go-fork-join
• https://github.com/spatially/go-workgroup