Suro: Netflix's Data Pipeline
Suro is a data pipeline service for collecting, aggregating, and dispatching large volume of application events including log data. It has the following features:
- It is distributed and can be horizontally scaled.
- It supports streaming data flow, large number of connections, and high throughput.
- It allows dynamically dispatching events to different locations with flexible dispatching rules.
- It has a simple and flexible architecture to allow users to add additional data destinations.
- It fits well into NetflixOSS ecosystem
- It is a best-effort data pipeline with support of flexible retries and store-and-forward to minimize message loss
Master Build Status
Pull Request Build Status
NetflixGraph is built via Gradle (www.gradle.org). To build from the command line:
build.gradle file for other gradle targets, like
Running the server
You can run the server locally by just running
More more advanced usage you may wish to run
./gradlew installApp and then:
cd suro-server java -cp "build/install/suro-server/lib/*" com.netflix.suro.SuroServer -m conf/routingmap.json -s conf/sink.json -i conf/input.json
To enable basic logging you can downloaded
slf4j-simple-1.7.7.jar and copy it into
suro-server then run:
cd suro-server java -cp "build/install/suro-server/lib/*:slf4j-simple-1.7.7.jar" com.netflix.suro.SuroServer -m conf/routingmap.json -s conf/sink.json -i conf/input.json
We will use the Google Group, Suro Users, to discuss issues: https://groups.google.com/forum/#!forum/suro-users