Skip to content
Privacy and Security focused Segment-alternative, in Golang and React
Go Lua Other
Branch: master
Clone or download
SumanthPuram Torpedo production (#93)
Session ID support across backend and transformer
Latest commit 1af19f5 Dec 6, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
build Changes to ginkgo tests for segment compatability Nov 6, 2019
config new endpoint for uploading event schema. using workspace token for ba… ( Dec 6, 2019
gateway Torpedo production (#93) Dec 6, 2019
jobsdb Order by job_id for table dumps Nov 13, 2019
processor Torpedo production (#93) Dec 6, 2019
router Dedup events at gateway using BadgerDB (#88) Nov 13, 2019
rudder-transformer @ 31ea3ea Updating transformer submodule Nov 29, 2019
scripts Modify ulimit on rudder.service Oct 21, 2019
services new endpoint for uploading event schema. using workspace token for ba… ( Dec 6, 2019
tests Torpedo production (#93) Dec 6, 2019
utils Torpedo production (#93) Dec 6, 2019
vendor Dedup events at gateway using BadgerDB (#88) Nov 13, 2019
.editorconfig Adding editor config and setting tab width to 4 Jul 19, 2019
.gitignore v0.1.1 Nov 5, 2019
.gitmodules Directory structure Cleanup (#41) Sep 19, 2019
.goreleaser.yml v0.1.1 Nov 5, 2019
.version Buildspec to generate docker images Jul 23, 2019
CODE_OF_CONDUCT.md Create CODE_OF_CONDUCT.md Sep 27, 2019
LICENSE Adding SSPL Sep 20, 2019
README.md Update README.md Nov 13, 2019
appspec.yml Buildspec to generate docker images Jul 23, 2019
batch.json Adding batch.json under rudder-server directory to ensure that event … Oct 16, 2019
docker-compose-enterprise.yml Fix ops dashboard path. Nov 1, 2019
docker-compose-multi.yml Multiple instances of transformer and removing oneToMany boolean Oct 23, 2019
docker-compose.yml Merging master Oct 10, 2019
go.mod Dedup events at gateway using BadgerDB (#88) Nov 13, 2019
go.sum Dedup events at gateway using BadgerDB (#88) Nov 13, 2019
main.go Dedup events at gateway using BadgerDB (#88) Nov 13, 2019
release.sh Removing debug log Nov 6, 2019

README.md

Build Status

What is Rudder?

Short answer: Rudder is an open-source Segment alternative written in Go, built for the enterprise. .

Long answer: Rudder is a platform for collecting, storing and routing customer event data to dozens of tools. Rudder is open-source, can run in your cloud environment (AWS, GCP, Azure or even your data-centre) and provides a powerful transformation framework to process your event data on the fly.

Rudder runs as a single go binary with Postgres. It also needs the destination (e.g. GA, Amplitude) specific transformation code which are node scripts. This repo contains the core backend and the transformation modules of Rudder. The client SDKs are in a separate repo (link below).

Rudder server is released under SSPL License

Questions? Join our Discord channel. Or please email soumyadeb at rudderlabs.com.

Why Rudder ?

We are building Rudder because we believe open-source and cloud-prem is important for three main reasons

  1. Privacy & Security: You should be able to collect and store your customer data without sending everything to a 3rd party vendor or embedding proprietary SDKs. With Rudder, the event data is always in your control. Besides, Rudder gives you fine-grained control over what data to forward to what analytical tool.

  2. Processing Flexibility: You should be able to enhance OR transform your event data by combining it with your other internal data, e.g. stored in your transactional systems. Rudder makes that possible because it provides a powerful JS-based event transformation framework. Furthermore, since Rudder runs inside your cloud or on-prem environment, you can access your production data to join with the event data.

  3. Unlimited Events: Event volume-based pricing of most commercial systems is broken. You should be able to collect as much data as possible without worrying about overrunning event budgets. Rudder's core BE is open-source and free to use.

See the HackerNews discussion around Rudder.

Features

  1. Google Analytics, Amplitude, MixPanel, Adjust, AppsFlyer & Facebook destinations. Lot more coming soon.
  2. S3 dump. Redshift and other data warehouses coming soon.
  3. User-specified transformation to filter/transform events.
  4. Stand-alone system. The only dependency is on Postgres.
  5. High performance. On a single m4.2xlarge, Rudder can process ~3K events/sec. Performance numbers on other instance types soon.
  6. Rich UI written in react.
  7. Javascript, Android or iOS. Server-side SDKs coming soon.

Contribute or Stay Connected

  1. Join our Discord
  2. Follow us on Twitter

Setup Instructions (Docker)

The docker setup is the easiest & fastest way to try out Rudder.

  1. Go to the dashboard https://app.rudderlabs.com and set up your account. Copy your workspace token from top of the home page.
  2. Clone this repository with SSH and
  3. Replace <your_workspace_token> in build/docker.env with the above token.
  4. (Optional) Uncomment and set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in build/docker.env if you want to add S3 as a destination on the UI.
  5. Run git submodule init and git submodule update to fetch the rudder-transformer repo.
  6. Run the command docker-compose up --build to bring up all the services.
  7. Follow (Send Test Events) instructions below to send test event.

Setup Instructions (Terraform)

Our terraform scripts and instructions are in a separate repo - Download Here

Setup Instructions (Native Installation)

If you want to run each of the services without docker please follow the following steps

  1. Install Golang 1.12 or above. Download Here
  2. Install NodeJS 10.6 or above. Download Here
  3. Install PostgreSQL 10 or above and set up the DB
createdb jobsdb
createuser --superuser rudder
psql "jobsdb" -c "alter user rudder with encrypted password 'rudder'";
psql "jobsdb" -c "grant all privileges on database jobsdb to rudder";
  1. Go to the dashboard and set up your account. Copy your workspace token from top of the home page
  2. Clone this repository. Run git submodule init and git submodule update to fetch the rudder-transformer repo. and navigate to the transformer directory cd rudder-transformer
  3. Start the destination transformer node destTransformer.js
  4. Navigate back to main directory cd rudder-server. Copy the sample.env to the main directory cp config/sample.env .env
  5. Update the CONFIG_BACKEND_TOKEN environment variable with the token fetched in step 4
  6. Run the backend server go run -mod=vendor main.go
  7. Follow (Send Test Events) instructions below to send test event.

Send Test Events

  1. If you already have a Google Analytics account, keep the tracking ID handy. If not, please create one and get the tracking ID. The Google Analytics account needs to have a Web Property (Web+App does't seem to work)
  2. Create one source (Android or iOS) and configure a Google Analytics destination for the same with the above tracking ID
  3. We have bundled a shell script that can generate test events. Get the source “writeKey” from our app dashboard and then run the following command. Run ./scripts/generate-event <writeKeyHere> http://localhost:8080/v1/batch. NOTE: writeKey is different from the your_workspace_token in step 2. Former is associated with the source while the latter is for your account.
  4. You can then login to your Google Analytics account and verify that events are delivered. Go to MainPage->RealTime->Events. RealTime view is important as the other dashboard can sometimes take 24-48 hrs to refresh.
  5. You can use our Javascript, Android or iOS SDKs for sending events from your app.

UI Pages

Connections Page

image

Events Page

image

Stats Page

image

Architecture

The following is a brief overview of the major components of Rudder Stack. image

Rudder Control Plane

The UI to configure the sources, destinations etc. It consists of

Config backend: This is the backend service that handles the sources, destinations and their connections. User management and access based roles are defined here.

Customer webapp: This is the front end application that enables the teams to set up their customer data routing with Rudder. These will show you high-level data on event deliveries and more stats. It also provides access to custom enterprise features.

Rudder Data Plane

Data plane is our core engine that receives the events, stores, transforms them and reliably delivers to the destinations. This engine can be customized to your business requirements by a wide variety of configuration options. Eg. You can choose to enable backing up events to an S3 bucket, the maximum size of the event for the server to reject malicious requests. Sticking to defaults will work well for most of the companies but you have the flexibility to customize the data plane.

The data plane uses Postgres as the store for events. We built our streaming framework on top of Postgres – that’s a topic for a future blog post. Reliable delivery and order of the events are the first principles in our design.

Rudder Destination Transformation

Conversion of events from Rudder format into destination-specific format is handled by the transformation module. The transformation codes are written in Javascript. I

The following blogs provide an overview of our transformation module

https://rudderlabs.com/transformations-in-rudder-part-1/

https://rudderlabs.com/transformations-in-rudder-part-2/

If you are missing a transformation, please feel free to add it to the repository.

Rudder User Transformation

Rudder also supports user-specific transformations for real-time operations like aggregation, sampling, modifying events etc. The following blog describes one real-life use case of the transformation module

https://rudderlabs.com/customer-case-study-casino-game/

Client SDKs

The client SDKs provide APIs collecting events and sending it to the Rudder Backend.

Coming Soon

  1. More performance benchmarks. On a single m4.2xlarge, Rudder can process ~3K events/sec. We will evaluate other instance types and publish numbers soon.
  2. More documentation
  3. More destination support
  4. HA support
  5. More SDKs (or Segment compatibility)
  6. Transformations from UI
You can’t perform that action at this time.