Optimus

Optimus helps your organization to build & manage data pipelines with ease.

Features

• BigQuery
  • Schedule SQL transformation
  • Query compile time templating (variables, loop, if statements, macros, etc)
  • Table creation
  • BigQuery View creation
  • Automatic dependency resolution: In BigQuery if a query references tables/views as source, jobs required to create these tables will be added as dependencies automatically and optimus will wait for them to finish first.
  • Cross tenant dependency: Optimus is a multi-tenant service, if there are two tenants registered, serviceA and serviceB then service B can write queries referencing serviceA as source and Optimus will handle this dependency as well
  • Dry run query: Before SQL query is scheduled for transformation, during deployment query will be dry-run to make sure it passes basic sanity checks
  • Sink BigQuery tables to Kafka [using additional plugins]
• Extensibility to support Python transformation
• Git based specification management
• REST/GRPC based specification management
• Multi-tenancy
• Pluggable transformation

Getting Started

Optimus has two components, Optimus service that is the core orchestrator installed on server side, and a CLI binary used to interact with this service.

Compiling from source

Prerequisites

Optimus requires the following dependencies:

• Golang (version 1.16 or above)
• Git

Run the following commands to compile optimus from source

git clone git@github.com:odpf/optimus.git
cd optimus
make build

Use the following command to run

./optimus version

Optimus service can be started with

./optimus serve

serve command has few required configurations that needs to be set for it to start. Configuration can either be stored in .optimus.yaml file or set as environment variable. Read more about it in getting started.

Installing via brew

You can install Optimus using homebrew on macOS:

brew install odpf/taps/optimus
optimus version

Stability and Compatibility

Optimus is currently undergoing heavy development with frequent, breaking API changes.

⭐ Current major version is zero (v0.x.x) to accommodate rapid development and fast iteration while getting early feedback from users (feedback on APIs are appreciated). The public API could change without a major version update before v1.0.0 release.

Credits

This project exists thanks to all the contributors.

License

Optimus is Apache 2.0 licensed.