Designed by Agile Lab, Witboost is a versatile platform that addresses a wide range of sophisticated data engineering challenges. It enables businesses to discover, enhance, and productize their data, fostering the creation of automated data platforms that adhere to the highest standards of data governance. Want to know more about Witboost? Check it out here or contact us!
This repository is a guide to our Starter Kit meant to showcase Witboost's integration capabilities and provide a "batteries-included" product.
A Specific Provisioner is a microservice which is in charge of deploying components that use a specific technology. When the deployment of a Data Product is triggered, the platform generates it descriptor and orchestrates the deployment of every component contained in the Data Product. For every such component the platform knows which Specific Provisioner is responsible for its deployment, and can thus send a provisioning request with the descriptor to it so that the Specific Provisioner can perform whatever operation is required to fulfill this request and report back the outcome to the platform.
You can learn more about how the Specific Provisioners fit in the broader picture here.
We provide two main kinds of projects:
- Provisioners: these are actual implementations for a specific technology that you can customize to suit your needs
- Scaffolds: these are projects that you can start from if you want to implement a provisioner yourself
| Tech | Kind | Project | Scope | Supported components | Notes |
|---|---|---|---|---|---|
 |
Provisioner | Airbyte SP | ELT - Airbyte | Workload | |
 |
Provisioner | CDP Impala SP | SQL Query Engine - CDP Impala | Output Port | |
 |
Provisioner | CDP S3 SP | Object Storage - CDP S3 | Output Port | |
 |
Provisioner | CDP Spark SP | Data Processing - CDP Spark | Workload | |
 |
Provisioner | MWAA SP | Scheduling - Airflow/MWAA | Workload | |
 |
Provisioner | Hasura SP | GraphQL - Hasura | Output Port | Needs the Hasura Authentication Webhook and Role Mapper |
 |
Provisioner | Snowflake SP | DWH - Snowflake | Output Port, Storage Area | |
 |
Scaffold | Java Scaffold | Generic - Java | NA | |
 |
Scaffold | Python Scaffold | Generic - Python | NA | |
 |
Scaffold | Terraform Scaffold | Generic - Terraform | NA |
A Template is a tool that helps create components inside a Data Mesh. Templates help establish a standard across the organization. This standard leads to easier understanding, management and maintenance of components. Templates provide a predefined structure so that developers don't have to start from scratch each time, which leads to faster development and allows them to focus on other aspects, such as testing and business logic.
For more information, please refer to the official documentation.
| Tech | Component | Project | Scope | Specific Provisioner | Notes |
|---|---|---|---|---|---|
 |
Data Product | Data Product | NA | No Specific Provisioner needed | |
|
Output Port | CDP CDW Impala Output Port | SQL Query Engine - CDP CDW Impala | CDP Impala SP | |
|
Output Port | CDP DL S3 Output Port | Object Storage - CDP DL S3 | CDP S3 SP | |
|
Output Port | Hasura Output Port | GraphQL - Hasura | Hasura SP | |
|
Output Port | Snowflake Output Port | DWH - Snowflake | Snowflake SP | |
|
Storage Area | Snowflake Storage Area | DWH - Snowflake | Snowflake SP | |
|
Workload | Airbyte Workload | ELT - Airbyte | Airbyte SP | |
|
Workload | CDP CDE Spark Workload | Data Processing - CDP CDE Spark | CDP Spark SP | |
 |
Workload | DBT Workload | Data processing - DBT | No Specific Provisioner needed | |
|
Workload | MWAA Workload | Scheduling - Airflow/MWAA | MWAA SP |
This project is available under the Apache License, Version 2.0; see LICENSE for full details.
Agile Lab creates value for its Clients in data-intensive environments through customizable solutions to establish performance driven processes, sustainable architectures, and automated platforms driven by data governance best practices.
Since 2014 we have implemented 100+ successful Elite Data Engineering initiatives and used that experience to create Witboost: a technology-agnostic, modular platform, that empowers modern enterprises to discover, elevate and productize their data both in traditional environments and on fully compliant Data mesh architectures.
Contact us or follow us on: