Ingenii Databricks Platform

Details

• Base image: databricksruntime/standard:9.x
• Registry: ingeniisolutions
• Repository: databricks-runtime
• Version: 0.6.4

Intermediate Images

• Base OS Repository: databricks-runtime-base-os
• Base OS Version: 0.1.0
• Base Python Repository: databricks-runtime-base-python
• Base Python Version: 0.1.0

Overview

This image is used with Databricks' Container Services to customise the cluster runtime in the engineering cluster of in the Ingenii Data Platform. This contains an installation of dbt and Ingenii's python package for data engineering.

Data Pipeline Overview

For an overview of the data pipeline and the stages it goes through, please refer to the Data Pipeline documentation

dbt Integration

For reading files and testing data we use dbt as a framework. For an explanation on how we use dbt and how to set up your own data sources, please refer to the Ingenii Data Engineering Example repository.

Contributions

• dbt-spark - retrying when any connections fail