data-fabric-patterns

Open-source reference for self-service ingestion, DDL/DML generation, and governed data-product delivery on a shared data fabric.

Inspired by enterprise trusted-data-fabric patterns: config-driven pipelines, platform standards, and medallion layering — without proprietary connectors or employer-specific implementations.

Architecture

flowchart LR
  subgraph domains [DomainTeams]
    D1[Insurance]
    D2[Finance]
  end
  subgraph fabric [DataFabric]
  B[Bronze]
  S[Silver]
  G[GoldProducts]
  end
  subgraph consume [Consumption]
    SQL[SQL_API]
    BI[BI_ML]
  end
  D1 --> B
  D2 --> B
  B --> S --> G --> SQL --> BI

Loading

Features

• YAML pipeline specs validated with Pydantic
• DDL generator (Jinja2 templates for Delta/Iceberg-style tables)
• Standardised DAG naming for Apache Airflow estates
• Insurance/finance example with synthetic CSV data
• Platform standards guide for naming, releases, and GDPR-aligned PII handling

Quickstart

# Install and run tests
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
pytest -q

# Render DDL from example pipeline spec
python -c "
from pathlib import Path
from data_fabric_patterns.generators.ddl import DDLGenerator
from data_fabric_patterns.ingestion.spec import load_pipeline_spec
spec = load_pipeline_spec('examples/insurance-finance/pipeline.yaml')
print(DDLGenerator().render_create_table(spec))
"

# Optional: local Airflow (requires Docker)
export AIRFLOW_UID=$(id -u)
docker compose up -d
# UI: http://localhost:8080  (admin / admin)

Project layout

src/data_fabric_patterns/   # Python package
examples/insurance-finance/   # Sample domain + pipeline.yaml
docs/platform-standards.md  # Naming, deployment, release conventions
airflow/dags/               # Example DAG wired to pipeline spec
tests/                      # Unit tests

Related work

Part of the Essid Solutions data platform portfolio:

• lakehouse-terraform-kit
• airflow-platform-standards
• data-contracts-governance

License

MIT — see LICENSE.