Skip to content

roypulseai/kelpmesh

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

63 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

KelpMesh

Code-native data transformation — SQL and Python models, no Jinja required.

License: Apache 2.0 Python 3.11+

pip install KelpMesh
kelpmesh init my_project && cd my_project
kelpmesh run

📖 Full user guide at docs/user-guide.md — covers CLI, Studio, VS Code, migration, and all features.


Why KelpMesh?

dbt models are Jinja-templated SQL. Your files look like this:

{{ config(materialized='table') }}
SELECT customer_id, {{ surrogate_key(['order_id','sku']) }} AS id
FROM {{ ref('stg_orders') }}
WHERE {{ is_incremental() }}

No IDE autocomplete. No AI assistant support. Weeks to learn.

KelpMesh models are plain SQL:

-- materialized: table
SELECT customer_id, surrogate_key(order_id, sku) AS id
FROM stg_orders
WHERE is_incremental()

Works in every SQL editor, every AI tool, every linter — on day one.


Features

SQL engine

  • Pure SQL models — no Jinja; files work natively in DBeaver, DataGrip, Cursor, any AI tool
  • 32 built-in SQL macrossurrogate_key, safe_divide, datediff, haversine, generate_surrogate_key, and 27 more; called as plain SQL functions, expanded at compile time
  • Python modelsdef model(dbt, session) interface; return SQL string or pandas DataFrame
  • All materializationstable, view, incremental (merge/append), ephemeral, SCD Type 2 snapshots
  • 9 warehouse adapters — DuckDB · Postgres · Snowflake · BigQuery · Databricks · Redshift · Microsoft Fabric · MySQL · Trino
  • Incremental merge — consistent MERGE/UPSERT API across all 9 warehouses
  • kelpmesh plan — Terraform-style impact analysis before any warehouse query runs
  • kelpmesh compile — render macros and substitutions without touching the warehouse

CI/CD and version control

  • kelpmesh ci — one command: diff → plan → run changed models → test → post PR comment
  • GitHub PR comments — posts a structured run report on every PR; updates (not spams) existing comment
  • GitLab MR comments — same, auto-detected from CI environment variables
  • Bitbucket PR comments — same
  • Ready-to-use CI templates.github/workflows/ci.yml, .gitlab-ci.yml, bitbucket-pipelines.yml included

Scheduling and orchestration

  • Built-in cron schedulerkelpmesh schedule start; cron syntax + every 1h intervals; no Airflow required
  • Dagster integrationKelpMeshResource, @asset, @op, schedule sensor
  • Prefect integrationKelpMeshBlock, @task, @flow, pre-built standard flow
  • AirflowKelpMeshOperator, DagFactory-compatible

Security (free in Core)

  • PII auto-classification — 7 types: email, phone, SSN, NPI, card, IP, DOB
  • Row-level security (RLS) — policy-based access control applied per model
  • Column masking — per-column masking strategy; format-preserving options
  • GDPR right-to-erasurekelpmesh security erasure deletes subject data across all models
  • Immutable audit log — append-only JSONL; every execution recorded
  • Secret scanning — scans for hardcoded credentials in SQL files
  • Zero telemetry — import blocklist enforced in code; no phone-home possible

Tooling

  • VS Code extension — 37 SQL snippets, model tree view, CodeLens run/test/preview buttons, plan panel
  • dbt migrationkelpmesh import converts models, tests, sources, seeds, snapshots
  • Data mesh — cross-project ref(), public/private access contracts, column guarantees
  • Semantic layer — metric definitions, BI export for LookML, Tableau, PowerBI, Qlik

Quickstart

# Install
pip install KelpMesh

# Create a project
kelpmesh init my_project
cd my_project

# Run all models
kelpmesh run

# Preview 100 rows from a model
kelpmesh preview orders

# See what would change before running
kelpmesh plan

# Test all models
kelpmesh test

# Run + test in one command
kelpmesh build

# Generate documentation site
kelpmesh docs

CI/CD — one line in any pipeline

kelpmesh ci    # detects changed models, runs them, tests, posts PR comment

In GitHub Actions — add to your workflow:

- name: KelpMesh CI
  run: kelpmesh ci
  env:
    GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Every PR gets a structured report showing exactly which models were affected, what ran, and whether tests passed — updated automatically on every push.


Packages

Package What it is Price
pip install KelpMesh Full CLI engine — 9 adapters, 32 macros, scheduler, security, CI/CD Free forever · Apache 2.0
pip install kelpmesh-studio Core + browser dashboard (DAG viz, run history, team features) Free for personal use · Freemium for commercial

KelpMesh will never have features gated behind a paid tier.


Studio (browser dashboard)

pip install kelpmesh-studio
kelpmesh-studio          # opens http://localhost:8765
Tier Price Limits
Free — personal use $0 1 user · 3 projects · 50-run history
Pro — commercial teams $29/user/mo 5 seats · unlimited projects · RBAC · API keys · Git sync · AI BYOK
Business — large orgs $79/user/mo Unlimited seats · SSO · BYOC · full audit log
Enterprise Contact us On-premises · custom SLA · dedicated support

Activate with:

export KELPMESH_STUDIO_LICENSE_KEY=km_pro_<your-key>
kelpmesh-studio

Warehouse support

# kelpmesh.yml
warehouse:
  type: duckdb          # zero install, great for local development
  path: ./warehouse.db

# Per-warehouse extras:
# pip install KelpMesh[postgres]
# pip install KelpMesh[snowflake]
# pip install KelpMesh[bigquery]
# pip install KelpMesh[databricks]
# pip install KelpMesh[redshift]
# pip install KelpMesh[mysql]
# pip install KelpMesh[trino]
# pip install KelpMesh[all-warehouses]   # everything at once

Security

# Scan for hardcoded credentials
kelpmesh scan secrets --fail

# Classify PII columns in your warehouse
kelpmesh security classify --table orders

# Apply column masking by role
kelpmesh security mask --table users --columns email,phone --role viewer

# GDPR erasure
kelpmesh security erasure --id-col email --id-value user@example.com

# View audit trail
kelpmesh security audit

Migrate from dbt or SQLMesh

KelpMesh auto-detects the source project format. No --from flag needed in most cases.

# From dbt
kelpmesh import ./dbt-project --output ./kelpmesh-project

# From SQLMesh
kelpmesh import ./sqlmesh-project --output ./kelpmesh-project

# Explicit (if auto-detection doesn't pick it up)
kelpmesh import ./my-project --from dbt
kelpmesh import ./my-project --from sqlmesh

dbt → KelpMesh

dbt KelpMesh
{{ ref('model') }} plain table name (auto-resolved by AST)
{{ source('src', 'tbl') }} plain table name
{{ config(materialized='table') }} -- { materialized: table } header
schema.yml tests (not_null, unique, accepted_values, relationships) SQL assertion files in tests/
Seeds (CSV) seeds/*.sql with VALUES blocks
Snapshots models/snapshots/*.sql with snapshot header
dbt packages Built-in SQL macros or kelpmesh-utils

SQLMesh → KelpMesh

SQLMesh KelpMesh
MODEL (name ..., kind FULL) block -- { materialized: table } header
kind INCREMENTAL_BY_UNIQUE_KEY (unique_key id) -- { materialized: incremental, unique_key: id }
grain payment_id -- { unique_key: payment_id }
audits (UNIQUE_VALUES(...), NOT_NULL(...)) SQL assertion files in tests/
cron '@daily' -- { cron: @daily }
@execution_dt CURRENT_DATE
@start_dt / @end_dt CURRENT_DATE - INTERVAL '7 days' / CURRENT_DATE
YAML unit test fixtures (inputs/outputs) Smoke tests (recreate with kelpmesh create_test)
audits/*.sql tests/*.sql SQL assertions
config.py / config.yaml kelpmesh.yml

SQLMesh note: SQLMesh YAML unit tests use an input/output fixture format that requires running a live warehouse query to generate. After import, use kelpmesh create_test <model> to recreate them properly.


Integrations

Tool How
GitHub Actions kelpmesh ci + built-in workflow at .github/workflows/ci.yml
GitLab CI kelpmesh ci + template at .gitlab-ci.yml
Bitbucket Pipelines kelpmesh ci + template at bitbucket-pipelines.yml
Dagster from kelpmesh_dagster import KelpMeshResource
Prefect from kelpmesh_prefect import KelpMeshBlock
Airflow pip install kelpmesh-airflow
VS Code Search "KelpMesh" in extensions marketplace
pre-commit .pre-commit-hooks.yaml included

Community


Development

git clone https://github.com/RoyPulseAI/kelpmesh
cd kelpmesh
pip install -e ".[dev]"
pytest tests/ -v

See CONTRIBUTING.md.


Author

KelpMesh is designed and built by Roy Pulse AI (@RoyPulseAI).

License

KelpMesh — Apache 2.0
kelpmesh-studio — Apache 2.0 (free tier) · Commercial license (Pro/Business/Enterprise)

See LICENSE.

About

Code-native data transformation platform — SQL & Python models, open-source alternative to dbt

Resources

License

Code of conduct

Contributing

Security policy

Stars

1 star

Watchers

0 watching

Forks

Packages

 
 
 

Contributors