Hodor — open source ETL

På svenska: README.sv.md

Hodor is an open-source ETL tool. It is delivered as a web application (Angular 21 + ASP.NET Core on .NET 10) deployed on-prem, e.g. in OpenShift.

• Fully on-prem. No cloud dependencies, no telemetry, no accounts. Data flows directly from source to target.
• Centralised access. One instance per organisation — everyone models, runs and monitors via the browser.
• Version control built in. Pipelines are JSON files (.hodor) checked into git.
• EUPL-1.2 licence. Strong copyleft with a network clause. See LICENSE.

The Data Vault modeller: source tables dropped on the canvas with auto-classified business keys, an auto-suggested link accepted, and the generated hub/satellite/link model with its DDL and load pipelines — one Apply away from running.

Screenshots

Designer: JOIN two source tables, map columns with transforms, generated SQL	Run history: per-step timings and data-quality warnings, streamed live
Lineage: where data comes from and where it flows, built from all pipelines	Operations & SLA: pipeline health, success rates and schedules

Architecture

┌─────────────────────────────────────────────┐
│  OpenShift pod                              │
│  ┌───────────────────────────────────────┐  │
│  │  Hodor.Api  (ASP.NET Core, .NET 10)   │  │
│  │  • REST API  /api/…                   │  │
│  │  • SignalR   /hubs/runs  (real-time)  │  │
│  │  • Quartz.NET scheduler               │  │
│  │  • Serves Angular SPA                 │  │
│  └───────────────────────────────────────┘  │
│  PostgreSQL/SQL Server  (run history, schedules) │
│  Volume  /pipelines  (.hodor files)         │
└───────────────────────┬─────────────────────┘
                        │
         ┌──────────────┼─────────────┐
         ▼              ▼             ▼
   SQL Server (src)  SQL Server (tgt)  REST API / CSV

Project	Contents
Hodor.Core	ETL engine: PipelineRunner, SqlServerHelper, ApiSource, CsvSource, Transforms, Roslyn/Python scripting
Hodor.Api	ASP.NET Core Web API (.NET 10), SignalR hub, Quartz.NET scheduler, PostgreSQL/SQL Server via EF Core
hodor-web	Angular 21 frontend: SVG canvas designer, run history, workflow/schedule view

Getting started

Try it in 5 minutes (Docker)

The fastest way to a clickable environment — no SDK, no database to install. Starts Hodor + SQL Server with a pre-seeded demo source (HodorDemo), a target warehouse (HodorDW) and two example pipelines ready to run:

docker compose --env-file demo/.env -f docker-compose.yml -f docker-compose.demo.yml up --build
# → http://localhost:5200  (log in: admin / Admin1234!)

--env-file demo/.env supplies the (deliberately weak) demo secrets. The base compose file requires SA_PASSWORD, JWT_KEY and HODOR_ADMIN_PASSWORD with no defaults, so a real deployment can never silently ship known values. See demo/.env. Demo only.

Open Workflows, run Demo_Load_DimCustomer, and watch the steps live. See demo/README.md for what's included.

Prerequisites (building from source)

• .NET 10 SDK
• Node.js 22+ and npm
• SQL Server or PostgreSQL (metadata database, source and/or target)
• Python 3 (optional, for Python scripts inside pipelines)

Build and run locally

# 1. Build the Angular app
cd hodor-web
npm install
npm run build        # output: hodor-web/dist/hodor-web/browser/

# 2. Start the API (also serves the Angular app)
cd ../Hodor.Api
dotnet run
# → http://localhost:5200

For frontend development with hot reload:

cd hodor-web
npm start            # ng serve with proxy to localhost:5200

Configuration

// Hodor.Api/appsettings.json
{
  "Hodor": {
    "PipelinesFolder": "/pipelines"
  },
  "Database": {
    "Provider": "SqlServer",
    "ConnectionString": "Server=...;Database=HodorMeta;..."
  }
}

Environment variables: HODOR__PIPELINESFOLDER, DATABASE__PROVIDER, DATABASE__CONNECTIONSTRING.

Features

Designer

• SVG canvas — drag tables onto the canvas, drag arrows from source columns to target columns.
• Multiple source tables → one target: add several SQL tables and drag a line between key columns to join them (INNER/LEFT). The tool generates the SELECT … FROM … JOIN query.
• Per-column transforms: Trim, UpperCase, LowerCase, EmptyToNull.
• Draft target: build a new target table schema directly on the canvas — drag source columns to the target side, then create the table in the database with one click.
• C# and Python scripts in the transform step (after mappings, before load).
• Post-load SQL runs on the target server after the load (UPDATE, MERGE, EXEC …).
• Variables / parameterisation — declare {{name}} variables and reference them in the extract SQL, post-load SQL, target table name or a request body, so one pipeline is config-driven instead of hard-coded. Built-in macros {{run_date}}, {{run_timestamp}} and {{watermark}} are always available; {{watermark}} resolves to the last successful high-watermark value, which is how an incremental extract filters on it (WHERE ModifiedDate > '{{watermark}}'). An undeclared reference fails the run before any I/O.
• Metadata-driven generation — use a templated pipeline (one with {{variables}}) as a stencil and stamp out one pipeline per row of a binding set: paste rows, or point at a control table (SELECT table_name AS table, schema_name AS schema FROM etl_config WHERE enabled = 1) and each row's columns become the variable values. Preview the result, then generate; the copies group together in the Workflows view. Build one load, run it for fifty tables.

Execution and observability

• Real-time status via SignalR — per-step progress without polling.
• Per-step timing (Extract, Transform, Script, Load, Post-load SQL).
• Error handling — skipBadRows: bulk copy attempt, row-by-row fallback on failure.
• Dead-letter quarantine — with quarantineBadRows, each row skipBadRows skips is written (with its load error, a UTC timestamp and the pipeline label) to a quarantine table that is created at run time on first use and kept across runs, so nothing is lost silently. Defaults to <target>_Quarantine; override with quarantineTable.
• Data-quality expectations — per-column rules (NotNull, NotBlank, Regex, Range, InSet, MaxLength) checked on the mapped rows of the primary target before they load. A violated Fail rule aborts the run before the bad chunk loads (so a non-streaming load rolls back entirely); Warn rules just record their violation counts on the run and in notifications.
• Transaction around DELETE / bulk load with TRUNCATE fallback.

Workflows and scheduling

• The Workflows view lists all saved pipelines with cron schedule and enable/disable toggle.
• Quartz.NET scheduler runs inside the server — pipelines fire even when no browser is open.
• Run history with per-step details in the Runs view.

Data Vault automation

The Data Vault view is a visual modeller: drag several source tables onto a canvas and design the whole vault at once.

1. Connect and drag tables in — double-click source tables to drop them on the canvas. Each column is auto-classified as business key (primary key) or descriptor; click the role chip (BK/D/—) to change it.
2. Draw relations — drag from a column to another table to create a link; the tool also auto-suggests links from matching key names. Each relation becomes a Lnk_ between the two hubs.
3. Generate — for every table the tool builds:
   • stg_<Entity> — a stage table with the source columns plus persisted computed hash columns (hash key, hash diff, link hashes) and LoadDate/RecordSource
   • Hub_<Entity> with hash key, business key(s), LoadDate, RecordSource
   • Sat_<Entity> with HashDiff for descriptor CDC
   • Lnk_<Entity>_<Ref> for each drawn relation
   • a stage load (source → stage) plus one runnable Hodor pipeline per vault table The result is shown as a model diagram, DDL and the generated pipelines.
4. Apply — creates the schemas and tables and saves the pipelines, which appear in the Workflows view (grouped per entity) ready to run or schedule.

Hashing happens in the stage layer (DV 2.0 "hash in stage"). Because the hub, satellite and link loads read from the stage table, they are insert-only, idempotent and independent of each other and of load order — they can run at any time and in parallel. Only the stage and the vault need to share a database; the source can live anywhere (a lake, CSV, REST API, another server), since only the stage load touches it.

Deploying to OpenShift / Kubernetes

The real Dockerfile (multi-stage: Angular build → .NET build → runtime, port 5200) lives at Dockerfile. A full Helm chart is available under charts/hodor — it creates a Deployment, Service, an OpenShift Route (or Ingress on plain Kubernetes), a PersistentVolumeClaim for /data (pipelines + Data Protection keys), and a Secret for the database password, JWT key, OIDC client secret and SMTP password:

helm install hodor charts/hodor \
  --set database.provider=Postgres \
  --set database.connectionString="Host=postgres;Database=hodor;Username=hodor;Password=..." \
  --set defaultAdmin.password="ChangeMe123!"

See charts/hodor/README.md for the full set of values, including how to point at your own Secret (existingSecret) or enable SMTP run notifications (email.enabled=true) or Microsoft Teams notifications (teams.enabled=true, teams.webhookUrl=…). Teams posts run outcomes (including any data-quality warnings) to one channel's incoming webhook, on the same per-pipeline opt-in as email.

Security

• Use Windows authentication (Trusted_Connection=True) for on-prem SQL Server — .hodor files will contain no secrets and can be checked into git safely.
• CORS is open by default in development. Restrict with AllowedOrigins in production.
• Report vulnerabilities via SECURITY.md.

Licence and contributing

Hodor is licenced under EUPL-1.2 — see LICENSE. Contributions are welcome from everyone. See CONTRIBUTING.md.