v0.19.0 — Project triggers and sub-projects

Start a project when a file arrives, a table changes, a webhook is called or another project ends — and let one project run another — without an external scheduler or message broker.

pip install -U hydra-etl

New features

Project triggers — triggers.yaml

• A triggers.yaml file at the project root, seven types:

  Type	Starts the project when…	Parameters received
  cron	the schedule matches (cron, IANA timezone)	—
  interval	every every	—
  file	a file matching path (glob, **) is added or modified	trigger.file, trigger.files
  marker	a marker file such as inbox/READY appears (removed unless consume: false)	—
  source_change	a job source changes (check: count | max | checksum)	trigger.value, trigger.previous
  webhook	POST …/triggers/{name}/webhook with a valid token	none (nothing from the request)
  after_project	another project ends (on: success | failure | any)	trigger.project, trigger.status

• Common keys: env, workflows (default: the thread), params (no secret), overlap (skip by default), timeout, retry: {max, delay}, enabled, description. A trigger always runs the project in its own process, under .hydra/run.lock.

• hdrctl watch [PROJECT]... [--env] runs the triggers of one or more projects. It prints each trigger and its next run, then every start and end.

• hdrctl serve runs the same engine for every project of the Studio list.

• hdrctl trigger list | validate | fire <name> | history | token <name> [--rotate]. validate checks workflows, called projects, sources, webhook secrets and refuses after_project loops. Every run is recorded in .hydra/triggers/history.jsonl.

• Guarantees:

  • a trigger still running is not started again (the missed run is recorded as skipped); runs missed while the engine was stopped are not replayed;
  • triggers.yaml is reloaded without restart; an invalid file keeps the previous triggers running;
  • a file is taken only once unchanged for debounce (no half-copied file); a file arriving during a run starts the next run;
  • a source_change measure runs apart from the engine (a slow database does not hold up other triggers); on PostgreSQL, MySQL/MariaDB and SQL Server count and max are a single SELECT COUNT(*) / SELECT MAX(col);
  • webhook tokens are ${SECRET:…} references, compared in constant time (401 / 404 / 503); keep hdrctl serve on 127.0.0.1 unless an HTTPS reverse proxy stands in front;
  • first Ctrl+C on hdrctl watch waits for running projects; a second stops at once.

• Cron expressions are parsed by Hydra itself (no extra package); tzdata is installed on Windows.

• Studio — Project settings › Triggers: triggers with event, environment, next run, running state and last run; a form per type; enable/disable, delete, Run now; create or rotate a webhook token (shown once, with PowerShell and curl calls); run history. An invalid change is not written; the previous file is backed up to ~/.hydra/_backups.

• API: GET|PUT /api/projects/{id}/triggers, …/triggers/history, POST …/triggers/{name}/fire, …/{name}/webhook, …/{name}/token.

Sub-projects — type: project step

• A workflow of a "master" project runs another project: its thread.yaml, or only the workflows listed. Keys: project (relative path, or ref: a Studio project), env (inherit by default), params, variables, workflows, outputs, isolation (inprocess or subprocess), timeout, overlap (wait, skip, fail).
• Nothing is passed implicitly except the environment; the called project keeps its own secrets, and ${SECRET:…} is refused in params and variables.
• Checked before anything runs (hdrctl run, --dry-run, workflow validate, Studio Check): project not found or ambiguous, cycles (a → b → a), more than 5 levels. workflow validate prints the tree of called projects.
• Results readable by the next steps: {{ project:STEP.status }}, .rows, .duration, .param.NAME.
• A called project runs once at a time (.hydra/run.lock, also between processes; a stale lock is taken over).
• hdrctl run <project> -w <workflow> (repeatable): run only these workflows.
• Studio — Project node: a Projects section in the palette; a cycle is refused with its path; one window sets path, environment, workflows, parameters, outputs, process, timeout and overlap.

Tools

• VS Code extension 0.1.8 (type: project) and 0.1.9 (triggers.yaml: the seven types, token as a secret reference only, 8 snippets).
• MCP: hydra_project_calls and hydra_project_triggers (read-only).
• API: POST /api/projects/{id}/project-step.

Patches and fixes

• Workflow actions lost the values of environments/<env>.yaml when a required parameter was given at run time (-P, or params of a project step): {{ param:X }} fell back to the defaults while the jobs of the same run used the environment.
• hdrctl workflow run refuses project steps that form a cycle, call a missing project or nest more than 5 levels, before anything runs.
• An optional {{ env:NAME }} or a parameter without value inside a text becomes an empty string instead of None.
• Parameter errors are in English: missing required parameter, unresolved parameter, missing environment variable.

Documentation

• White paper (FR/EN): 1.4 Sub-projects and 1.5 Project triggers. README and 03-workflows.md updated.

Full changelog: v0.18.0...v0.19.0

v0.18.0 — Environments, global variables and secrets per environment

One project, several environments: global variables shared by the machine, parameter values and secrets per environment, managed from the CLI or Studio.

pip install -U hydra-etl

New features

• Global environment variables, shared by every project of the machine: ~/.hydra/env (all environments) and ~/.hydra/env.<env> (one environment), selected by --env / HYDRA_ENV; location HYDRA_HOME. Order, weakest first: env < env.<env> < project .env < job .env < --env-file. When a run switches environment (Studio, API), the variables of the previous one are removed. Workflow actions see these variables and the project .env from the first step.
• Secrets per environment: .hydra/secrets.<env> (plain or SOPS-encrypted) wins over .hydra/secrets with --env <env>; sops.file accepts {env}. The .gitignore written by hdrctl init ignores .hydra/secrets.* but keeps secrets.yaml and encrypted files.
• --env-file FILE on hdrctl run and hdrctl workflow run: extra variables for one run, above every other layer.
• hdrctl workflow run accepts --env and -P/--param.
• New commands:
  • hdrctl env list|show|create|rename|delete|set|unset — environments and global variables; rename and delete also move parameter values and secrets, after a backup to ~/.hydra/_backups/;
  • hdrctl param list|set|unset — parameters, default value and value per environment;
  • hdrctl secret list|set|delete — the value is typed hidden or read from stdin, never passed as an argument; list shows names only.
• Studio:
  • active environment selector on the project page and in the editor, used by every run and by the Expression Assistant;
  • Global settings › Environments: create, rename, delete environments and edit their global variables;
  • Project settings: tabs Execution order, Parameters (one column per environment) and Secrets (names only).
• parameters.yaml accepts the short form name: value (it stopped the run with an internal error).
• VS Code extension 0.1.7: schemas and snippets for parameters.yaml, environments/<env>.yaml and .hydra/secrets.yaml.
• MCP: hydra_project_settings (read-only; never a secret or variable value); hydra_run_workflow accepts env.
• API: /api/global-env; /api/secrets accepts env; inline job and action runs accept env.

Patches and fixes

• hdrctl run jobs/<job> reads the .env at the project root, not only the one of the job's parent folder.
• bash, powershell and python actions without working_dir run in the project root; a relative working_dir is resolved from there. Scheduled and webhook runs started by hdrctl serve used to run in the server's folder, so relative paths failed.
• Studio: a project whose .hydra/project.json had no id got the id ""; the id is now added to the file.
• Studio: the schedule workflows of a project opened in Studio are planned at once, not at the next start of hdrctl serve.
• Studio: runs with no environment selected sent env: "default"; "default" now means no environment.

Full changelog: v0.17.0...v0.18.0

v0.17.0 — Secrets masked everywhere, SOPS, window and text_to_rows

New features

Secrets

• Secrets are masked everywhere Hydra shows or keeps text: logs (every logger, step logs, API), error messages, tracebacks, the output of bash, powershell, python and ssh actions, webhook and email log lines, the rollback report and the runs stored by the API. URL-encoded and JSON-escaped forms are masked too. Values sent to connectors, scripts and webhooks are unchanged.
• Visibility, as in GitLab: plain (debugging, with a warning), masked (default, [MASKED]) or hidden (lines containing a secret are dropped and counted). Set it in .hydra/secrets.yaml, with HYDRA_SECRETS_VISIBILITY or --secrets-visibility. Values shorter than 4 characters are not masked; a warning names them.
• .hydra/secrets.yaml: project secrets settings, with no secret inside, safe to commit (visibility, provider). Unknown keys and providers are reported, never ignored.
• SOPS provider (provider: sops; age, PGP, cloud KMS): secrets decrypted in memory on first use; nested keys read as ${SECRET:db.password}. A SOPS-encrypted .hydra/secrets or --secrets-file is detected and decrypted as well.
• Provider interface (SecretProvider, register_provider) for external vaults.

Transformations

• join with how: cross (cartesian product), guarded by max_rows (default 10,000,000 rows produced): the run stops before the batch that would exceed it.
• text_to_rows: one row per value of a text column ("a;b;c" → 3 rows). Options sep, regex, trim, keep_empty, index_column. No input row is lost.
• window: window functions that add columns and keep every row in order — rank (row_number, rank, dense_rank), offset (lag, lead), running (cumsum, cumcount, cummin, cummax) and whole-partition (sum, mean, min, max, count, first, last). Runs in DuckDB with the large-volume engine, same results as pandas.

Studio and tools

• Expression Assistant: lists the project's parameters (with their value in the active environment), environment variables and, on reference fields, secrets. Double-click or drag an item; Insert writes it into the field. Opens from fx on expressions and from a new {x} button on fields such as paths, host, SQL query, URL or timeouts. Numeric fields accept a reference.
• Text to Rows and Window nodes; the Join node offers cross with a Max rows field.
• DSL specification: 20 operations. MCP: hydra_list_operations lists them, hydra_explain_error explains the new errors, hydra_check_job warns when a request needs window, text_to_rows or join.
• VS Code extension 0.1.6: schema and snippets for the additions.

Patches and fixes

• Studio: the Script node showed a generic icon in the palette and its dialog.
• Studio: the MySQL and PostgreSQL nodes show the official logos; the SQL Server node shows a database icon.

Full changelog: v0.16.1...v0.17.0

v0.16.1 — MongoDB schema drift policy now applies

New features

• None.

Patches and fixes

• MongoDB source: schema.drift_policy now applies. The schema block never reached the connector, and documents were normalised before drift detection, so fail / warn / adapt / ignore had no effect. Now:

  • fail raises SchemaDriftError naming the field and the document;
  • warn logs each new field once, plus a summary;
  • adapt extends the schema;
  • ignore stays silent.

  schema.sample_size is honoured. Regression tests: tests/test_mongodb_drift.py.

Full changelog: v0.16.0...v0.16.1

v0.16.0 — engine: key per job, right and outer join in DuckDB

New features

• engine: auto | pandas | duckdb in transformations.yaml (at the root or under transformations:) chooses the engine of a job's global operations; HYDRA_ENGINE still takes precedence. Studio: Engine selector on a job's canvas, kept on round-trips with the YAML files. JSON schemas (Hydra DSL, VS Code extension 0.1.5) updated.
• Right and outer join run in DuckDB in the large-volume engine, with the same rows, columns, order and types as pandas; only the columns the following aggregation needs are copied. 20 million rows, right join then aggregation (2 CPUs): 71.8 s / 3.4 GB → 27.7 s / 2.6 GB.

Patches and fixes

• sort is stable: rows with equal sort keys keep their input order (it already was for several columns, not always for one).

Full changelog: v0.15.0...v0.16.0

v0.15.0 — Large-volume engine: 20 M rows in 34 s instead of 113 s

New features

• Large-volume engine (optional: pip install "hydra-etl[duckdb]", which now also installs pyarrow). Above HYDRA_LARGE_VOLUME_ROWS estimated rows (2,000,000 by default), a job with a global operation reads its CSV source with pyarrow and runs aggregate and sort in a temporary on-disk DuckDB database. pandas keeps every other step and stays the default below the threshold.
• HYDRA_ENGINE=auto | pandas | duckdb forces the choice.
• No silent fallback: a global step DuckDB cannot translate goes back to pandas with a log message; the fast reader hands over to the Python reader with a warning.
• Measured on 20 million rows (2 CPUs): 112.8 s / 5.4 GB → 34.3 s / 1.4 GB, identical results.
• --profile shows the engine chosen and why.

Patches and fixes

• DuckDB sums are byte-identical to pandas (compensated, ordered summation), so switching engine never changes a result.

Full changelog: v0.14.0...v0.15.0

v0.14.0 — Fallback after a failure, timeout per workflow

New features

• Fallback after a failure (fallback.on_failure: true): a workflow that fails is replaced by its first backup, only if it left nothing published — no load published and no outside action done (except actions marked undo: none), or everything undone by rollback: true. If the backup fails too, the next one in use runs. Otherwise the project stops and says why: ⛔ load_crm: fallback load_crm_backup not started — step 'load1' had already published its data. The replaced run is reported as recovered (CLI ↯, Studio badge replaced by …); the project succeeds if the backup does.
• timeout per workflow in thread.yaml (30s, 5m, 1h): beyond it the workflow is stopped at the next step or batch (an all-or-nothing load is cancelled; an action already started finishes) and counts as failed, which can trigger the fallback. Shown by --dry-run, in Studio, in the VS Code schema and in the DSL reference.
• hdrctl run --dry-run and Studio's Check warn when a workflow with on_failure publishes in several steps without rollback: true.
• Studio — wait field (delay_before) on each workflow of the execution order; the Preview shows the wait above the block, the run page a waited … badge. The run API returns waited on each step.
• Studio — Settings has two tabs: Global settings (editor preferences) and Project settings (execution order, fallbacks, loops, written to workflows/thread.yaml).
• Studio — the project page lists the workflows with Project settings and Run project (the project is checked first; errors link to Project settings).

Patches and fixes

• Studio: the run page of a workflow kept by its condition, then replaced after a failure, is shown correctly.
• Studio: a workflow opened from the Preview opens on its Workflow configuration tab and returns to Project settings.
• Studio asks before leaving the execution order with unsaved changes.
• CLI: durations of whole minutes or hours are shown as 5m, 1h.

Full changelog: v0.13.0...v0.14.0

v0.13.0 — Fallback workflows, all-or-nothing loads and workflow rollback

A failed or risky run no longer leaves half-written data behind: each job loads all or nothing, a workflow can undo what it did, and a backup workflow can take over before the start.

New features

Declarative fallback (thread.yaml)

• A workflow can declare backup workflow(s): use (tried in order), when (condition checked just before the start) and on_probe_error (keep by default, or replace).
• when uses a closed language analysed by Hydra, never Python eval: source_available, destination_available, file_exists, file_size, file_age, url_ok, secret_defined, db.load, previous.status, previous.rows, and/or/not, comparisons, durations (24h) and sizes (5MB).
• At run time, quick probes with a time limit (HYDRA_PROBE_TIMEOUT, 15 s) read no data. True → the first backup runs in place (↪ Fallback: a → a_backup — <reason>).
• hdrctl run <project> --use-fallback <workflow> forces a fallback to rehearse a recovery.
• hdrctl workflow add <name> --fallback-for <workflow>; workflow list shows backups; workflow remove cleans the references.
• --dry-run and Studio's Check check backups, their jobs and the sources/destinations named in when, and warn when a backup writes elsewhere.
• Studio: fallback editor in Define (condition checked as you type, helper to insert functions and project resources), backups drawn as dashed blocks in Preview, the decision and its reason on the run page.

All-or-nothing load per job

• A job writes its destination through one load session (begin → stage → publish | abort), common to every connector. Nothing is visible before the end of the job; a failure or Ctrl+C leaves the destination as it was.
• On by default for replace (PostgreSQL / SQL Server: TRUNCATE in one transaction; MySQL / MariaDB: DELETE in one transaction, InnoDB; CSV / Parquet: temporary file then swap) and for CSV / Parquet append.
• load.atomic: true | false in destinations.yaml; true on a connector that cannot honour it is an error before anything is read.
• Two jobs of the same run writing the same destination take turns.

Workflow rollback

• rollback: true in workflow.yaml: if the workflow fails, the steps already done are undone, last first. Files restored, CSV append truncated back, replaced tables restored, rows added by append/upsert removed and changed rows restored (PostgreSQL, MySQL / MariaDB, SQL Server).
• Actions declare undo: (an action, or none), required for webhook, email, bash, powershell, ssh, python.
• A destination changed by someone else since the run is not undone: the rest is kept in .hydra/undo/<workflow>.json and the workflow refuses to run until hdrctl workflow undo <workflow> finishes it.
• Studio: Rollback on/off in the toolbar, Undo section on actions, undone steps on the run page.

Also

• delay_before on a workflow: wait before starting it (staggered starts in a parallel stage).
• API: GET /api/projects/{id}/conditions, POST /api/projects/{id}/conditions/check; project run steps carry a fallback object.
• VS Code schema and DSL reference updated.

Patches and fixes

• Studio: a job whose operations were written as top-level steps: in transformations.yaml (the form of hdrctl init and the CLI) lost them when saved. Both forms are now read.
• A job keeps one connection / one open file for all its batches; Parquet no longer copies the whole file for every batch.
• Without all-or-nothing, a failed job logs that the destination keeps the batches written before the failure.

Full changelog: v0.12.3...v0.13.0

v0.12.3 — Studio and disk stay in sync, validate counts every error

New features

• Studio and the job files stay in sync, both ways. Studio keeps a job and its jobs/<name>/*.yaml files in step, and the latest change wins, made in Studio or outside (editor, hdrctl, git).
  • The sync button (Hydra DSL panel, or Sync with disk in the toolbar) compares canvas and files: only the disk changed → the canvas is rebuilt (node positions kept); only Studio changed → the files are written; both changed → the most recent change wins.
  • Automatic sync of the open job every few seconds (Settings, default 5 s, 0 = off), and when a job is opened.
  • Formatting and comments are not changes: Studio compares the content.
  • API: GET /api/jobs/files also returns hashes, hash and mtime.
• The Hydra DSL panel buttons show three states: ready, busy (spinning) and unavailable.
• /api/health returns studio_build; a tab running an older Studio shows New Studio version — reload.

Patches and fixes

• Studio no longer loses what the canvas does not edit. A job rebuilt from its YAML and saved again reset batch_size to 1000, dropped fields such as delimiter or encoding, and renamed source/destination keys, which changed pipeline.from/to.
• hdrctl validate printed "0 error(s) detected" when the only errors were in pipeline.yaml, in strict checks or a missing file. The total now counts every error shown.
• pipeline.yaml is checked the way the engine reads it: old nested format, missing or empty from/to, unknown keys. Same check in hdrctl run <job> --dry-run.
• hdrctl run <project> (and --dry-run, Studio's Check and Run project) checks pipeline.yaml of every job used by an active workflow before starting: a broken pipeline.from no longer passes as valid.
• hdrctl serve sends index.html with Cache-Control: no-cache, so an update is not hidden by the browser cache.
• Connector messages (CSV, JSON, Parquet, PostgreSQL, MySQL/MariaDB, SQL Server) and Studio templates are in English.

Full changelog: v0.12.2...v0.12.3

v0.12.2 — Rearrange workflow cards in Studio

New features

• Studio: the workflow cards of a project page can be rearranged by drag and drop. Grab a card by its title and drop it on another card (dashed outline) to swap them. Display only: the order is kept in .hydra/card-order.json and does not change thread.yaml or hdrctl run. API: PUT /api/projects/{id}/card-order.

Patches and fixes

• None.

Full changelog: v0.12.1...v0.12.2