Skip to content

v0.8.0 — data exports: Parquet / Arrow / TSV to a folder or S3

Choose a tag to compare

View all tags
@ralfbecher ralfbecher released this 25 Jul 21:37
v0.8.0
625fa5b
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Query results can now be written as Parquet, Arrow IPC, or TSV into a local folder or an S3 bucket (or any S3-compatible store), with an export-only run mode for jobs that feed a warehouse rather than a reader.

exports: — a list of destinations

exports:
  - format: parquet                    # parquet | arrow | tsv
    uri: exports/{name}/{date}/        # local folder
    compression: zstd
  - format: parquet
    uri: s3://analytics-landing/orionbelt/{date}/
    region: eu-central-1
    # endpoint_override: http://localhost:9000   # MinIO / R2 / Ceph

One file per query per target, named after the query, written on a fully successful run. uri takes the same placeholders as report.output and names a directory prefix; local relative paths rebase under --output-dir like the report path. Several targets can run side by side.

Natively typed values, without paying twice

Parquet and Arrow are typed formats, so those targets execute with format_values=false and write double / int64 / timestamp[us] rather than locale-formatted strings. The runner executes each query only as often as the spec actually needs:

spec executes per query
report (or a tsv target) only 1 — formatted
report (or tsv) + parquet / arrow 2 — formatted for the report, raw for the export
no_report: true with only parquet / arrow 1 — raw

Report runs are unchanged: they still render from the formatted call, so OBSL stays authoritative for display.

Export-only runs — no_report: true

Queries execute and exports are written, no report is rendered, and the run log still lands (that's the audit trail). report: becomes optional in that mode. See examples/revenue-export-only.yaml.

Failures are visible

An unreachable bucket or a failed raw re-execute is logged and skipped — never discarding an already-rendered report — but the CLI then prints what didn't land and exits 1, so a scheduled job whose data never arrived can't look like a clean run. A query whose raw re-execute failed is omitted from typed targets rather than written with string columns, so a downstream schema never silently changes shape.

Packaging

  • New optional extra: uv sync --extra arrow (pyarrow >= 16) covers both the file formats and the S3 client. Core markdown / HTML / TSV runs stay dependency-free.
  • The Docker image now bundles pyarrow, so Parquet / Arrow and s3:// destinations work out of the box in a scheduled container — the image is correspondingly larger. PDF still isn't bundled (WeasyPrint needs Pango / Cairo system libraries).
  • S3 credentials come only from the standard AWS chain (env vars, ~/.aws, instance / task / IRSA role) — never from the spec, so spec files stay safe to commit.

Compatibility

Backward compatible: exports: defaults to empty and report: is still required unless no_report: true, so every 0.7.0 spec behaves identically. The OBSL pin is unchanged — 0.8.x ↔ OBSL 2.23.x. This runner minor advances on its own; the two version numbers don't move in lockstep.

Verified: ruff, ruff format, mypy, pytest on 3.12 and 3.14 (108 passed, 1 skipped); end-to-end against a stub OBSL server producing typed Parquet.

Full changes: v0.7.0...v0.8.0

Assets 2
Loading