Skip to content

v0.9.0

Choose a tag to compare

View all tags
@github-actions github-actions released this 12 Sep 21:40
· 96 commits to main since this release
v0.9.0
d6f3dab
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Removed

  • Removed jinja2 from the CLI package's runtime dependencies; it was only ever imported by images/dagster/generate_config.py, a Docker build-time script, and is now installed explicitly in images/dagster/requirements.txt instead. Added a new test extra (and updated Makefile, CONTRIBUTING.md, and CI) since the test suite still exercises generate_config.py directly (#470).

Added

  • Added a Kubernetes runtime target: cds render/cds validate/cds security/cds test/cds up/cds down/cds state now accept --target helm alongside the existing Docker Compose target. cli/k8s_renderer.py renders the resolved plan as a Helm chart (Secrets, ConfigMaps, Deployments/StatefulSets and PVCs, release-scoped Services), cli/k8s_security.py runs Kubernetes-specific security checks (effective per-container root/non-root posture), and cli/k8s_runner.py provides bounded helm upgrade --install plus kubectl wait/rollout status lifecycle operations. Includes a sibling-safe, per-worktree k3d local-dev harness (scripts/k8s/, make k3d-*) with an isolated k3s CI proof workflow, a TenderNed procurement-data Superset/Dagster analytics demo wired through the new target, and docs/kubernetes.md (#608).
  • Added scripts/ai_profile_review.py, an optional AI-assisted guardrail/simplification review for CDS profiles: it runs cds validate/cds plan and then asks an LLM to flag repository-convention violations and simplification opportunities not already covered by schema/contract validation, seeing only profile YAML and the resolved plan (secrets are always placeholders, never resolved values). Supports --json and --dry-run, and a vendored single-seam LLM client (scripts/_vendor/llm/) with three explicit providers (copilot_cli, azure_openai, ollama) and no silent fallback (#652).
  • Added scripts/compose_to_module.py, a scaffolding tool that automates the mechanical parts of docs/from-docker-to-cds-profile.md: it converts an existing docker-compose.yml into a starter module.yaml, lifting ports/environment into a configSchema, replacing literal values with ${config.*} placeholders, and detecting secret references, hardcoded connection strings, and dependencies on other compose services (flagging well-known infra images for binding to an existing shared contract and provider module instead of being scaffolded anew) (#670).
  • Added config.image.registry (dockerhub default | ghcr) to the orchestration/dagster and bi/superset module schemas, so config.image.source: registry can pull the same signed image publish-images.yml already publishes to GHCR, not just Docker Hub; existing profiles are unaffected since the default keeps them pointed at Docker Hub (#613).
  • Promoted the dbt transformation module from modules-experimental/ to
    modules/transformation/, with production-suitable hardening and
    PostgreSQL/DuckDB warehouse support (#594).
  • Added regression tests for planner default materialization in nested configSchema structures: array-item object defaults filled in per-item without overwriting explicitly provided sibling properties, and partially provided nested objects preserving explicit falsy values (False/0) while still materializing omitted siblings (#459).
  • Added CLI-level test coverage asserting cds validate reports precise diagnostic codes and data paths for common validation failures: a module entry missing a required field (E010) and a consume binding with an unresolvable contractRef (E041) (#460).
  • Added cli.loader.save_generated_profile() and cli.main.generate_profile() so a runtime/programmatically composed profile can be persisted at its normal profiles/<name>/profile.yaml location (honoring CDS_PROFILE_PATH), then handed to the existing validate_profile()/build_plan() entry points completely unchanged -- same relative module-source resolution and extends/environment-overlay semantics as any hand-authored profile. Refuses to write outside the profiles root or silently overwrite an existing profile without force=True. Exposed as a new cds generate-profile <file> CLI command (reads JSON/YAML from a file path or - for stdin, with --name/--force options) (#349).
  • Added test_fetch_profile_rejects_dockerfile_copy_traversal_escaping_source_repo, a regression test proving a Dockerfile COPY/ADD source containing .. that Path.glob() matches outside the source repository is rejected as a stable GetError (via the existing _add_copy_action guard from #454), not an unhandled ValueError (#475).
  • Pinned build, twine, and yamllint's CI-installed versions, and the renovate npm package version used by renovate-config-validator, matching this repo's existing exact-pin convention for CI-only tooling (e.g. ruff==0.16.6), addressing SonarCloud's githubactions:S8544 findings triaged in #622. Added matching Renovate custom managers so these pins stay up to date automatically.

Fixed

  • Set the Docker Hub short description for every published image (dagster, superset, dbt, dlt) via peter-evans/dockerhub-description's short-description input, instead of relying on it being set manually per repository. dbt and dlt were missing it entirely since their Docker Hub repositories were auto-created by CI without ever going through that manual step.
  • Fixed a quadratic (super-linear) regex backtracking hazard in cli/preflight.py's _ENV_REFERENCE pattern, used to scan rendered Compose YAML for ${VAR...} references: an unterminated reference could make the identifier and suffix capture groups' overlapping character classes retry every possible split point. Required the suffix group to start with one of its actual delimiters (:, ?, -), making the two groups' character classes disjoint, flagged by SonarCloud as python:S8786.
  • images/superset/init.sh now uses [[ ... ]] instead of [ ... ] for its conditional tests, addressing SonarCloud's shelldre:S7688 findings triaged in #622.

Changed

  • Raised the coverage-enforced cli/ coverage gate from 65% to 80%, matching actual measured coverage and the industry norm for a security-focused tool (pyproject.toml's [tool.coverage.report] fail_under) (#471).

What's Changed

  • docs(changelog): retroactively add the missing 0.8.0 release section by @RonaldHensbergen in #615
  • Wire dbt module to optionally target DuckDB (#593) by @RonaldHensbergen in #599
  • chore(images): refresh signed-images fixture by @github-actions[bot] in #617
  • docs: soften dagster hardened-variant CVE-count claim by @RonaldHensbergen in #611
  • ci(images): add alpine-latest/alpine- tag aliases for hardened publish by @RonaldHensbergen in #612
  • chore(images): refresh signed-images fixture by @github-actions[bot] in #618
  • Expand release-tag-reminder into a full release completeness checklist by @RonaldHensbergen in #614
  • feat(images): support pulling published images from GHCR, not just Docker Hub by @RonaldHensbergen in #613
  • chore(images): refresh signed-images fixture by @github-actions[bot] in #628
  • fix(superset): use [[ ]] instead of [ ] in init.sh conditionals (#622) by @RonaldHensbergen in #627
  • chore(images): refresh signed-images fixture by @github-actions[bot] in #630
  • docs(readme): add SonarCloud, CI, PyPI, and license badges by @RonaldHensbergen in #629
  • chore(ci): exact-pin CI-only tool versions flagged by SonarCloud (#622) by @RonaldHensbergen in #626
  • ci: run SonarCloud scan on pushes to main, not just same-repo PRs by @RonaldHensbergen in #632
  • ci(biweekly-issue-audit): install and authenticate Copilot CLI by @RonaldHensbergen in #631
  • Add experimental ingestion module for dlt (data load tool) by @RonaldHensbergen in #592
  • Raise coverage gate to 80% and close security-module test gaps by @RonaldHensbergen in #619
  • fix(preflight): remove quadratic regex backtracking in _ENV_REFERENCE by @RonaldHensbergen in #635
  • ci: set Docker Hub short description for all published images by @RonaldHensbergen in #636
  • chore(images): refresh signed-images fixture by @github-actions[bot] in #638
  • test: add planner default-materialization and validate diagnostics-path regression tests by @RonaldHensbergen in #637
  • test(getter): add regression test for Dockerfile COPY traversal escape by @RonaldHensbergen in #633
  • chore(deps): update all dependencies by @renovate[bot] in #639
  • chore: remove jinja2 runtime dependency, unused by the CLI by @RonaldHensbergen in #634
  • chore(images): refresh signed-images fixture by @github-actions[bot] in #640
  • chore(images): refresh signed-images fixture by @github-actions[bot] in #641
  • Promote dbt module out of modules-experimental/ to production-suitable by @RonaldHensbergen in #594
  • Normalize Mako package name for Renovate by @RonaldHensbergen in #644
  • chore(deps): update nginx docker tag to v1.31 by @renovate[bot] in #643
  • chore(images): refresh signed-images fixture by @github-actions[bot] in #642
  • chore(images): refresh signed-images fixture by @github-actions[bot] in #645
  • chore(deps): update dependency renovate to v44.68.1 by @renovate[bot] in #646
  • chore(deps): update dependency renovate to v44.69.6 by @renovate[bot] in #650
  • chore(deps): update dependency renovate to v44.69.7 by @renovate[bot] in #651
  • chore(deps): update all dependencies by @renovate[bot] in #653
  • chore(images): refresh signed-images fixture by @github-actions[bot] in #654
  • Batch renovate's own npm self-update into a weekly PR by @RonaldHensbergen in #658
  • Fix/sonarcloud pinned java by @RonaldHensbergen in #659
  • chore(deps): update actions/setup-java action to v6 by @renovate[bot] in #660
  • chore(deps): update dependency psycopg2-binary to v2.9.13 by @renovate[bot] in #666
  • chore(images): refresh signed-images fixture by @github-actions[bot] in #667
  • Linefeeds between badge groups by @RonaldHensbergen in #671
  • chore(deps): update dependency build to v1.6.1 by @renovate[bot] in #672
  • fix(deps): update dependency ruff to v0.16.7 by @renovate[bot] in #673
  • docs: add Haven parity plan by @RonaldHensbergen in #674
  • chore(deps): update all dependencies by @renovate[bot] in #675
  • chore(images): refresh signed-images fixture by @github-actions[bot] in #676
  • Add AI-assisted guardrail/simplification review for CDS profiles by @RonaldHensbergen in #652
  • feat: allow runtime-generated profiles for planning and composition by @RonaldHensbergen in #647
  • Add scripts/compose_to_module.py: scaffold a module.yaml from docker-compose by @RonaldHensbergen in #670
  • docs: group README badges into labeled categories by @RonaldHensbergen in #687
  • fix: add trivyignore exceptions for new Debian trixie CVEs in dagster base image by @RonaldHensbergen in #688
  • feat: add Kubernetes target and Tender analytics demo by @vanderheijden86 in #608
  • chore(deps): update all dependencies by @renovate[bot] in #692
  • chore(deps): update azure/setup-helm action to v5 by @renovate[bot] in #693
  • chore(release): bump version to 0.9.0 by @github-actions[bot] in #623

New Contributors

Full Changelog: v0.8.0...v0.9.0

What's Changed

  • docs(changelog): retroactively add the missing 0.8.0 release section by @RonaldHensbergen in #615
  • Wire dbt module to optionally target DuckDB (#593) by @RonaldHensbergen in #599
  • chore(images): refresh signed-images fixture by @github-actions[bot] in #617
  • docs: soften dagster hardened-variant CVE-count claim by @RonaldHensbergen in #611
  • ci(images): add alpine-latest/alpine- tag aliases for hardened publish by @RonaldHensbergen in #612
  • chore(images): refresh signed-images fixture by @github-actions[bot] in #618
  • Expand release-tag-reminder into a full release completeness checklist by @RonaldHensbergen in #614
  • feat(images): support pulling published images from GHCR, not just Docker Hub by @RonaldHensbergen in #613
  • chore(images): refresh signed-images fixture by @github-actions[bot] in #628
  • fix(superset): use [[ ]] instead of [ ] in init.sh conditionals (#622) by @RonaldHensbergen in #627
  • chore(images): refresh signed-images fixture by @github-actions[bot] in #630
  • docs(readme): add SonarCloud, CI, PyPI, and license badges by @RonaldHensbergen in #629
  • chore(ci): exact-pin CI-only tool versions flagged by SonarCloud (#622) by @RonaldHensbergen in #626
  • ci: run SonarCloud scan on pushes to main, not just same-repo PRs by @RonaldHensbergen in #632
  • ci(biweekly-issue-audit): install and authenticate Copilot CLI by @RonaldHensbergen in #631
  • Add experimental ingestion module for dlt (data load tool) by @RonaldHensbergen in #592
  • Raise coverage gate to 80% and close security-module test gaps by @RonaldHensbergen in #619
  • fix(preflight): remove quadratic regex backtracking in _ENV_REFERENCE by @RonaldHensbergen in #635
  • ci: set Docker Hub short description for all published images by @RonaldHensbergen in #636
  • chore(images): refresh signed-images fixture by @github-actions[bot] in #638
  • test: add planner default-materialization and validate diagnostics-path regression tests by @RonaldHensbergen in #637
  • test(getter): add regression test for Dockerfile COPY traversal escape by @RonaldHensbergen in #633
  • chore(deps): update all dependencies by @renovate[bot] in #639
  • chore: remove jinja2 runtime dependency, unused by the CLI by @RonaldHensbergen in #634
  • chore(images): refresh signed-images fixture by @github-actions[bot] in #640
  • chore(images): refresh signed-images fixture by @github-actions[bot] in #641
  • Promote dbt module out of modules-experimental/ to production-suitable by @RonaldHensbergen in #594
  • Normalize Mako package name for Renovate by @RonaldHensbergen in #644
  • chore(deps): update nginx docker tag to v1.31 by @renovate[bot] in #643
  • chore(images): refresh signed-images fixture by @github-actions[bot] in #642
  • chore(images): refresh signed-images fixture by @github-actions[bot] in #645
  • chore(deps): update dependency renovate to v44.68.1 by @renovate[bot] in #646
  • chore(deps): update dependency renovate to v44.69.6 by @renovate[bot] in #650
  • chore(deps): update dependency renovate to v44.69.7 by @renovate[bot] in #651
  • chore(deps): update all dependencies by @renovate[bot] in #653
  • chore(images): refresh signed-images fixture by @github-actions[bot] in #654
  • Batch renovate's own npm self-update into a weekly PR by @RonaldHensbergen in #658
  • Fix/sonarcloud pinned java by @RonaldHensbergen in #659
  • chore(deps): update actions/setup-java action to v6 by @renovate[bot] in #660
  • chore(deps): update dependency psycopg2-binary to v2.9.13 by @renovate[bot] in #666
  • chore(images): refresh signed-images fixture by @github-actions[bot] in #667
  • Linefeeds between badge groups by @RonaldHensbergen in #671
  • chore(deps): update dependency build to v1.6.1 by @renovate[bot] in #672
  • fix(deps): update dependency ruff to v0.16.7 by @renovate[bot] in #673
  • docs: add Haven parity plan by @RonaldHensbergen in #674
  • chore(deps): update all dependencies by @renovate[bot] in #675
  • chore(images): refresh signed-images fixture by @github-actions[bot] in #676
  • Add AI-assisted guardrail/simplification review for CDS profiles by @RonaldHensbergen in #652
  • feat: allow runtime-generated profiles for planning and composition by @RonaldHensbergen in #647
  • Add scripts/compose_to_module.py: scaffold a module.yaml from docker-compose by @RonaldHensbergen in #670
  • docs: group README badges into labeled categories by @RonaldHensbergen in #687
  • fix: add trivyignore exceptions for new Debian trixie CVEs in dagster base image by @RonaldHensbergen in #688
  • feat: add Kubernetes target and Tender analytics demo by @vanderheijden86 in #608
  • chore(deps): update all dependencies by @renovate[bot] in #692
  • chore(deps): update azure/setup-helm action to v5 by @renovate[bot] in #693
  • chore(release): bump version to 0.9.0 by @github-actions[bot] in #623

New Contributors

Full Changelog: v0.8.0...v0.9.0

Contributors

renovate, vanderheijden86, and RonaldHensbergen
Assets 2
Loading