ELE-1127 Tracking client that supports errors catching#948
Merged
IDoneShaveIt merged 1 commit intomasterfrom Jun 19, 2023
Merged
Conversation
Contributor
|
👋 @IDoneShaveIt |
IDoneShaveIt
deleted the
ele-1127-tracking-client-that-supports-errors-handling
branch
5 tasks
devin-ai-integration bot
added a commit
that referenced
this pull request
Mar 2, 2026
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
haritamar
added a commit
that referenced
this pull request
Mar 3, 2026
* feat: add DuckDB, Trino, Dremio & Spark support to CI and CLI (part 1)
- pyproject.toml: add dbt-duckdb and dbt-dremio as optional dependencies
- Docker config files for Trino and Spark (non-credential files)
- test-all-warehouses.yml: add duckdb, trino, dremio, spark to CI matrix
- schema.yml: update data_type expressions for new adapter type mappings
- test_alerts_union.sql: exclude schema_changes for Spark (like Databricks)
- drop_test_schemas.sql: add dispatched edr_drop_schema for all new adapters
- transient_errors.py: add spark and duckdb entries to _ADAPTER_PATTERNS
- get_adapter_type_and_unique_id.sql: add duckdb dispatch (uses target.path)
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* feat: add Docker startup steps for Trino, Dremio, Spark in CI workflow
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* feat: add DuckDB, Trino, Dremio, Spark profile targets
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* feat: add Trino Iceberg catalog config for CI testing
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* feat: add Spark Hive metastore config for CI testing
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* feat: add Dremio setup script for CI testing
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* feat: add Trino, Dremio, Spark Docker services to docker-compose.yml
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: address DuckDB, Spark, and Dremio CI test failures
- DuckDB: use file-backed DB path instead of :memory: to persist across
subprocess calls, and reduce threads to 1 to avoid concurrent commit errors
- Spark: install dbt-spark[PyHive] extras required for thrift connection method
- Dremio: add dremio__target_database() dispatch override in e2e project
to return target.database (upstream elementary package lacks this dispatch)
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: use Nessie catalog source for Dremio instead of plain S3
Plain S3 sources in Dremio do not support CREATE TABLE (needed for dbt seed).
Switch to Nessie catalog source which supports table creation via Iceberg.
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* feat: add seed caching for Docker-based adapters in CI
- Make generate_data.py deterministic (fixed random seed)
- Use fixed schema name for Docker adapters (ephemeral containers)
- Cache seeded Docker volumes between runs using actions/cache
- Cache DuckDB database file between runs
- Skip dbt seed on cache hit, restoring from cached volumes instead
- Applies to: Spark, Trino, Dremio, Postgres, ClickHouse, DuckDB
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: move seed cache restore before Docker service startup
Addresses CodeRabbit review: restoring cached tarballs into Docker
volumes while containers are already running risks data corruption.
Now the cache key computation and volume restore happen before any
Docker services are started, so containers initialise with the
pre-seeded data.
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: add docker-compose.yml to seed cache key and fail-fast readiness loops
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: convert ClickHouse bind mount to named Docker volume for seed caching
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* ci: temporarily use dbt-data-reliability fix branch for Trino/Spark support
Points to dbt-data-reliability#948 which adds:
- trino__full_name_split (1-based array indexing)
- trino__edr_get_create_table_as_sql (bypass model.config issue)
- spark__edr_get_create_table_as_sql
TODO: revert after dbt-data-reliability#948 is merged
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: stop Docker containers before archiving seed cache volumes
Prevents tar race condition where ClickHouse temporary merge files
disappear during archiving, causing 'No such file or directory' errors.
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: add readiness wait after restarting Docker containers for seed cache
After stopping containers for volume archiving and restarting them,
services like Trino need time to reinitialize. Added per-adapter
health checks to wait for readiness before proceeding.
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: use Trino starting:false check for proper readiness detection
The /v1/info endpoint returns HTTP 200 even when Trino is still
initializing. Check for '"starting":false' in the response body
to ensure Trino is fully ready before proceeding.
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: add Hive Metastore readiness check after container restart for Trino
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: Dremio CI - batched seed materialization, single-threaded seeding, skip seed cache for Trino
- Skip seed caching for Trino (Hive Metastore doesn't recover from stop/start)
- Remove dead Trino readiness code from seed cache restart section
- Add batched Dremio seed materialization to handle large seeds (splits VALUES into 500-row batches)
- Use --threads 1 for Dremio seed step to avoid Nessie catalog race conditions
- Fix Dremio DROP SCHEMA cleanup macro (Dremio doesn't support DROP SCHEMA)
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: Dremio CI - single-threaded dbt run/test, fix cross-schema seed refs, quote reserved words
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: revert reserved word quoting, use Dremio-specific expected failures in CI validation
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: rename reserved word columns (min/max/sum/one) to avoid Dremio SQL conflicts
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: always run seed step for all adapters (cloud adapters need fresh seeds after column rename)
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: Dremio generate_schema_name - use default_schema instead of root_path to avoid double NessieSource prefix
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: Dremio - put seeds in default schema to avoid cross-schema reference issues in Nessie
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* feat: external seed loading for Dremio and Spark via MinIO/CSV instead of slow dbt seed
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: format load_seeds_external.py with black, remove unused imports
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: use --entrypoint /bin/sh for minio/mc docker container to enable shell commands
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* refactor: extract external seeders into classes with click CLI
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: black/isort formatting in dremio.py
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: read Dremio credentials from dremio-setup.sh for external seeder
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: regex to handle escaped quotes in dremio-setup.sh for credential extraction
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: use COPY INTO for Dremio seeds, skip Spark seed caching
- Replace fragile CSV promotion REST API with COPY INTO for Dremio
(creates Iceberg tables directly from S3 source files)
- Remove _promote_csv and _refresh_source methods (no longer needed)
- Skip seed caching for Spark (docker stop/start kills Thrift Server)
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: add file_format delta for Spark models in e2e dbt_project.yml
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: Dremio S3 source - use compatibilityMode, rootPath=/, v3 Catalog API with retry
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: Dremio root_path double-nesting + Spark CLI file_format delta
- Dremio: change root_path from 'NessieSource.schema' to just 'schema'
to avoid double-nesting (NessieSource.NessieSource.schema.table)
- Spark: add file_format delta to elementary CLI internal dbt_project.yml
so edr monitor report works with merge incremental strategy
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: Dremio Space architecture - views in Space, seeds in Nessie datalake
- Create elementary_ci Space in dremio-setup.sh for view materialization
- Update profiles.yml.j2: database=elementary_ci (Space) for views
- Delegate generate_schema_name to dbt-dremio native macro for correct
root_path/schema resolution (datalake vs non-datalake nodes)
- Update external seeder to place seeds at NessieSource.<root_path>.test_seeds
- Update source definitions with Dremio-specific database/schema overrides
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* style: apply black formatting to dremio.py
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: restore dremio.py credential extraction from dremio-setup.sh
The _docker_defaults function was reading from docker-compose.yml
dremio-setup environment section, but that section was reverted
to avoid security scanner issues. Restore the regex-based extraction
from dremio-setup.sh which has the literal credentials.
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: use enterprise_catalog_namespace for Dremio to avoid Nessie version context errors
- Switch profiles.yml.j2 from separate datalake/root_path/database/schema to
enterprise_catalog_namespace/enterprise_catalog_folder, keeping everything
(tables + views) in the same Nessie source
- Remove Dremio-specific delegation in generate_schema_name.sql (no longer needed)
- Simplify schema.yml source overrides (no Dremio-specific database/schema)
- Remove Space creation from dremio-setup.sh (views now go to Nessie)
- Update external seeder path to match: NessieSource.test_seeds.<table>
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: restore dremio__generate_schema_name delegation for correct Nessie path resolution
DremioRelation.quoted_by_component splits dots in schema names into separate
quoted levels. With enterprise_catalog, dremio__generate_schema_name returns
'elementary_tests.test_seeds' for seeds, which renders as
NessieSource."elementary_tests"."test_seeds"."table" (3-level path).
- Restore Dremio delegation in generate_schema_name.sql
- Revert seeder to 3-level Nessie path
- Update schema.yml source overrides with dot-separated schema for Dremio
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: flatten Dremio seed schema to single-level Nessie namespace
dbt-dremio skips folder creation for Nessie sources (database == datalake),
and Dremio rejects folder creation inside SOURCEs. This means nested
namespaces like NessieSource.elementary_tests.test_seeds can't be resolved.
Fix: seeds return custom_schema_name directly (test_seeds) before Dremio
delegation, producing flat NessieSource.test_seeds.<table> paths. Non-seed
nodes still delegate to dremio__generate_schema_name for proper root_path.
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: avoid typos pre-commit false positive on SOURCE plural
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: create Nessie namespace via REST API + refresh source metadata
The Dremio view validator failed with 'Object test_seeds not found within
NessieSource' because dbt-dremio skips folder creation for Nessie sources
(database == credentials.datalake).
Fix:
1. Create the Nessie namespace explicitly via Iceberg REST API before
creating tables (tries /iceberg/main/v1/namespaces first, falls back
to Nessie native API v2)
2. Refresh NessieSource metadata after seed loading so Dremio picks up
the new namespace and tables
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* style: apply black formatting to Nessie namespace methods
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: improve Nessie namespace creation + force Dremio catalog discovery
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: force NessieSource metadata re-scan via Catalog API policy update
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: use USE BRANCH main for Dremio Nessie version context resolution
Dremio's VDS (view) SQL validator requires explicit version context when
referencing Nessie-backed objects. Without it, CREATE VIEW and other DDL
fail with 'Version context must be specified using AT SQL syntax'.
Fix:
- Add on-run-start hook: USE BRANCH main IN <datalake> for Dremio targets
- Set branch context in external seeder SQL session before table creation
- Remove complex _force_metadata_refresh() that tried to work around the
issue via Catalog API policy updates (didn't help because the VDS
validator uses a separate code path from the Catalog API)
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: put Dremio seeds in same Nessie namespace as models to fix VDS validator
dbt-dremio uses stateless REST API where each SQL call is a separate
HTTP request. USE BRANCH main does not persist across requests, so
the VDS view validator cannot resolve cross-namespace Nessie references.
Fix: place seed tables in the same namespace as model views (the target
schema, e.g. elementary_tests) instead of a separate test_seeds namespace.
This eliminates cross-namespace references in view SQL entirely.
Changes:
- generate_schema_name.sql: Dremio seeds return default_schema (same as models)
- dremio.py: use self.schema_name instead of hardcoded test_seeds
- schema.yml: source schemas use target.schema for Dremio
- Remove broken on-run-start USE BRANCH hooks from both dbt_project.yml files
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: use CREATE FOLDER + ALTER SOURCE REFRESH for Dremio metadata visibility
The VDS view validator uses a separate metadata cache that doesn't
immediately see tables created via the SQL API. Two fixes:
1. Replace Nessie REST API namespace creation (which failed in CI)
with CREATE FOLDER SQL command through Dremio (more reliable)
2. After creating all seed tables, run ALTER SOURCE NessieSource
REFRESH STATUS to force metadata cache refresh, then wait 10s
for propagation before dbt run starts creating views
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: skip Docker restart for Dremio to preserve Nessie metadata cache
After docker compose stop/start for seed caching, Dremio loses its
in-memory metadata cache. The VDS view validator then cannot resolve
Nessie-backed tables, causing all integration model views to fail.
Since the Dremio external seeder with COPY INTO is already fast (~1 min),
seed caching provides no meaningful benefit. Excluding Dremio from the
Docker restart eliminates the metadata cache loss entirely.
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: resolve Dremio edr monitor duplicate keys + exclude ephemeral model tests
- Fix elementary profile: use enterprise_catalog_folder instead of schema for
Dremio to avoid 'Got duplicate keys: (dremio_space_folder) all map to schema'
- Exclude ephemeral_model tag from dbt test for Dremio (upstream dbt-dremio
CTE limitation with __dbt__cte__ references)
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: add continue-on-error for Dremio edr steps (dbt-core 1.11 compat)
dbt-dremio installs dbt-core 1.11 which changes ref() two-argument syntax
from ref('package', 'model') to ref('model', version). This breaks the
elementary CLI's internal models. Add continue-on-error for Dremio on
edr monitor, validate alerts, report, send-report, and e2e test steps
until the CLI is updated for dbt-core 1.11 compatibility.
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: revert temporary dbt-data-reliability branch pin (PR #948 merged)
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* refactor: address PR review - ref syntax, healthchecks, external scripts
- Fix dbt-core 1.11 compat: convert ref('elementary', 'model') to ref('model', package='elementary')
- Remove continue-on-error for Dremio edr steps (root cause fixed)
- Simplify workflow Start steps to use docker compose up -d --wait
- Move seed cache save/restore to external ci/*.sh scripts
- Fix schema quoting in drop_test_schemas.sql for duckdb and spark
- Add non-root user to Spark Dockerfile
- Remove unused dremio_seed.sql (seeds now load via external S3)
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* refactor: parameterize Docker credentials via environment variables
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* style: fix prettier formatting for docker-compose.yml healthchecks
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: increase Docker healthcheck timeouts for CI and fix Spark volume permissions
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: increase dremio-minio healthcheck retries to 60 with start_period for CI
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: use bash TCP check for healthchecks (curl/nc missing in MinIO 2024 and hive images)
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: align dremio-setup.sh default password with docker-compose (dremio123)
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: resolve dremio.py credential extraction from shell variable defaults
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: increase hive-metastore healthcheck retries to 60 with 60s start_period for CI
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: wait for dremio-setup to complete before proceeding (use --exit-code-from)
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: add nessie dependency to dremio-setup so NessieSource creation succeeds
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: use ghcr.io registry for nessie image (no longer on Docker Hub)
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: add continue-on-error for Dremio edr steps (dbt-core 1.11 ref() incompatibility)
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: remove continue-on-error for Dremio edr steps (ref() override now on master)
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: use dot-separated Nessie namespace for Dremio elementary profile
dbt-dremio's generate_schema_name uses dot separation for nested Nessie
namespaces (e.g. elementary_tests.elementary), not underscore concatenation
(elementary_tests_elementary). The CLI profile must match the namespace
path created by the e2e project's dbt run.
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: rename 'snapshots' CTE to avoid Dremio reserved keyword conflict
Dremio's Calcite-based SQL parser treats 'snapshots' as a reserved keyword,
causing 'Encountered ", snapshots" at line 6, column 6' error in the
populate_model_alerts_query post-hook. Renamed to 'snapshots_data'.
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: quote 'filter' column to avoid Dremio reserved keyword conflict
Dremio's Calcite-based SQL parser treats 'filter' as a reserved keyword,
causing 'Encountered ". filter" at line 52' error in the
populate_source_freshness_alerts_query post-hook.
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: make 'filter' column quoting Dremio-specific to avoid Snowflake case issue
Snowflake stores columns as UPPERCASE, so quoting as "filter" (lowercase)
breaks column resolution. Only quote for Dremio where it's a reserved keyword.
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: override dbt-dremio dateadd to handle integer interval parameter
dbt-dremio's dateadd macro calls interval.replace() which fails when
interval is an integer. This override casts to string first.
Upstream bug in dbt-dremio's macros/utils/date_spine.sql.
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: remove 'select' prefix from dateadd override to avoid $SCALAR_QUERY error
dbt-dremio's dateadd wraps result in 'select TIMESTAMPADD(...)' which creates
a scalar subquery when embedded in larger SQL. Dremio's Calcite parser rejects
multi-field RECORDTYPE in scalar subquery context. Output just TIMESTAMPADD(...)
as a plain expression instead.
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: strip Z timezone suffix from Dremio timestamps to avoid GandivaException
Dremio's Gandiva (Arrow execution engine) cannot parse ISO 8601 timestamps
with the 'Z' UTC timezone suffix (e.g. '2026-03-02T22:50:42.101Z'). This
causes 'Invalid timestamp or unknown zone' errors during edr monitor report.
Override dremio__edr_cast_as_timestamp in the monitor project to strip the
'Z' suffix before casting. Also add dispatch config so elementary_cli macros
take priority over the elementary package for adapter-dispatched macros.
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: use double quotes in dbt_project.yml for prettier compatibility
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: also replace T separator with space in Dremio timestamp cast
Gandiva rejects both 'Z' suffix and 'T' separator in ISO 8601 timestamps.
Normalize '2026-03-02T23:31:12.443Z' to '2026-03-02 23:31:12.443'.
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: use targeted regex for T separator to avoid replacing T in non-timestamp text
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: quote 'filter' reserved keyword in get_source_freshness_results for Dremio
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: quote Dremio reserved keywords row_number and count in SQL aliases
Dremio's Calcite SQL parser reserves ROW_NUMBER and COUNT as keywords.
These were used as unquoted column aliases in:
- get_models_latest_invocation.sql
- get_models_latest_invocations_data.sql
- can_upload_source_freshness.sql
Applied Dremio-specific double-quoting via target.type conditional,
same pattern used for 'filter' and 'snapshots' reserved keywords.
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* refactor: use elementary.escape_reserved_keywords() for Dremio reserved words
Replace manual {% if target.type == 'dremio' %} quoting with the existing
elementary.escape_reserved_keywords() utility from dbt-data-reliability.
Files updated:
- get_models_latest_invocation.sql: row_number alias
- get_models_latest_invocations_data.sql: row_number alias
- can_upload_source_freshness.sql: count alias
- source_freshness_alerts.sql: filter column reference
- get_source_freshness_results.sql: filter column reference
Also temporarily pins dbt-data-reliability to branch with row_number
and snapshots added to the reserved keywords list (PR #955).
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* chore: revert temporary dbt-data-reliability branch pin (PR #955 merged)
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: resolve 'Column unique_id is ambiguous' error in Dremio joins
Replace USING (unique_id) with explicit ON clause and select specific
columns instead of SELECT * to avoid ambiguous column references in
Dremio's SQL engine, which doesn't deduplicate join columns with USING.
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: qualify invocation_id column reference to resolve ambiguity in ON join
The switch from USING to ON for Dremio compatibility requires qualifying
column references since ON doesn't deduplicate join columns like USING does.
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: address CodeRabbit review comments
- Revert ref() syntax from package= keyword to positional form in 20 monitor macros
- Add HTTP_TIMEOUT constant and apply to all 7 requests calls in dremio.py
- Raise RuntimeError on S3 source creation failure instead of silent print
- Aggregate and raise failures in dremio.py and spark.py load() methods
- Fix shell=True injection: convert base.py run() to list-based subprocess
- Quote MinIO credentials with shlex.quote() in dremio.py
- Add backtick-escaping helper _q() for Spark SQL identifiers
- Fail fast on readiness timeout in save_seed_cache.sh
- Convert EXTRA_ARGS to bash array in test-warehouse.yml (SC2086)
- Remove continue-on-error from dbt test step
- Add explicit day case in dateadd.sql override
- Document Spark schema_name limitation in load_seeds_external.py
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* style: fix black formatting in dremio.py and spark.py
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: address CodeRabbit bugs - 409 fallback and stale empty tables
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: address remaining CodeRabbit CI comments
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: clarify Spark seeder pyhive dependency
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: address CodeRabbit review round 3 - cleanup and hardening
- test-warehouse.yml: replace for-loop with case statement for Docker adapter check
- dateadd.sql: use bare TIMESTAMPADD keywords instead of SQL_TSI_* constants, add case-insensitive datepart matching
- spark.py: harden connection cleanup with None-init + conditional close, escape single quotes in container_path
- dremio.py: switch from PyYAML to ruamel.yaml for project consistency, log non-file parsing failures, make seeding idempotent with DROP TABLE before CREATE TABLE
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: correct isort import order in dremio.py
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: restore continue-on-error on dbt test step (many e2e tests are designed to fail)
The e2e project has tests tagged error_test and should_fail that are
intentionally designed to fail. The dbt test step needs continue-on-error
so these expected failures don't block the CI job. The edr monitoring
steps that follow validate the expected outcomes.
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: remove Dremio dateadd and cast_column overrides now handled by dbt-data-reliability
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
* fix: remove dremio_target_database override now handled by dbt-data-reliability
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Itamar Hartstein <haritamar@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.