Update onpush.yml#1
Merged
Merged
Conversation
andre-salvati
added a commit
that referenced
this pull request
Jun 11, 2026
## Summary
Three changes on this branch:
1. **Move category to the product dimension.** `prod_category_id` lived on the order fact with `category_name` derived inline as `"Category " + id`. Moved both onto `external_source.product` (`category_id`, `category_name`); silver now pulls them from the product join in both the `job1` batch path (`generate_orders.enrich_order`) and the `job1_sdp` path (`transforms.enrich_order`). A category rename now flows correctly through to gold and the dashboard.
2. **Commit the dashboard JSON, remove code generation.** `resources/orders_dashboard.lvdash.json` is now the canonical, version-controlled dashboard definition (the `_build_dashboard_json` generator is gone). Because DABs 0.298.0 does not substitute `${var.catalog}` inside `.lvdash.json` file content, the generator writes a gitignored deploy copy (`orders_dashboard_deploy.lvdash.json`) with the catalog resolved at deploy time; the committed file keeps the `${var.catalog}` placeholder.
3. **Replace `make truncate` with `make drop`; rename `make test` to `make unit-test`.** `make truncate` ran `TRUNCATE TABLE`, which preserves the schema and is useless for a schema migration (as change #1 proved on prod). `scripts/sdk_truncate_tables.py` → `sdk_drop_tables.py` now `DROP`s every medallion table so the next pipeline run recreates it with the current schema. The `make test` target is renamed to `make unit-test` (CI, CLAUDE.md, README updated).
## Files
- `src/template/commonSchemas.py` — `order_schema` drops `prod_category_id`; `product_schema` gains `category_id`/`category_name`
- `src/template/job1/generate_orders.py`, `src/template/job1_sdp/transforms.py` — category from the product join
- `src/template/job1/seed_sources.py` — product seeding adds category; order seeding drops it; `_build_price_updates` updated
- `tests/job1/{integration_setup,integration_validate,unit_test,unit_test_sdp}.py` — fixtures/expectations updated
- `scripts/sdk_generate_template_job.py`, `databricks.yml`, `.gitignore`, `resources/orders_dashboard.lvdash.json` — dashboard JSON + deploy-time catalog substitution
- `scripts/sdk_drop_tables.py` (renamed from `sdk_truncate_tables.py`), `Makefile`, `.github/workflows/onpush.yml`, `CLAUDE.md`, `README.md` — `make drop` + `make unit-test`
## Testing
- **Unit:** 15/15 pass (via `make unit-test`)
- **Dev / Staging integration:** pass
- **Prod integration:** ran `job1_prod_integration` after a full medallion-table reset via the new drop flow (schema change is incompatible under `overwriteSchema=false`); verified `category_id`/`category_name` flow to both `report.order_agg` and `report.order_agg_sdp` (all 10 categories, 181,500 rows each, identical across paths)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.