diff --git a/.changeset/add-map-step-type-infrastructure.md b/.changeset/add-map-step-type-infrastructure.md index 1df728b7f..03f5c00a0 100644 --- a/.changeset/add-map-step-type-infrastructure.md +++ b/.changeset/add-map-step-type-infrastructure.md @@ -4,37 +4,39 @@ Add map step type infrastructure in SQL core -> [!WARNING] -> **This migration includes automatic data migration** -> -> The migration will automatically update existing `step_states` rows to satisfy new constraints. This should complete without issues due to strict check constraints enforced in previous versions. - -> [!TIP] -> **Optional: Verify before deploying to production** -> -> If you have existing production data and want to verify the migration will succeed cleanly, run this **read-only check query** (does not modify data) in **Supabase Studio** against your **production database**: -> -> 1. Open Supabase Studio → SQL Editor -> 2. Copy contents of `pkgs/core/queries/PRE_MIGRATION_CHECK_20251006073122.sql` -> 3. Execute against your production database (not local dev!) -> 4. Review results -> -> **Expected output for successful migration:** -> ``` -> type | identifier | details -> ----------------------|---------------------------|--------------------------- -> ISSUE_1_AUTO_FIXED | run=abc12345 step=load | status=created remaining_tasks=1 → will set to NULL -> ISSUE_2_AUTO_FIXED | run=def67890 step=process | status=started → will set initial_tasks=1 -> INFO_SUMMARY | total_step_states=42 | created=5 started=2 completed=30 failed=5 -> ``` -> -> **Interpretation:** -> - ✅ Only `ISSUE_1_AUTO_FIXED`, `ISSUE_2_AUTO_FIXED`, and `INFO_SUMMARY` rows? **Safe to migrate** -> - 🆘 Unexpected issues or errors? Copy output and share on Discord for help -> -> **Note:** This check only returns results indicating correctness or problems - it does not modify any data. Only useful for production databases with existing runs. +⚠️ **This migration includes automatic data migration** + +The migration will automatically update existing `step_states` rows to satisfy new constraints. This should complete without issues due to strict check constraints enforced in previous versions. + +💡 **Recommended: Verify before deploying to production** + +If you have existing production data and want to verify the migration will succeed cleanly, run this **read-only check query** (does not modify data) in **Supabase Studio** against your **production database**: + +1. Open Supabase Studio → SQL Editor +2. Copy contents of `pkgs/core/queries/PRE_MIGRATION_CHECK_20251006073122.sql` +3. Execute against your production database (not local dev!) +4. Review results + +**Expected output for successful migration:** + +``` +type | identifier | details +---------------------------|---------------------------|------------------------------------------ +DATA_BACKFILL_STARTED | run=def67890 step=process | initial_tasks will be set to 1 (...) +DATA_BACKFILL_COMPLETED | Found 100 completed steps | initial_tasks will be set to 1 (...) +INFO_SUMMARY | total_step_states=114 | created=0 started=1 completed=113 failed=0 +``` + +**Interpretation:** + +- ✅ Only `DATA_BACKFILL_*` and `INFO_SUMMARY` rows? **Safe to migrate** +- ⚠️ These are expected data migrations handled automatically by the migration +- 🆘 Unexpected rows or errors? Copy output and share on Discord for help + +📝 **Note:** This check identifies data that needs migration but does not modify anything. Only useful for production databases with existing runs. **Automatic data updates:** + - Sets `initial_tasks = 1` for all existing steps (correct for pre-map-step schema) - Sets `remaining_tasks = NULL` for 'created' status steps (new semantics) @@ -47,6 +49,7 @@ No manual intervention required. This patch introduces the foundation for map step functionality in the SQL core layer: ### Schema Changes + - Added `step_type` column to `steps` table with constraint allowing 'single' or 'map' values - Added `initial_tasks` column to `step_states` table (defaults to 1, stores planned task count) - Modified `remaining_tasks` column to be nullable (NULL = not started, >0 = active countdown) @@ -54,13 +57,15 @@ This patch introduces the foundation for map step functionality in the SQL core - Removed `only_single_task_per_step` constraint from `step_tasks` table to allow multiple tasks per step ### Function Updates + - **`add_step()`**: Now accepts `step_type` parameter (defaults to 'single') with validation that map steps can have at most 1 dependency - **`start_flow()`**: Sets `initial_tasks = 1` for all steps (map step array handling will come in future phases) - **`start_ready_steps()`**: Copies `initial_tasks` to `remaining_tasks` when starting a step, maintaining proper task counting semantics ### Testing + - Added comprehensive test coverage for map step creation and validation - All existing tests pass with the new schema changes - Tests validate the new step_type parameter and dependency constraints for map steps -This is Phase 2a of the map step implementation, establishing the SQL infrastructure needed for parallel task execution in future phases. \ No newline at end of file +This is Phase 2a of the map step implementation, establishing the SQL infrastructure needed for parallel task execution in future phases. diff --git a/pkgs/core/queries/PRE_MIGRATION_CHECK_20251006073122.sql b/pkgs/core/queries/PRE_MIGRATION_CHECK_20251006073122.sql index 66fbb5fc7..af4da8f7b 100644 --- a/pkgs/core/queries/PRE_MIGRATION_CHECK_20251006073122.sql +++ b/pkgs/core/queries/PRE_MIGRATION_CHECK_20251006073122.sql @@ -1,25 +1,24 @@ -- ================================================================================ -- PRE-MIGRATION CHECK for 20251006073122_pgflow_add_map_step_type.sql -- ================================================================================ --- Purpose: Verify the migration will succeed cleanly on your database +-- Purpose: Identify data that requires migration before schema changes -- When to run: BEFORE applying the migration -- What to do with output: -- - ✅ Only INFO rows? You're good to migrate --- - ⚠️ ISSUE_1/ISSUE_2? Safe - migration auto-fixes these --- - 🆘 Unexpected issues? Copy/paste output to Discord for help +-- - ⚠️ DATA_* rows? Expected - migration handles these automatically +-- - 🆘 Unexpected rows? Copy/paste output for help -- ================================================================================ WITH issues AS ( - -- Issue 1: Created steps with remaining_tasks (will be set to NULL) + -- Created steps with remaining_tasks set (should be NULL in new schema) SELECT 1 AS priority, - 'ISSUE_1_AUTO_FIXED' AS type, + 'DATA_CLEANUP_CREATED' AS type, format('run=%s step=%s', LEFT(run_id::text, 8), step_slug ) AS identifier, - format('status=%s remaining_tasks=%s → will set to NULL', - status, + format('remaining_tasks=%s will be set to NULL (created steps not started yet)', remaining_tasks ) AS details FROM pgflow.step_states @@ -27,24 +26,38 @@ WITH issues AS ( UNION ALL - -- Issue 2: Started steps (will backfill initial_tasks) + -- Started steps that need initial_tasks backfilled SELECT 2 AS priority, - 'ISSUE_2_AUTO_FIXED' AS type, + 'DATA_BACKFILL_STARTED' AS type, format('run=%s step=%s', LEFT(run_id::text, 8), step_slug ) AS identifier, - format('status=%s → will set initial_tasks=%s', - status, - COALESCE(remaining_tasks::text, '1') + format('initial_tasks will be set to %s (inferred from remaining_tasks=%s)', + COALESCE(remaining_tasks::text, '1'), + COALESCE(remaining_tasks::text, 'NULL') ) AS details FROM pgflow.step_states WHERE status = 'started' UNION ALL - -- Info: Summary stats + -- Completed steps that need initial_tasks backfilled + SELECT + 3 AS priority, + 'DATA_BACKFILL_COMPLETED' AS type, + format('Found %s completed steps', COUNT(*)) AS identifier, + format('initial_tasks will be set to 1 (old schema enforced single-task)', + '' + ) AS details + FROM pgflow.step_states + WHERE status = 'completed' + HAVING COUNT(*) > 0 + + UNION ALL + + -- Summary stats SELECT 999 AS priority, 'INFO_SUMMARY' AS type, @@ -67,11 +80,14 @@ ORDER BY priority, identifier; -- ================================================================================ -- HOW TO READ THE OUTPUT: -- ================================================================================ --- type | identifier | details --- ----------------------|-------------------------|--------------------------- --- ISSUE_1_AUTO_FIXED | run=abc12345 step=foo | Will be cleaned up --- ISSUE_2_AUTO_FIXED | run=def67890 step=bar | Will be backfilled --- INFO_SUMMARY | total_step_states=42 | Overall stats +-- type | identifier | details +-- ---------------------------|-------------------------|----------------------------- +-- DATA_CLEANUP_CREATED | run=abc12345 step=foo | Cleanup needed +-- DATA_BACKFILL_STARTED | run=def67890 step=bar | Backfill needed +-- DATA_BACKFILL_COMPLETED | Found 100 completed... | Backfill needed +-- INFO_SUMMARY | total_step_states=114 | Overall stats -- --- ✅ Safe to migrate if you only see ISSUE_1/ISSUE_2 + INFO_SUMMARY +-- ✅ Safe to migrate - these data issues are handled automatically by the migration +-- ⚠️ CRITICAL: This migration MUST use the correct version that splits ALTER TABLE +-- statements. The packaged version combines them and will FAIL on started steps. -- ================================================================================