diff --git a/.changeset/add-array-method-to-dsl.md b/.changeset/add-array-method-to-dsl.md deleted file mode 100644 index 2a6a08c95..000000000 --- a/.changeset/add-array-method-to-dsl.md +++ /dev/null @@ -1,17 +0,0 @@ ---- -'@pgflow/dsl': minor ---- - -Add `.array()` method for type-safe array step creation - -Introduces a new `.array()` method that provides compile-time type safety for array-returning handlers with zero runtime overhead. - -- Enforces array return types at compile time -- Pure delegation to existing `.step()` method -- Full support for dependencies and runtime options -- Backward compatible - -```typescript -flow.array({ slug: 'items' }, () => [1, 2, 3]); // ✅ Valid -flow.array({ slug: 'invalid' }, () => 42); // ❌ Compile error -``` diff --git a/.changeset/add-map-method-to-dsl.md b/.changeset/add-map-method-to-dsl.md deleted file mode 100644 index f413e236f..000000000 --- a/.changeset/add-map-method-to-dsl.md +++ /dev/null @@ -1,29 +0,0 @@ ---- -'@pgflow/dsl': minor ---- - -Add `.map()` method to Flow DSL for defining map-type steps - -The new `.map()` method enables defining steps that process arrays element-by-element, complementing the existing SQL Core map infrastructure. Key features: - -- **Root maps**: Process flow input arrays directly by omitting the `array` property -- **Dependent maps**: Process another step's array output using `array: 'stepSlug'` -- **Type-safe**: Enforces Json-compatible types with full TypeScript inference -- **Compile-time duplicate slug detection**: TypeScript now prevents duplicate step slugs at compile-time -- **Different handler signature**: Receives individual items `(item, context)` instead of full input object -- **Always returns arrays**: Return type is `HandlerReturnType[]` -- **SQL generation**: Correctly adds `step_type => 'map'` parameter to `pgflow.add_step()` - -Example usage: - -```typescript -// Root map - processes array input -new Flow({ slug: 'process' }).map({ slug: 'uppercase' }, (item) => - item.toUpperCase() -); - -// Dependent map - processes another step's output -new Flow<{}>({ slug: 'workflow' }) - .array({ slug: 'items' }, () => [1, 2, 3]) - .map({ slug: 'double', array: 'items' }, (n) => n * 2); -``` diff --git a/.changeset/add-map-step-type-infrastructure.md b/.changeset/add-map-step-type-infrastructure.md deleted file mode 100644 index 03f5c00a0..000000000 --- a/.changeset/add-map-step-type-infrastructure.md +++ /dev/null @@ -1,71 +0,0 @@ ---- -'@pgflow/core': minor ---- - -Add map step type infrastructure in SQL core - -⚠️ **This migration includes automatic data migration** - -The migration will automatically update existing `step_states` rows to satisfy new constraints. This should complete without issues due to strict check constraints enforced in previous versions. - -💡 **Recommended: Verify before deploying to production** - -If you have existing production data and want to verify the migration will succeed cleanly, run this **read-only check query** (does not modify data) in **Supabase Studio** against your **production database**: - -1. Open Supabase Studio → SQL Editor -2. Copy contents of `pkgs/core/queries/PRE_MIGRATION_CHECK_20251006073122.sql` -3. Execute against your production database (not local dev!) -4. Review results - -**Expected output for successful migration:** - -``` -type | identifier | details ----------------------------|---------------------------|------------------------------------------ -DATA_BACKFILL_STARTED | run=def67890 step=process | initial_tasks will be set to 1 (...) -DATA_BACKFILL_COMPLETED | Found 100 completed steps | initial_tasks will be set to 1 (...) -INFO_SUMMARY | total_step_states=114 | created=0 started=1 completed=113 failed=0 -``` - -**Interpretation:** - -- ✅ Only `DATA_BACKFILL_*` and `INFO_SUMMARY` rows? **Safe to migrate** -- ⚠️ These are expected data migrations handled automatically by the migration -- 🆘 Unexpected rows or errors? Copy output and share on Discord for help - -📝 **Note:** This check identifies data that needs migration but does not modify anything. Only useful for production databases with existing runs. - -**Automatic data updates:** - -- Sets `initial_tasks = 1` for all existing steps (correct for pre-map-step schema) -- Sets `remaining_tasks = NULL` for 'created' status steps (new semantics) - -No manual intervention required. - ---- - -## Changes - -This patch introduces the foundation for map step functionality in the SQL core layer: - -### Schema Changes - -- Added `step_type` column to `steps` table with constraint allowing 'single' or 'map' values -- Added `initial_tasks` column to `step_states` table (defaults to 1, stores planned task count) -- Modified `remaining_tasks` column to be nullable (NULL = not started, >0 = active countdown) -- Added constraint `remaining_tasks_state_consistency` to ensure `remaining_tasks` is only set when step has started -- Removed `only_single_task_per_step` constraint from `step_tasks` table to allow multiple tasks per step - -### Function Updates - -- **`add_step()`**: Now accepts `step_type` parameter (defaults to 'single') with validation that map steps can have at most 1 dependency -- **`start_flow()`**: Sets `initial_tasks = 1` for all steps (map step array handling will come in future phases) -- **`start_ready_steps()`**: Copies `initial_tasks` to `remaining_tasks` when starting a step, maintaining proper task counting semantics - -### Testing - -- Added comprehensive test coverage for map step creation and validation -- All existing tests pass with the new schema changes -- Tests validate the new step_type parameter and dependency constraints for map steps - -This is Phase 2a of the map step implementation, establishing the SQL infrastructure needed for parallel task execution in future phases. diff --git a/.changeset/fix-compile-config-flag.md b/.changeset/fix-compile-config-flag.md deleted file mode 100644 index c8cab1259..000000000 --- a/.changeset/fix-compile-config-flag.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -'pgflow': patch ---- - -Fix: Use --config instead of --import-map for Deno compilation - -The compile command now uses Deno's --config flag instead of --import-map, enabling full deno.json support including nodeModulesDir, compilerOptions, unstable features, and other configuration options. Previously, these options would cause "Invalid top-level key" warnings. - -This is a backward-compatible bug fix. Existing deno.json files with only "imports" continue to work as before. diff --git a/.changeset/fix-minimal-config.md b/.changeset/fix-minimal-config.md deleted file mode 100644 index 61c2b73f4..000000000 --- a/.changeset/fix-minimal-config.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -'pgflow': patch ---- - -Fix config.toml corruption when updating minimal configurations (issue #143) diff --git a/.changeset/improve-failure-handling-prevent-orphaned-messages.md b/.changeset/improve-failure-handling-prevent-orphaned-messages.md deleted file mode 100644 index 38cc06f65..000000000 --- a/.changeset/improve-failure-handling-prevent-orphaned-messages.md +++ /dev/null @@ -1,12 +0,0 @@ ---- -'@pgflow/core': patch ---- - -Improve failure handling and prevent orphaned messages in queue - -- Archive all queued messages when a run fails to prevent resource waste -- Handle type constraint violations gracefully without exceptions -- Store output on failed tasks (including type violations) for debugging -- Add performance index for efficient message archiving -- Prevent retries on already-failed runs -- Update table constraint to allow output storage on failed tasks diff --git a/.changeset/preserve-toml-comments.md b/.changeset/preserve-toml-comments.md deleted file mode 100644 index cc0eb2dd7..000000000 --- a/.changeset/preserve-toml-comments.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -'pgflow': patch ---- - -Fix config.toml corruption with minimal configs while preserving comments. Switch to @decimalturn/toml-patch 0.3.7 (maintained fork) which fixes issue #143 and preserves TOML comments and formatting. Thanks to @DecimalTurn for maintaining the fork and contributing this fix. diff --git a/pkgs/cli/CHANGELOG.md b/pkgs/cli/CHANGELOG.md index e6bd2cfda..f5ece0b65 100644 --- a/pkgs/cli/CHANGELOG.md +++ b/pkgs/cli/CHANGELOG.md @@ -1,5 +1,21 @@ # pgflow +## 0.7.0 + +### Patch Changes + +- 9553bfd: Fix: Use --config instead of --import-map for Deno compilation + + The compile command now uses Deno's --config flag instead of --import-map, enabling full deno.json support including nodeModulesDir, compilerOptions, unstable features, and other configuration options. Previously, these options would cause "Invalid top-level key" warnings. + + This is a backward-compatible bug fix. Existing deno.json files with only "imports" continue to work as before. + +- 28252d9: Fix config.toml corruption when updating minimal configurations (issue #143) +- 0d4623e: Fix config.toml corruption with minimal configs while preserving comments. Switch to @decimalturn/toml-patch 0.3.7 (maintained fork) which fixes issue #143 and preserves TOML comments and formatting. Thanks to @DecimalTurn for maintaining the fork and contributing this fix. +- Updated dependencies [524db03] +- Updated dependencies [524db03] + - @pgflow/core@0.7.0 + ## 0.6.1 ### Patch Changes diff --git a/pkgs/cli/package.json b/pkgs/cli/package.json index 32697fa49..96bb55d8a 100644 --- a/pkgs/cli/package.json +++ b/pkgs/cli/package.json @@ -1,6 +1,6 @@ { "name": "pgflow", - "version": "0.6.1", + "version": "0.7.0", "type": "module", "main": "./dist/index.js", "typings": "./dist/index.d.ts", diff --git a/pkgs/client/CHANGELOG.md b/pkgs/client/CHANGELOG.md index 01432a121..fb2d8ea12 100644 --- a/pkgs/client/CHANGELOG.md +++ b/pkgs/client/CHANGELOG.md @@ -1,5 +1,16 @@ # @pgflow/client +## 0.7.0 + +### Patch Changes + +- Updated dependencies [524db03] +- Updated dependencies [524db03] +- Updated dependencies [524db03] +- Updated dependencies [524db03] + - @pgflow/dsl@0.7.0 + - @pgflow/core@0.7.0 + ## 0.6.1 ### Patch Changes diff --git a/pkgs/client/package.json b/pkgs/client/package.json index 73e7852c0..785f89e90 100644 --- a/pkgs/client/package.json +++ b/pkgs/client/package.json @@ -1,6 +1,6 @@ { "name": "@pgflow/client", - "version": "0.6.1", + "version": "0.7.0", "type": "module", "scripts": { "verify-exports": "node scripts/verify-exports.js" diff --git a/pkgs/core/CHANGELOG.md b/pkgs/core/CHANGELOG.md index 1816be67f..b70fa8d74 100644 --- a/pkgs/core/CHANGELOG.md +++ b/pkgs/core/CHANGELOG.md @@ -1,5 +1,92 @@ # @pgflow/core +## 0.7.0 + +### Minor Changes + +- 524db03: Add map step type infrastructure in SQL core + + ⚠️ **This migration includes automatic data migration** + + The migration will automatically update existing `step_states` rows to satisfy new constraints. This should complete without issues due to strict check constraints enforced in previous versions. + + 💡 **Recommended: Verify before deploying to production** + + If you have existing production data and want to verify the migration will succeed cleanly, run this **read-only check query** (does not modify data) in **Supabase Studio** against your **production database**: + + 1. Open Supabase Studio → SQL Editor + 2. Copy contents of `pkgs/core/queries/PRE_MIGRATION_CHECK_20251006073122.sql` + 3. Execute against your production database (not local dev!) + 4. Review results + + **Expected output for successful migration:** + + ``` + type | identifier | details + ---------------------------|---------------------------|------------------------------------------ + DATA_BACKFILL_STARTED | run=def67890 step=process | initial_tasks will be set to 1 (...) + DATA_BACKFILL_COMPLETED | Found 100 completed steps | initial_tasks will be set to 1 (...) + INFO_SUMMARY | total_step_states=114 | created=0 started=1 completed=113 failed=0 + ``` + + **Interpretation:** + + - ✅ Only `DATA_BACKFILL_*` and `INFO_SUMMARY` rows? **Safe to migrate** + - ⚠️ These are expected data migrations handled automatically by the migration + - 🆘 Unexpected rows or errors? Copy output and share on Discord for help + + 📝 **Note:** This check identifies data that needs migration but does not modify anything. Only useful for production databases with existing runs. + + **Automatic data updates:** + + - Sets `initial_tasks = 1` for all existing steps (correct for pre-map-step schema) + - Sets `remaining_tasks = NULL` for 'created' status steps (new semantics) + + No manual intervention required. + + *** + + ## Changes + + This patch introduces the foundation for map step functionality in the SQL core layer: + + ### Schema Changes + + - Added `step_type` column to `steps` table with constraint allowing 'single' or 'map' values + - Added `initial_tasks` column to `step_states` table (defaults to 1, stores planned task count) + - Modified `remaining_tasks` column to be nullable (NULL = not started, >0 = active countdown) + - Added constraint `remaining_tasks_state_consistency` to ensure `remaining_tasks` is only set when step has started + - Removed `only_single_task_per_step` constraint from `step_tasks` table to allow multiple tasks per step + + ### Function Updates + + - **`add_step()`**: Now accepts `step_type` parameter (defaults to 'single') with validation that map steps can have at most 1 dependency + - **`start_flow()`**: Sets `initial_tasks = 1` for all steps (map step array handling will come in future phases) + - **`start_ready_steps()`**: Copies `initial_tasks` to `remaining_tasks` when starting a step, maintaining proper task counting semantics + + ### Testing + + - Added comprehensive test coverage for map step creation and validation + - All existing tests pass with the new schema changes + - Tests validate the new step_type parameter and dependency constraints for map steps + + This is Phase 2a of the map step implementation, establishing the SQL infrastructure needed for parallel task execution in future phases. + +### Patch Changes + +- 524db03: Improve failure handling and prevent orphaned messages in queue + + - Archive all queued messages when a run fails to prevent resource waste + - Handle type constraint violations gracefully without exceptions + - Store output on failed tasks (including type violations) for debugging + - Add performance index for efficient message archiving + - Prevent retries on already-failed runs + - Update table constraint to allow output storage on failed tasks + +- Updated dependencies [524db03] +- Updated dependencies [524db03] + - @pgflow/dsl@0.7.0 + ## 0.6.1 ### Patch Changes diff --git a/pkgs/core/package.json b/pkgs/core/package.json index e3086c9b6..2e93cfaeb 100644 --- a/pkgs/core/package.json +++ b/pkgs/core/package.json @@ -1,6 +1,6 @@ { "name": "@pgflow/core", - "version": "0.6.1", + "version": "0.7.0", "license": "AGPL-3.0", "type": "module", "main": "./dist/index.js", diff --git a/pkgs/dsl/CHANGELOG.md b/pkgs/dsl/CHANGELOG.md index 50b74b98d..3134bfac3 100644 --- a/pkgs/dsl/CHANGELOG.md +++ b/pkgs/dsl/CHANGELOG.md @@ -1,5 +1,49 @@ # @pgflow/dsl +## 0.7.0 + +### Minor Changes + +- 524db03: Add `.array()` method for type-safe array step creation + + Introduces a new `.array()` method that provides compile-time type safety for array-returning handlers with zero runtime overhead. + + - Enforces array return types at compile time + - Pure delegation to existing `.step()` method + - Full support for dependencies and runtime options + - Backward compatible + + ```typescript + flow.array({ slug: 'items' }, () => [1, 2, 3]); // ✅ Valid + flow.array({ slug: 'invalid' }, () => 42); // ❌ Compile error + ``` + +- 524db03: Add `.map()` method to Flow DSL for defining map-type steps + + The new `.map()` method enables defining steps that process arrays element-by-element, complementing the existing SQL Core map infrastructure. Key features: + + - **Root maps**: Process flow input arrays directly by omitting the `array` property + - **Dependent maps**: Process another step's array output using `array: 'stepSlug'` + - **Type-safe**: Enforces Json-compatible types with full TypeScript inference + - **Compile-time duplicate slug detection**: TypeScript now prevents duplicate step slugs at compile-time + - **Different handler signature**: Receives individual items `(item, context)` instead of full input object + - **Always returns arrays**: Return type is `HandlerReturnType[]` + - **SQL generation**: Correctly adds `step_type => 'map'` parameter to `pgflow.add_step()` + + Example usage: + + ```typescript + // Root map - processes array input + new Flow({ slug: 'process' }).map({ slug: 'uppercase' }, (item) => + item.toUpperCase() + ); + + // Dependent map - processes another step's output + new Flow<{}>({ slug: 'workflow' }) + .array({ slug: 'items' }, () => [1, 2, 3]) + .map({ slug: 'double', array: 'items' }, (n) => n * 2); + ``` + ## 0.6.1 ## 0.6.0 diff --git a/pkgs/dsl/package.json b/pkgs/dsl/package.json index c8e5b32a3..184af567e 100644 --- a/pkgs/dsl/package.json +++ b/pkgs/dsl/package.json @@ -1,6 +1,6 @@ { "name": "@pgflow/dsl", - "version": "0.6.1", + "version": "0.7.0", "type": "module", "main": "./dist/index.js", "module": "./dist/index.js", diff --git a/pkgs/edge-worker/CHANGELOG.md b/pkgs/edge-worker/CHANGELOG.md index 7d182f158..f1e765706 100644 --- a/pkgs/edge-worker/CHANGELOG.md +++ b/pkgs/edge-worker/CHANGELOG.md @@ -1,5 +1,16 @@ # @pgflow/edge-worker +## 0.7.0 + +### Patch Changes + +- Updated dependencies [524db03] +- Updated dependencies [524db03] +- Updated dependencies [524db03] +- Updated dependencies [524db03] + - @pgflow/dsl@0.7.0 + - @pgflow/core@0.7.0 + ## 0.6.1 ### Patch Changes diff --git a/pkgs/edge-worker/jsr.json b/pkgs/edge-worker/jsr.json index 961505960..a42a8564f 100644 --- a/pkgs/edge-worker/jsr.json +++ b/pkgs/edge-worker/jsr.json @@ -1,6 +1,6 @@ { "name": "@pgflow/edge-worker", - "version": "0.6.1", + "version": "0.7.0", "license": "AGPL-3.0", "exports": { ".": "./src/index.ts", @@ -9,8 +9,8 @@ "imports": { "@henrygd/queue": "jsr:@henrygd/queue@^1.0.7", "postgres": "npm:postgres@3.4.5", - "@pgflow/core": "npm:@pgflow/core@0.6.1", - "@pgflow/dsl": "npm:@pgflow/dsl@0.6.1" + "@pgflow/core": "npm:@pgflow/core@0.7.0", + "@pgflow/dsl": "npm:@pgflow/dsl@0.7.0" }, "publish": { "include": [ diff --git a/pkgs/edge-worker/package.json b/pkgs/edge-worker/package.json index c70fd9b6d..d928b7c85 100644 --- a/pkgs/edge-worker/package.json +++ b/pkgs/edge-worker/package.json @@ -1,6 +1,6 @@ { "name": "@pgflow/edge-worker", - "version": "0.6.1", + "version": "0.7.0", "license": "AGPL-3.0", "type": "module", "main": "./dist/index.js", diff --git a/pkgs/example-flows/CHANGELOG.md b/pkgs/example-flows/CHANGELOG.md index 3e441721a..b64f94dcd 100644 --- a/pkgs/example-flows/CHANGELOG.md +++ b/pkgs/example-flows/CHANGELOG.md @@ -1,5 +1,16 @@ # @pgflow/example-flows +## 0.7.0 + +### Patch Changes + +- Updated dependencies [524db03] +- Updated dependencies [524db03] +- Updated dependencies [524db03] +- Updated dependencies [524db03] + - @pgflow/dsl@0.7.0 + - @pgflow/core@0.7.0 + ## 0.6.1 ### Patch Changes diff --git a/pkgs/example-flows/package.json b/pkgs/example-flows/package.json index 208788355..5e6237c86 100644 --- a/pkgs/example-flows/package.json +++ b/pkgs/example-flows/package.json @@ -1,6 +1,6 @@ { "name": "@pgflow/example-flows", - "version": "0.6.1", + "version": "0.7.0", "dependencies": { "@pgflow/core": "workspace:*", "@pgflow/dsl": "workspace:*" diff --git a/pkgs/website/CHANGELOG.md b/pkgs/website/CHANGELOG.md index 8e7734f96..d1ca96612 100644 --- a/pkgs/website/CHANGELOG.md +++ b/pkgs/website/CHANGELOG.md @@ -1,5 +1,7 @@ # @pgflow/website +## 0.7.0 + ## 0.6.1 ## 0.6.0 diff --git a/pkgs/website/package.json b/pkgs/website/package.json index 925c8187c..59ae4f61a 100644 --- a/pkgs/website/package.json +++ b/pkgs/website/package.json @@ -1,7 +1,7 @@ { "name": "@pgflow/website", "type": "module", - "version": "0.6.1", + "version": "0.7.0", "main": "./dist/index.js", "module": "./dist/index.js", "types": "./dist/index.d.ts",