This release introduces a pipeline-centric API that dramatically reduces
boilerplate while maintaining full backward compatibility with existing
code.
New High-Level API
------------------
Add FlowStone.run/2-3 for running assets with automatic dependency
resolution. Pipelines are auto-registered on first use, eliminating the
need for manual registry setup and server configuration.
{:ok, result} = FlowStone.run(MyPipeline, :asset)
{:ok, result} = FlowStone.run(MyPipeline, :asset, partition: ~D[2025-01-15])
Add FlowStone.get/2-3 for retrieving cached results, FlowStone.exists?/2-3
for checking asset existence, FlowStone.invalidate/2-3 for cache management,
and FlowStone.status/2-3 for execution status queries.
Add FlowStone.backfill/3 for processing multiple partitions efficiently
with optional parallel execution:
{:ok, stats} = FlowStone.backfill(MyPipeline, :asset,
partitions: Date.range(~D[2025-01-01], ~D[2025-01-31]),
parallel: 4
)
Add FlowStone.graph/1-2 for DAG visualization in ASCII and Mermaid formats,
FlowStone.assets/1 for listing pipeline assets, and FlowStone.asset_info/2
for detailed asset information.
Configuration System
--------------------
Add FlowStone.Config module providing centralized configuration with
sensible defaults. FlowStone now works with zero configuration using
in-memory storage. Adding a repo automatically enables Postgres storage
and lineage tracking:
config :flowstone, repo: MyApp.Repo
The configuration system auto-detects appropriate defaults based on what
is configured, reducing boilerplate in environment-specific configs.
DSL Improvements
----------------
Add short-form asset syntax for simple assets:
asset :config, do: {:ok, %{batch_size: 100}}
Add implicit result wrapping with wrap_results option:
defmodule MyPipeline do
use FlowStone.Pipeline, wrap_results: true
asset :numbers do
execute fn _, _ -> [1, 2, 3] end # Wrapped as {:ok, [1, 2, 3]}
end
end
Add pipeline module injection - each pipeline now gets convenience methods
injected automatically:
MyPipeline.run(:asset)
MyPipeline.get(:asset)
MyPipeline.exists?(:asset)
MyPipeline.assets()
MyPipeline.graph()
Test Helpers
------------
Add FlowStone.Test module providing test isolation and helper functions:
use FlowStone.Test
test "runs with mocked deps" do
{:ok, result} = run_asset(MyPipeline, :downstream,
with_deps: %{upstream: "mocked"}
)
assert result == expected
end
Add assert_asset_exists/3 and refute_asset_exists/3 for declarative
assertions about asset state.
Error Improvements
------------------
Add FlowStone.Error.asset_not_found_with_suggestion/3 which uses Jaro
distance to suggest similar asset names when an unknown asset is
requested, improving developer experience.
Documentation
-------------
Add comprehensive guides:
- guides/getting-started.md: Step-by-step introduction
- guides/configuration.md: Configuration options and patterns
- guides/testing.md: Testing patterns and helpers
Add runnable examples:
- examples/01_hello_world.exs
- examples/02_dependencies.exs
- examples/03_partitions.exs
- examples/04_backfill.exs
Restructure README to lead with the new simplified API while preserving
documentation of the low-level API for advanced use cases.
Update ExDoc configuration to organize modules into High-Level API, Core,
and Testing groups for better discoverability.
Implementation Details
----------------------
Add lib/flowstone/api.ex containing the new API implementation with
per-pipeline isolation using dynamically named registry and IO agent
processes.
Update lib/flowstone/materializer.ex to support wrap_results option by
checking for __flowstone_wrap_results__/0 callback on pipeline modules.
Update lib/flowstone/pipeline.ex with macro improvements for short-form
detection, module injection, and wrap_results configuration.
Backward Compatibility
----------------------
Both the new simplified API and the existing low-level API are fully
supported. The new API is recommended for new code while existing code
continues to work unchanged. The low-level API remains available for
advanced use cases requiring direct control over registries, IO managers,
and Oban job execution.