Skip to content

v0.5.0

@nshkrdotcom nshkrdotcom tagged this 19 Dec 08:57
This release introduces a pipeline-centric API that dramatically reduces
boilerplate while maintaining full backward compatibility with existing
code.

New High-Level API
------------------

Add FlowStone.run/2-3 for running assets with automatic dependency
resolution. Pipelines are auto-registered on first use, eliminating the
need for manual registry setup and server configuration.

    {:ok, result} = FlowStone.run(MyPipeline, :asset)
    {:ok, result} = FlowStone.run(MyPipeline, :asset, partition: ~D[2025-01-15])

Add FlowStone.get/2-3 for retrieving cached results, FlowStone.exists?/2-3
for checking asset existence, FlowStone.invalidate/2-3 for cache management,
and FlowStone.status/2-3 for execution status queries.

Add FlowStone.backfill/3 for processing multiple partitions efficiently
with optional parallel execution:

    {:ok, stats} = FlowStone.backfill(MyPipeline, :asset,
      partitions: Date.range(~D[2025-01-01], ~D[2025-01-31]),
      parallel: 4
    )

Add FlowStone.graph/1-2 for DAG visualization in ASCII and Mermaid formats,
FlowStone.assets/1 for listing pipeline assets, and FlowStone.asset_info/2
for detailed asset information.

Configuration System
--------------------

Add FlowStone.Config module providing centralized configuration with
sensible defaults. FlowStone now works with zero configuration using
in-memory storage. Adding a repo automatically enables Postgres storage
and lineage tracking:

    config :flowstone, repo: MyApp.Repo

The configuration system auto-detects appropriate defaults based on what
is configured, reducing boilerplate in environment-specific configs.

DSL Improvements
----------------

Add short-form asset syntax for simple assets:

    asset :config, do: {:ok, %{batch_size: 100}}

Add implicit result wrapping with wrap_results option:

    defmodule MyPipeline do
      use FlowStone.Pipeline, wrap_results: true

      asset :numbers do
        execute fn _, _ -> [1, 2, 3] end  # Wrapped as {:ok, [1, 2, 3]}
      end
    end

Add pipeline module injection - each pipeline now gets convenience methods
injected automatically:

    MyPipeline.run(:asset)
    MyPipeline.get(:asset)
    MyPipeline.exists?(:asset)
    MyPipeline.assets()
    MyPipeline.graph()

Test Helpers
------------

Add FlowStone.Test module providing test isolation and helper functions:

    use FlowStone.Test

    test "runs with mocked deps" do
      {:ok, result} = run_asset(MyPipeline, :downstream,
        with_deps: %{upstream: "mocked"}
      )
      assert result == expected
    end

Add assert_asset_exists/3 and refute_asset_exists/3 for declarative
assertions about asset state.

Error Improvements
------------------

Add FlowStone.Error.asset_not_found_with_suggestion/3 which uses Jaro
distance to suggest similar asset names when an unknown asset is
requested, improving developer experience.

Documentation
-------------

Add comprehensive guides:
  - guides/getting-started.md: Step-by-step introduction
  - guides/configuration.md: Configuration options and patterns
  - guides/testing.md: Testing patterns and helpers

Add runnable examples:
  - examples/01_hello_world.exs
  - examples/02_dependencies.exs
  - examples/03_partitions.exs
  - examples/04_backfill.exs

Restructure README to lead with the new simplified API while preserving
documentation of the low-level API for advanced use cases.

Update ExDoc configuration to organize modules into High-Level API, Core,
and Testing groups for better discoverability.

Implementation Details
----------------------

Add lib/flowstone/api.ex containing the new API implementation with
per-pipeline isolation using dynamically named registry and IO agent
processes.

Update lib/flowstone/materializer.ex to support wrap_results option by
checking for __flowstone_wrap_results__/0 callback on pipeline modules.

Update lib/flowstone/pipeline.ex with macro improvements for short-form
detection, module injection, and wrap_results configuration.

Backward Compatibility
----------------------

Both the new simplified API and the existing low-level API are fully
supported. The new API is recommended for new code while existing code
continues to work unchanged. The low-level API remains available for
advanced use cases requiring direct control over registries, IO managers,
and Oban job execution.
Assets 2
Loading