Skip to content
master
Go to file
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
lib
 
 
 
 
 
 
 
 
 
 
 
 
 
 

README.md

Pipelinex

Pipelinex is a simple library to build data pipelines in a clean and structured way.

It's mainly built for personal usage to help with structuring big data processing flows and automatically apply things like logging, which otherwise would make the code very unclean.

Installation

The package can be installed by adding pipelinex to your list of dependencies in mix.exs:

def deps do
  [{:pipelinex, "~> 0.1.0"}]
end

Usage

Pipelinex provides two macros to build pipelines, task and pipeline.

Simple Example

Build simple pipelines using the task macro:

defmodule MyPipeline do
  use Pipelinex

  task "add 5", num do
    num + 5
  end

  task "add 10", num do
    num + 10
  end
end

Execute the pipeline via it's injected run/1 function:

result = MyPipeline.run(5)
result == 20

Reference Other Pipelines

Pipelines can execute other pipelines using the pipeline macro:

defmodule MyOtherPipeline do
  use Pipelinex

  pipeline MyPipeline

  task "add 20", num do
    num + 20
  end
end

result = MyOtherPipeline.run(5)
result == 40

Pattern Matching

The task macro supports pattern matching for its data argument.

defmodule MyPipeline do
  use Pipelinex

  task "add 5", %{:value => val} = map do
    Map.put map, :value, val + 5
  end

  task "add 10", %{:value => val} = map do
    Map.put map, :value, val + 10
  end
end

If two or more tasks with the same name are defined one after another, the task will be called once and normal pattern matching behaviour will be applied.

defmodule MyPipeline do
  use Pipelinex

  task "add 5", %{:value => val} = map do
    Map.put map, :value, val + 5
  end

  task "add 5", num do
    num + 5
  end
end

value = MyPipeline.run(%{:value => 5})
value == %{value: 10}

value = MyPipeline.run(5)
value == 10

Logging

Pipelinex logs when a task starts and ends using Logger.info

defmodule MyPipeline do
  use Pipelinex

  task "add 5", num do
    num + 5
  end

  task "add 10", num do
    num + 10
  end
end

MyPipeline.run(5)
# 20
#
# 13:27:43.870 [info]  Start: add 5
#
# 13:27:43.870 [info]  End: add 5
#
# 13:27:43.870 [info]  Start: add 10
#
# 13:27:43.870 [info]  End: add 10

About

Elixir library to build simple data pipelines

Topics

Resources

License

Packages

No packages published

Languages

You can’t perform that action at this time.