μPipe

Pipeline microframework for data processing

---

Microframework

μPipe (/mjuː.paɪp/) is a microframework for implementing data processing pipelines simply and very quickly. Depending on the type of processing to be done, you want an easy-to-use tool running in minutes, not a complex tool with dedicated hosting. That is why this project was born.

Build

To compile μPipe for Java, you must ensure that Java 11 (or above) and Maven are correctly installed.

#> mvn package
#> mvn install

To speed up process, you can ignore unit tests by using: -DskipTests=true -Dmaven.test.skip=true.

How to use

<dependency>
  <groupId>io.github.thibaultmeyer</groupId>
  <artifactId>mupipe</artifactId>
  <version>x.y.z</version>
</dependency>

final Pipeline pipeline = Pipeline.newBuilder()
    .addSource(new CollectionSource<>(List.of("1", "2", "3", "4", "5", "6")))
    .addTask((element, dataStore, isLastElementFromSource) -> Integer.parseInt(element))
    .addTask(new FilterTask<>((element) -> element % 2 == 0))
    .addSink(new PrintStreamOutputSink<>(System.out))
    .build();

License

This project is released under terms of the MIT license.

Third party

Logo is from Flaticon with free usage for personal and commercial purpose. Pipeline icons created by surang - Flaticon