How to complete example

How to complete example

Step 1: Install Jetline using pip

To install Jetline, run the following command:

pip3 install --upgrade jetline

or

pip install --upgrade jetline

Step 2: Setup your project

To setup Jetline, run the following command (project name and folder name for your pipes are optional default is "project" & "pipelines":

jetline-setup

Step 3: Navigate to your project folder

Before proceeding with further commands, ensure that your current directory is your project's root folder. Navigate to your project folder using the following command:

cd YOUR/PROJECT/PATH

This should look something like this:

In the project folder, you will find the following files and directories:

• project.toml: This file contains system configurations for your project.
• data.py: This Python script stores all data connections, and initializes the data before running the pipelines.
• main.py: This file executes the pipelines.
• pipelines: This directory contains the pipeline scripts.

Step 4: Add a Data Class to your data.py

Now, let's add a data class to your data.py file. For this example, we'll use fruit data. Ensure that the class inherits from Data class from jetline.data.data.

from jetline.data.data import Data

class Fruits(Data):
    def __init__(self):
        super().__init__(
            name="Fruits",
            data=self.__getter__()
        )

    def __getter__(self):
        return ['banana', 'apple', 'mango']

    class Meta:
        description = "This is the Fruit Data Class"
        y = 0
        x = 0

In this data class, name is a string representing the name of the data, and data can be of any type, ranging from strings to pandas DataFrames. The Meta class is optional and is mainly used for visualization purposes, providing information like description, y, and x for the graph.

Step 5: Create a Pipe

Now, let's create a pipe for your project. You can name it according to your preference. For this example, let's name it raw.

To create a new pipe, run the following command in your terminal:

jetline-create-pipe

After running this command, you should see the raw folder along with its contents generated within your pipelines directory.

Step 6: Define Node Registration in pipeline.py

Now, let's define the node registration in your pipeline.py file. This step involves registering functions as nodes within your pipeline. Ensure consistency in design for clarity and maintainability.

from jetline.pipeline.pipeline import Pipeline
from jetline.pipeline.node import Node
from raw import nodes as func

def register(data_manager) -> Pipeline:
    return Pipeline(
        nodes=[
            Node(
                function=func.validate,
                inputs=["Fruits"],
                outputs=["Fruits"],
                viz={"y": 150, "x": 0, "sources": ['Fruits']}
            ),
            Node(
                function=func.export,
                inputs=["Fruits"],
                viz={"y": 300, "x": 0, "sources": ['validate']}
            ),
        ],
        data_manager=data_manager)

In this registration, we define two functions (validate and export) as nodes in the pipeline. The inputs and outputs lists contain strings corresponding to the names defined in data.py. The viz dictionary is optional and used for visualization purposes.

Step 7: Define Nodes in nodes.py

Next, let's define the nodes in your nodes.py file. Nodes are essentially functions that perform specific tasks within your pipeline.

def validate(fruits):
    if 'apple' in fruits:
        fruits.remove('apple')
    return fruits

def export(fruits):
    with open('fruits.txt', 'w') as file:
        for fruit in fruits:
            file.write(fruit + '\n')

In this example, validate removes 'apple' from the list of fruits, and export writes the fruits to a text file named 'fruits.txt'. These functions represent the nodes in your pipeline.

Step 8: Run Pipelines

Now, let's set up your main.py file to run the pipelines.

from jetline.pipeline.pipeline import PipelineManager

def run_pipelines():
    """Run administrative tasks."""
    try:
        pipeline_manager = PipelineManager()

        # Define the order of pipelines by folder name
        #   [["example_pipeline"], ["example_pipeline2"]] to run after
        #   [["example_pipeline", "example_pipeline2"]] to run at the same time
        #   [["example_pipeline", "example_pipeline2"], ["example_pipeline3"]] combination
        PIPELINE_ORDER = [['raw']]

        pipeline_manager.run(PIPELINE_ORDER)
    except Exception as e:
        raise RuntimeError(f"Error running pipelines: {e}") from e

if __name__ == "__main__":
    run_pipelines()

In this code, we define a function run_pipelines() that initializes a PipelineManager object and runs the pipelines. The PIPELINE_ORDER variable determines the order in which pipelines are executed. Pipelines listed in the same sublist will run concurrently, while those listed in separate sublists will run sequentially.

In your main.py, add the raw pipeline to the PIPELINE_ORDER list to register and run it. All pipelines registered here will be executed when you run main.py.

PIPELINE_ORDER = [['raw']]

To run your pipeline, execute the following command in your terminal:

jetline-run

This command will execute all the pipelines defined in your main.py file, including the raw pipeline that you've set up. Make sure you are in the correct directory containing your project files when running this command.

<img width="1023" alt="Screenshot 2024-04-30 at 11 53 36" src="https://github.com/SheezyJay/Jetline/assets/112690216/6a2a5249-bd1d-4528-baec-2760454040ea">


## To visualize your pipelines, run the following command:

```bash
jetline-viz

Screen.Recording.2024-04-30.at.12.32.09.mov