FlowHub Base Plugin Pack for Heddle

The FlowHub Base Plugin Pack (fhub-base) provides essential base processing steps and stateful resources for the Heddle execution environment. Designed natively using the Heddle Software Development Kit (SDK) for Go, this repository delivers high-performance database querying and dynamic Large Language Model (LLM) prompt orchestration steps. It leverages Heddle's columnar data streaming format for fast, zero-copy data pipelines.

Note

This repository serves both as a library of production-ready steps and as a reference implementation for building advanced plugins using the Heddle SDK for Go.

---

📋 Table of Contents

• Features
• Project Architecture
• Prerequisites & Installation
• Configuration and Environment Variables
• Quick Start: Direct Execution
• Local Development & Contribution
• License & Support

---

✨ Features

This base plugin pack includes key capabilities optimized for high-performance data processing:

• PostgreSQL Database Connector: A lifecycle-managed database connection pool resource that wraps pgx and handles initialization, queries, and termination.
• Dynamic SQL Schema Discovery: Automated runtime Structured Query Language (SQL) schema inspection that maps database column types directly to Heddle column types.
• Multi-Provider LLM Orchestrator: Uniform prompting capabilities across major providers including OpenAI, Anthropic, Gemini, Ollama, and Llama.cpp.
• Reasoning Extraction: Specialized response mapping that captures reasoning and thinking tracks for advanced models like OpenAI o1 or Anthropic Claude 3.7.
• Double Execution Mode: Direct local in-memory execution for testing or standard remote gRPC-based Inter-Process Communication (IPC) daemon mode.

---

🏗️ Project Architecture

The codebase organizes logic into steps, executable runner flows, and binary entry points:

graph TD
    cmd_flow["cmd/flow (Direct local runner)"] -->|Loads| flow_pkg["flow (Orchestration packages)"]
    cmd_register["cmd/register (gRPC plugin daemon)"] -->|Starts IPC server| sdk_plugin["Heddle SDK Plugin"]
    flow_pkg -->|Executes steps via| sdk_plugin
    sdk_plugin -->|Wires resources & runs| step_postgres["step/postgres (Postgres Steps)"]
    sdk_plugin -->|Runs API calls| step_llm["step/llm (LLM Steps)"]
    step_postgres -->|Leverages| postgres_res["Connection (Stateful Resource)"]

Loading

The primary directories structure the functionality as follows:

• cmd/: Houses binary entry points. cmd/flow runs local workflows directly. cmd/register starts the steps as gRPC plugins.
• flow/: Defines the orchestration loops that chain steps together.
• step/: Implements the actual processing steps. step/postgres handles SQL execution. step/llm manages external generative intelligence providers.

---

🛠️ Prerequisites & Installation

Prerequisites

Ensure your environment meets the following specifications:

• Go: Version 1.26 or higher.
• Operating System (OS): A POSIX-compliant system (such as Linux or macOS) to support Unix Domain Sockets (UDS) and Shared Memory (SHM) integrations.

Installation

Clone the repository into your Heddle workspace or Go development directory:

git clone https://github.com/galgotech/fhub-base.git
cd fhub-base

Verify your installation by running the test suite:

go test ./...

---

⚙️ Configuration and Environment Variables

The binaries utilize environment variables to configure stateful resources and connection options:

Postgres Connection Options

• PGHOST: Specifies the host address of the database (defaults to localhost).
• PGPORT: Specifies the connection port of the database (defaults to 5432).
• PGUSER: Defines the username for database authentication (defaults to postgres).
• PGPASSWORD: Defines the password for database authentication (defaults to postgrespassword).
• PGDATABASE: Selects the target database name (defaults to agent_news).
• PGSSLMODE: Configures the transport security protocol (defaults to disable).

LLM Orchestrator Options

• FLOW_TYPE: Sets the execution pipeline (choose postgres or llm).
• LLM_PROVIDER: Selects the model provider (choose openai, anthropic, gemini, ollama, or llama_cpp).
• LLM_API_KEY: Supplies the credential Application Programming Interface (API) key for the chosen provider.
• LLM_MODEL: Overrides the default model name.
• LLM_BASE_URL: Overrides the default API gateway host address.

---

🚀 Quick Start: Direct Execution

You can run the base plugin pipelines locally without a running Heddle engine. This enables rapid validation of database queries and model generations in-process.

Running the Postgres Query Flow

1. Ensure a PostgreSQL database is running and accessible.
2. Run the direct flow command:

export FLOW_TYPE="postgres"
export PGHOST="127.0.0.1"
export PGPORT="5432"
export PGUSER="postgres"
export PGPASSWORD="yourpassword"
export PGDATABASE="yourdatabase"

go run ./cmd/flow

The local runner will configure the Postgres connection pool, execute a table generation query, insert test records, and print the active rows:

Postgres resource accessed successfully!

--- Step 1: Initialize temporary table and populate data ---
Temporary table setup and population completed successfully.

--- Step 2: Query active users ---
Frame output: 3 rows
Row 0: { active: true, id: 1, name: Alice Smith, role: Admin }
Row 1: { active: true, id: 2, name: Bob Jones, role: Developer }
Row 2: { active: true, id: 4, name: Diana Prince, role: Manager }
Direct flow execution finished successfully!

Running the LLM Prompt Flow

1. Set your provider configurations.
2. Execute the runner command:

export FLOW_TYPE="llm"
export LLM_PROVIDER="openai"
export LLM_API_KEY="sk-..."

go run ./cmd/flow

The runner will output the model response along with any extracted reasoning data:

--- Running LLM Prompt Flow ---
Using LLM Provider: openai, Model: gpt-4o-mini
Prompting LLM: "Tell me a short, 1-sentence joke about databases."

--- LLM Response ---
Response: Why do database administrators always look so stressed? Because they have too many relations!

---

🛠️ Local Development & Contribution

We welcome contributions to this repository. You can verify changes, execute tests, and build binary assets using the following instructions:

Running Unit Tests

The test suite includes standard mocks for HTTP and database services. Run them using:

go test -v -race ./...

Creating New Steps

To extend the capability of this base pack:

1. Define your new action step within a distinct package in the step/ folder.
2. Register your steps group in both cmd/flow/main.go and cmd/register/main.go.
3. Add corresponding integration tests under your step package directory.

---

📜 License & Support

The Apache License, Version 2.0 governs this project. If you encounter any bugs, or have feature proposals, please submit an issue on the official bug tracker.