Installation

I'm using Ubuntu (server) 24.04.4 LTS on a ~10 year old machine with 32GB 800MHz DDR3, Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz, SSD for the OS + models. The GPU is a NVIDIA GeForce GTX 1060 with 6GB VRAM, so I appear to be limited to LLMs of 8 billion parameters (or less).

The LLMs are running on Ollama, which can be installed by running the following command:

curl -fsSL https://ollama.com/install.sh | sh

Ollama is a wrapper for llama.cpp by Georgi Gerganov. It abstracts away the configuration required to run llama.cpp via scripts which detect likely good choices and exposes OpenAI model compliant LLMs through a local API endpoint or chat interface directly on the terminal. Ollama also hosts LLMs and allows you to effortlessly pick and choose a (tagged) model that suits your needs. In my case 6GB VRAM is a bottleneck. Replicating the flow presented by mmajurski/NIST requires a model that implements "thinking" (reasoning tokens).

For me, that means nemotron-3-nano:4b. Here is a screenshot of the Ollama site, "Models" section (Note: The search bar):

^^Underlined in red is the "thinking" tag we're after (reasoning tokens)

^^Underlined in red is the default context length, which effectively represents how much memory running the model will require

Note: Ollama will happily run a model that uses more memory than can be provided by your physical VRAM. It does this by 1. Swapping to disk in the event you are constrained by mainboard memory (RAM) or 2. Using a combination of mainboard memory and GPU memory. By adjusting the model context size via the OLLAMA_CONTEXT_LENGTH environment variable, you can fine tune Ollama so that it makes most efficient use of your fastest components.

The command ollama ps will show you how resources are being used, as in the following example:

NAME	ID	SIZE	PROCESSOR	CONTEXT	UNTIL
nemotron-3-nano:4b	6cc467f05439	5.0 GB	100% GPU	167936	Forever

In order to get the highest memory use entirely on GPU, I came to use the value (164 x 1024) = 167936 through trial and error. The number 167936 is tokens, not MB, and the length of a token can vary from model to model.

More information on getting started with Ollama can be found here.

Note: Before installing Ollama, install the NVIDIA drivers recommended for your graphics card. The rule of thumb seems to be that "if nvidia-smi returns results (rather than an error), you should be good to use your GPU/VRAM with Ollama".

In my case, I installed Ollama before installing the NVIDIA drivers, assuming that they were detected by the Ubuntu installation process, since I selected "use third-party drivers" (enabling the restricted, multiverse apt package repositories), but this proved to be incorrect and Ollama threw an error explicitly identifying which drivers to install for my graphics card (580x), which is curious because ubuntu-drivers devices recommends the 535x "non-free" (NVIDIA). I have yet to benchmark performance on the 535x vs. the 580x (if they even work), because at this point I'm only looking for a working proof of concept.

uv is a fast Python package manager, written in Rust. Installation instructions in detail are available here. It enables you to fetch dependencies (libraries) from their official distribution sources faster than when using pip alone.

1. Install uv

curl -LsSf https://astral.sh/uv/install.sh | sh

2. Create a directory for airflow at your desired location

mkdir airflow

3. Create virtual environment for airflow

uv venv --python=3.12

4. Activate the newly created venv

source .venv/bin/activate

5. Install airflow via uv

uv pip install "apache-airflow[celery]==3.2.2" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-3.2.2/constraints-3.12.txt"

6. Install openai provider for airflow

uv pip install apache-airflow-providers-openai

7. Install common ai module

uv pip install apache-airflow-providers-common-ai

8. Run airflow

airflow standalone

Note: On first run, the admin password will be echoed to terminal. After the first run, you can find it in the file simple_auth_manager_passwords.json.generated in the directory

9. Create Pydantic AI connection

Create a connection to your OpenAI model compliant LLM via the Apache Airflow UI at http://localhost:8080 (do not use "standalone" for production), using the following:

• Connection Type: Pydantic AI
• Connection ID: openai_default
• Description: [optional]
• Host: http://127.0.0.1:11434/v1
• API Key: [not required but apparently cannot be left blank]
• Extra JSON: { "extra": { "model": "openai:nemotron-3-nano:4b" } }

Note: "model" in extra JSON must be prefixed with "openai:", but you can use whatever model is desired, as long as it supports "thinking"

10. Create your directed acyclic graph (DAG)

Note: The default path for directed acyclic graphs (DAG) is "./dags/".