Orchestration

.gitignore

@@ -0,0 +1,2 @@
+symlink/
+symlink

README.md

@@ -1,7 +1,32 @@
 # PAM
 Personal Assistance Machine
+This artificial intelligence consists of a large language model for tropical storm simulations.
+
+# Large Language Model (LLM) Files (GGUF)
+The system is locally hosted. The prerequisite are the associated GGUF files. Please create a directory called `symlink` in the root that directs the orchestrator to find the LLM files. To create the symbolic link, use the following command by replacing the `/path/to/ggufs` with the path to the LLM files.
+
+`ln -s /path/to/ggufs ./symlink`
+
+## Updating Model
+If we are switching the LLM or otherwise updating it, we need to ensure that the llama.cpp server configuration reflects it. In the `docker/llm` directory there is a  `entrypoint.sh` file that contains the initial configuration. Please update the GGUF filename with the first in the sequence for the `--model` flag.
+
+## Downloading Models
+Currently, the design supports command line interfaces such as the [huggingface-cli](https://huggingface.co/docs/huggingface_hub/en/guides/cli). Please reference the following command to understand how to download a LLM to a locally specified directory.
+
+```
+sudo huggingface-cli download DevQuasar/swiss-ai.Apertus-70B-Instruct-2509-GGUF --include "*Q8*" --local-dir .
+```
+
+# Trusted Platform Module (TPM)
+The TPM can be configured such that the application is compliant with rigorous security standards and for data science purposes such as random number generation. The following command describes how to configure a TPM chip to work with a container.
+
+`docker run --device /dev/tpm0:/dev/tpm0 --device /dev/tpmrm0:/dev/tpmrm0`
 
 # Quickstart
 
 1. `docker build -t pam docker/`
 2. `sudo docker run -d -v .:/data/ -p 10000:10000 pam`
+
+# Networking Quickstart
+
+The reverse proxy is configured externally. For example, [this configuration file](https://github.com/hammad93/hurricane-server/blob/main/docker/proxy/conf.d/open-webui.conf) is an example of how to configure it.

docker-compose.yml

@@ -0,0 +1,18 @@
+services:
+  llm:
+    build:
+      context: ./docker/llm
+    ports:
+        - "10000:10000"
+    volumes:
+        - ./symlink:/data
+    restart: unless-stopped
+  open-webui:
+    image: ghcr.io/open-webui/open-webui:main
+    container_name: open-webui
+    ports:
+        - "8080:8080"
+    volumes:
+        - /var/lib/docker/volumes/open-webui/_data:/app/backend/data
+    network_mode: host
+    restart: unless-stopped

docker/Dockerfile → docker/llm/Dockerfile

@@ -1,11 +1,12 @@
 FROM continuumio/anaconda3
 RUN apt-get update && apt-get install -y build-essential
+# RUN apt-get install -y tpm2tools libtss2-dev
 
 # Install llama.cpp
-RUN git clone https://github.com/ggerganov/llama.cpp.git
+RUN git clone -b apertus-implementation https://github.com/pwilkin/llama.cpp.git
 WORKDIR llama.cpp
 RUN apt install cmake -y
-RUN cmake -B build
+RUN cmake -B build -DLLAMA_CURL=OFF
 RUN cmake --build build --config Release
 
 # Entrypoint

docker/llm/entrypoint.sh

@@ -0,0 +1,10 @@
+cd /llama.cpp/build/bin/
+./llama-server \
+    --model /data/swiss-ai.Apertus-70B-Instruct-2509.Q8_0-00001-of-00006.gguf \
+    --port 10000 \
+    --ctx-size 4096 \
+    --threads 64 \
+    --host 0.0.0.0 \
+    --mlock \
+    --no-mmap \
+    --offline

start.sh

@@ -0,0 +1,12 @@
+#!/bin/bash
+# Check if the symbolic link exists
+if [[ -L "./symlink" ]]; then
+    echo "Symbolic link exists: ./symlink with files:"
+    ls -la symlink/
+else
+    echo "Error: Symbolic link does not exist: ./symlink" >&2
+    exit 1
+fi
+
+docker compose build --no-cache
+docker compose up -d

stop.sh

@@ -0,0 +1,2 @@
+#!/bin/bash
+docker compose down