Skip to content

Releases: ian-cowley/Glacier.Inference.Distributed

Release list

v1.2.0 - OpenAI & Ollama Compatible Distributed Cluster HTTP Service

v1.2.0 - OpenAI & Ollama Compatible Distributed Cluster HTTP Service Latest
Latest

Choose a tag to compare

View all tags
@ian-cowley ian-cowley released this 12 Oct 20:16
v1.2.0
b20519d

🚀 Glacier Distributed Cluster v1.2.0 Release

🌐 Drop-In OpenAI & Ollama Compatible HTTP Service (--serve)

The cluster coordinator can now be launched as an OpenAI and Ollama compatible HTTP service, allowing external client applications to connect without writing code or compiling binaries:

\\�ash
glacier-distributed serve
--model ./models/qwen2-57b-a14b-instruct-q3_k_m.gguf
--stages 192.168.1.108:5055,192.168.1.101:5056,192.168.1.200:5057
--http-port 8080
--host 0.0.0.0
\\

✨ Key Features & Endpoints

  • OpenAI Chat Completions (\POST /v1/chat/completions): Full support for streaming Server-Sent Events (\stream: true) and non-streaming JSON responses.
  • OpenAI Text Completions (\POST /v1/completions): Text completion endpoint with streaming SSE.
  • OpenAI Models (\GET /v1/models): Model discovery endpoint returning cluster model metadata.
  • Cluster Status & Telemetry (\GET /v1/cluster/status, \GET /health): Detailed node topology, layer allocations, and backend hardware info.
  • Ollama Compatibility (\POST /api/chat, \POST /api/generate, \GET /api/tags): Drop-in compatibility for tools designed for Ollama.
  • Security & Authentication: Optional API key authentication via Bearer token or \x-api-key\ header.
  • Client Integration: Tested with OpenAI Python SDK (\openai.OpenAI(base_url='http://localhost:8080/v1')\), LangChain, AutoGen, Open WebUI, Continue.dev, and Cursor.

📦 Pre-compiled Standalone Native Binaries Included

  • \glacier-distributed-v1.2.0-win-x64.zip\ (Windows x64)
  • \glacier-distributed-v1.2.0-win-arm64.zip\ (Windows ARM64)
  • \glacier-distributed-v1.2.0-linux-x64.tar.gz\ (Linux x64)
  • \glacier-distributed-v1.2.0-osx-arm64.tar.gz\ (macOS Apple Silicon)

📖 Read the deep dive article on DEV.to: Want to Run Massive AI Models on a Collection of Home PCs?

Full Changelog: v1.1.0...v1.2.0

Assets 8
Loading

v1.1.0 - Heterogeneous Cluster LLM Inference & LAN Transport Runtime

v1.1.0 - Heterogeneous Cluster LLM Inference & LAN Transport Runtime

Choose a tag to compare

View all tags
@ian-cowley ian-cowley released this 12 Oct 19:36
v1.1.0
1950615

🌐 Glacier.Inference.Distributed v1.1.0

📖 Deep Dive Article

Read the complete architectural walkthrough and benchmark breakdown on DEV.to:
Want to Run Massive AI Models on a Collection of Home PCs?

🚀 Highlights & Features

  • Fused 64-Byte Aligned Socket Framing: Combines 32-byte ClusterFrameHeader with 32-byte FusedActivationHeader for naturally machine-aligned zero-copy System.IO.Pipelines streaming.
  • Ultra-Low Wire Latency: 0.45 ms per LAN hop (standard 1000BASE-T Gigabit) and 0.013 ms local loopback.
  • Dynamic Multi-Precision Wire Protocols: On-the-fly streaming in FP32, FP16, and Symmetric INT8 (5,124 bytes/token, 75% bandwidth reduction).
  • 27:21 Pareto Topology Partitioning: Tested and verified across heterogeneous local hardware (RTX 4060 + RTX 3060 + Radeon 680M APU + AMD Ryzen Zen 5 CPU).
  • Speculative Pipeline Overlap: 2-stage asynchronous coordination achieving 23.24 tok/s (1.25x speedup over 18.52 tok/s non-overlapped baseline).
  • Zero-Allocation Hot Path: 0 Gen 0/1/2 GC collections, 0 managed heap bytes allocated per forward token.
  • glacier-distributed CLI Tool: Standalone command-line interface packaged as a .NET Tool for starting coordinators and cluster workers.

📦 Packages Published

  • Glacier.Inference.Distributed 1.1.0
  • Glacier.Inference.Distributed.Cli 1.1.0

Full Changelog: https://github.com/ian-cowley/Glacier.Inference.Distributed/commits/v1.1.0

Loading