Hilbert

---

WARNING: This model has NOT been trained yet. This repository contains the complete architecture specification, production CUDA kernels, GGUF export pipeline, and training configuration. Weights do not exist. Do not attempt to load or deploy — there is no checkpoint to load. Training requires 8x H100 GPUs (or equivalent compute).

---

Sovereign language model built from scratch. No fine-tune. No LoRA. No dependency on NVIDIA, Meta, Mistral, or any other lab's weights.

Named after David Hilbert — the mathematician who attempted to formalize all of mathematics.

What This Is

A complete, from-scratch transformer trained exclusively on public internet data. Every component is original:

• Architecture: Standard decoder-only transformer (GQA + SwiGLU + RMSNorm + RoPE)
• CUDA Kernels: Hand-written for H100/A100 tensor cores
• Training Data: Public domain + permissive-licensed sources only
• GGUF Export: Direct deployment via llama.cpp without any vendor lock-in

Why

Fine-tuning is a lease. You modify someone else's weights, you carry their name, you depend on their license, and you can't clearly say what's yours. If NVIDIA changes their license tomorrow, your "fine-tune" is gone.

Hilbert is built from the ground up so the weights are ours. The name is ours. The architecture is public knowledge that no one can patent. The training data is public. The result is sovereign.

Architecture

Parameter	Value
Parameters	~4B
Hidden Dim	2048
Layers	24
Attention Heads	16 (GQA: 4 KV heads)
MLP Dim	5504 (SwiGLU)
Context	8192 tokens
Vocab	32K BPE
Position	RoPE (theta=10000, full head rotation)
Norm	RMSNorm (eps=1e-5)
Training	BF16, AdamW, cosine LR
Deployment	Q4_K_M via GGUF/llama.cpp

Structure

hilbert/
├── config/
│   └── architecture.json       # Full structural specification
├── kernels/
│   ├── rmsnorm.cu              # Warp-shuffle RMSNorm
│   ├── attention_gqa.cu        # FlashAttention-2 GQA
│   ├── mlp_swiglu.cu           # Fused Gate+Up GEMM with SwiGLU epilogue
│   └── rope.cu                 # Rotary Position Embedding
├── gguf/
│   └── export_gguf.py          # PyTorch → GGUFv3 conversion
├── training/
│   └── pretrain_config.yaml    # Full training specification
└── LICENSE.tri

CUDA Kernels

All kernels target NVIDIA Hopper (H100) and Ampere (A100) tensor cores:

Kernel	Technique	Key Feature
rmsnorm.cu	Warp-shuffle cooperative reduction	FP32 accumulation, single-pass
attention_gqa.cu	FlashAttention-2 tiling	Online softmax, GQA head sharing
mlp_swiglu.cu	Fused dual-GEMM	Gate+Up computed simultaneously, SiLU in epilogue
rope.cu	Per-pair rotation	Supports partial rotation, KV-cache continuation

Training Data

Public sources only. No proprietary datasets. No scraped content behind ToS walls.

Source	Weight	License
RedPajama v2 (CC dedupe)	40%	Public domain
arXiv (math/physics/cs)	15%	CC-BY
GitHub (permissive only)	15%	MIT/Apache/BSD
Stack Exchange	10%	CC-BY-SA
Wikipedia (all langs)	10%	CC-BY-SA
Public domain books	5%	Pre-1928
Formal proofs (Lean4/Agda/Coq)	5%	Permissive OSS

Deployment

# Export trained checkpoint to GGUF
python gguf/export_gguf.py \
    --checkpoint checkpoints/hilbert-4b-final.pt \
    --config config/architecture.json \
    --output hilbert-4b-q4km.gguf

# Run via llama.cpp
./llama-server -m hilbert-4b-q4km.gguf -c 8192 --port 8080

Legal Clarity

• Architecture: Standard transformer (public knowledge, no patents applicable)
• Kernels: Original implementation (written by us)
• Training data: All public domain or permissive-licensed
• Weights: Trained from random initialization (no one else's weights)
• Name: Hilbert (no trademark conflict)

This model belongs to us. Not as a legal argument — as a mathematical fact.

Status

Phase	Status	Notes
Architecture Design	COMPLETE	Fully specified, all tensor shapes defined
CUDA Kernels	COMPLETE	RMSNorm, FlashAttn-2 GQA, fused SwiGLU, RoPE
GGUF Pipeline	COMPLETE	Export script ready for trained checkpoint
Training Config	COMPLETE	Data mix, hyperparameters, hardware plan
Tokenizer Training	NOT STARTED	BPE on RedPajama subset
Pre-Training	NOT STARTED	Requires 8x H100 compute
Evaluation	NOT STARTED	Benchmarks TBD post-training
GGUF Release	NOT STARTED	Q4_K_M quantized model for llama.cpp

---

Disclosure

This is an independent research project. Hilbert is not affiliated with, endorsed by, or derived from any existing model or lab.

IP Provenance:

• The transformer architecture (attention + FFN + residual) is published academic work (Vaswani et al., 2017) — no entity owns it
• Grouped-Query Attention is published (Ainslie et al., 2023) — no patent encumbrance
• SwiGLU is published (Shazeer, 2020) — standard practice
• RoPE is published (Su et al., 2021) — open technique
• FlashAttention tiling is published (Dao, 2022; Dao, 2023) — algorithm only, our implementation is original
• All CUDA kernel code in this repository is original work
• No weights, activations, distillation outputs, or training artifacts from any existing model are used
• Training data consists exclusively of public domain and permissive-licensed sources

What this is NOT:

• NOT a fine-tune of Nemotron, LLaMA, Mistral, or any other model
• NOT a quantized/pruned/distilled version of an existing model
• NOT using any proprietary training data
• NOT dependent on any vendor SDK beyond the CUDA compiler

Naming: "Hilbert" refers to David Hilbert (1862-1943), mathematician. No trademark conflict exists with any AI product or service.

---

License

Tri-licensed: BSL-1.1 + AGPL-3.0 + MPL-2.0. See LICENSE.tri.

Copyright (C) 2026 Jessica L. Williams / SNAPKITTYWEST