DCA Inference Engine

DCA Inference Engine is a C++17 research inference engine for testing Discrete Computer Arithmetic (DCA) on quantized language models.

The short version: this project tries to make transformer inference look less like "a pile of floats" and more like an auditable finite computation. Tokens, weights, activations, recurrent state, KV state, samplers and chat history are represented as explicit integers or fixed-point values.

The current target is a Qwen3.5/Qwen3-Next style GGUF model. The engine can load GGUF metadata and tensors, tokenize with a Qwen-style byte-level BPE path, generate real tokens through a 24-layer integer/fixed-point path, stream output, and run multi-turn chat with reusable session state.

Relationship To DCA

This repository is the inference-engine companion to DCA: Discrete Computer Arithmetic.

The DCA repository is the broader finite-computation-oriented draft: it keeps the definitions, scope, mathematical notes, proof sketches and references around arithmetic, algebra, discrete analysis, formal verification, quantized AI computation and related finite structures.

This repository is narrower and more practical:

• it treats the DCA draft as the arithmetic and engineering contract;
• it implements one concrete LLM inference path under that contract;
• it turns DCA ideas into inspectable code, tests, tensor layouts, tokenizers and session-state behavior;
• it provides a place to compare CPU reference kernels with future accelerator kernels.

So the relationship is:

DCA-Discrete-Computer-Arithmetic
  -> concepts, definitions, boundaries, references

DCA-Inference-Engine
  -> executable transformer inference experiment under those boundaries

The goal here is not to claim a new universal AI runtime. More modestly: this is a working laboratory for first testing whether the DCA route can run through end to end. Code optimization and hardware backends are deliberately left for the next stage.

Why This Exists

Modern LLM inference is usually optimized around floating-point kernels. That is fast, but it can make exact behavior harder to inspect: rounding, overflow, quantization, sampling and cache state often live in several different mental models.

DCA takes the opposite angle:

• every value has a finite representation,
• every rounding or saturation rule is explicit,
• model state can be inspected as integer arrays,
• tests can run without private model files,
• deterministic replay and session-cache comparison are first-class checks.

This is still a research prototype. It is not trying to beat llama.cpp today. It is trying to be a clean, hackable place to explore integer-first LLM inference.

Main Selling Points

• Integer/fixed-point inference path: Q8.8 activations, Q16 weights and nonlinear maps, integer RoPE, finite-table softmax and bounded integer sampling.
• Real token generation: the default path runs embedding, 24 decoder layers, output norm, tied output head and full-vocabulary argmax/sampling.
• Reusable chat state: --chat keeps token history, recurrent state and K/V state across turns instead of replaying the whole conversation.
• Streaming output: --stream emits token pieces as they are selected.
• Tokenizer discipline: GGUF vocabulary, token types, ranked BPE merges, special tokens and Qwen3.5 pre-tokenization are tested explicitly.
• Distributable CI: tests create a synthetic GGUF fixture at runtime, so a fresh clone can run ctest without a private model file.
• Hardware-friendly direction, not yet a backend: the arithmetic contract is shaped so later accelerator kernels can be compared against the CPU reference. No Ascend/CANN or other NPU backend is implemented yet.

Current Status

Working today:

• Windows/MSVC CMake build and GitHub Actions CI.
• GGUF v3 header, metadata, tensor-info and aligned tensor-data reading.
• Tensor loading for F32, F16, I8, Q4_K, Q5_K, Q6_K and Q8_0 paths.
• Direct Q4_K/Q5_K/Q6_K dot products with int8 activations and Q16 accumulators.
• Integer RMSNorm, finite lookup-table sigmoid/exp/softplus/SiLU, integer RoPE and finite softmax.
• DCA layered generation, integer Top-K/Top-P/temperature sampling, streaming and multi-turn chat.
• Optional target-model validation for the local Qwen GGUF layout.

Honest limits:

• The engine is correctness-first scalar code, not a production-speed runtime.
• Code-level performance optimization has not started in earnest; the current milestone is proving that the DCA inference path is executable and testable.
• Hardware optimization has not started; Ascend/CANN and other accelerator notes are roadmap material, not current functionality.
• The main tested platform is Windows with MSVC.
• Model quality is experimental while tokenizer and layer math continue to be compared against external references.
• Full target-model tests require you to provide a compatible GGUF model file.

Quick Start

1. Install Tools

On Windows:

• Visual Studio 2022 Build Tools
• CMake 3.15 or newer
• Ninja, or the Visual Studio CMake generator

2. Build

From a Visual Studio developer shell:

cd E:\DCA\dca_transformer
cmake -S . -B build_codex -G Ninja -DCMAKE_BUILD_TYPE=Release -DCA_BUILD_TESTS=ON
cmake --build build_codex --config Release

3. Run Tests

ctest --test-dir build_codex --output-on-failure

Expected public test result:

100% tests passed, 0 tests failed out of 2

The e2e test creates a tiny GGUF file automatically and validates parser, tokenizer, tensor loading, DCA arithmetic, streaming and session reuse.

Running A Model

Set a local GGUF model path:

$env:DCA_TEST_MODEL = 'E:\DCA\qwen3.5-2b-q4km\Qwen3.5-2B-Q4_K_M.gguf'
.\build_codex\dca_test_e2e.exe

Single prompt:

.\build_codex\dca_transformer.exe `
  --model 'E:\DCA\qwen3.5-2b-q4km\Qwen3.5-2B-Q4_K_M.gguf' `
  --prompt 'DCA' `
  --max-tokens 2 `
  --echo `
  --verbose

Streaming:

.\build_codex\dca_transformer.exe --model '<model.gguf>' --prompt 'DCA' --max-tokens 8 --stream

Multi-turn chat with reusable state:

.\build_codex\dca_transformer.exe `
  --model '<model.gguf>' `
  --chat `
  --max-tokens 64 `
  --system 'You are concise.'

Scripted chat and cold-replay comparison:

.\build_codex\dca_transformer.exe `
  --model '<model.gguf>' `
  --chat-script '.\chat_turns.txt' `
  --max-tokens 16 `
  --compare-session

Example comparison from a local target model:

turn 1: reused prefix tokens 0,  reuse time 25.0864s, cold replay 23.7718s, output match yes
turn 2: reused prefix tokens 16, reuse time 20.2476s, cold replay 48.9397s, output match yes

CLI Cheatsheet

--model PATH              GGUF model path
--prompt TEXT             Single prompt
--prompt-file PATH        Read prompt from UTF-8 file
--stdin                   Read prompt from stdin
--max-tokens N            Maximum generated tokens
--temperature F           Q16 temperature; 0 means greedy
--top-k N                 Integer Top-K; 1 means greedy
--top-p F                 Q16 nucleus threshold
--seed N                  Integer sampler seed
--stream                  Stream token pieces
--chat                    Interactive chat REPL
--chat-script PATH        One user turn per UTF-8 line
--compare-session         Compare session reuse with cold replay
--head-only               Diagnostic embedding/norm/output-head path
--deterministic-smoke     Deterministic test generator

DCA Arithmetic Contract

The inference path is intentionally explicit:

• machine words are finite words, not mathematical integers;
• K-quant binary16 scales are converted to Q16 before inference arithmetic;
• activations are Q8.8 between current sublayers;
• projection paths use int8 activations with recorded right shifts;
• nonlinear maps are finite Q16 lookup tables with integer interpolation;
• Q16 multiply-shift operations use saturating helpers;
• sampling uses Q16 controls, finite integer candidate arrays and a uint64_t xorshift RNG;
• chat sessions keep token ids, Q8.8 hidden state, Q16 recurrent state, Q16 K/V state and explicit position counters.

Details live in docs/dca_arithmetic_contract.md.

Future Accelerator Direction

DCA is not tied to one hardware vendor. The interesting part is the shape of the computation: explicit integer/fixed-point tensors, bounded lookup tables, finite state and deterministic kernels.

This section is a roadmap, not a claim of current acceleration. Today the engine is a CPU reference implementation used to test whether the DCA inference path can run correctly.

That shape is especially relevant to China-made accelerator stacks such as Huawei Ascend:

• Ascend/CANN exposes custom operator development through Ascend C and a hardware model built around AI Core style vector/matrix execution.
• DCA kernels are already decomposed into small finite operators such as integer matvec, RMSNorm, lookup-table nonlinearities, finite softmax and stateful KV/recurrent updates.
• The current scalar kernels can be used as a reference implementation before writing Ascend C kernels.
• Q8.8/Q16-style contracts make quantization, rounding and saturation choices visible at the operator boundary, which is useful when porting to NPU toolchains.

Near-term accelerator work should target:

1. First keep expanding CPU reference tests and output comparison fixtures.
2. Then port Q4_K/Q5_K/Q6_K packed-weight matvec kernels.
3. Then port Q8.8 -> int8 activation packing.
4. Then port integer RMSNorm, residual saturation and finite softmax.
5. Finally move session-reuse decode kernels toward accelerator execution.

See docs/accelerator_roadmap.md.

Repository Map

src/
  dca_core/          finite words and arithmetic helpers
  dca_tensor/        integer tensor kernels
  gguf_reader/       GGUF parser, K-quant loading and projection helpers
  tokenizer/         Qwen byte-level BPE tokenizer
  transformer/       DCA norm, softmax, layer helpers and KV state
  model/             Qwen3.5-style inference and session API
docs/                design notes, arithmetic contract and test reports
tools/               helper scripts such as tokenizer golden export
tests/data/          checked-in small test corpora

Documentation

• Documentation index
• Relationship to DCA
• DCA arithmetic contract
• Session reuse chat
• Tokenizer parity harness
• CI synthetic GGUF fixture
• Target tensor layout validation
• Accelerator roadmap

Contributing

Contributions are welcome, especially around:

• tokenizer parity,
• GGUF model coverage,
• integer kernel correctness,
• CPU reference fixtures for future Ascend/CANN or other NPU backend experiments,
• documentation and reproducible tests.

Please read CONTRIBUTING.md. Keep the DCA contract visible: new inference code should state its finite representation, rounding and overflow behavior.

License

Apache License 2.0. See LICENSE.

Copyright 2026 Wang Bingqin.