crit 0.1.0

Overview

First public release of crit: reusable infrastructure for building a bespoke inference
engine around a chosen model, workload and target hardware, with a common evaluation
suite and an OpenAI-compatible serving API. The Python distribution crit and the
native Core SDK (critCore) share version 0.1.0.

Added

• Core (guide): a C++20 SDK with Model, Session and Backend
  interfaces for generation and query scoring, request scheduling and admission,
  resource accounting, readiness, cancellation, deadlines, quarantine and safe
  retirement. Mixed batching of up to 4096 request lanes with shared token-work budgets,
  decode priority with prefill aging and an optional async batch pipeline. Requests
  carry sampling rules (temperature, top-k, top-p, min-p, penalties, logit bias,
  min_tokens), log-probabilities with top alternatives, stop strings, decoded text and
  per-token output events, and report phase timing and reused prompt tokens.
• Core component libraries, each optional and documented beside its sources:
  text (crit/text.h: stop matching, streaming detokenization), sampling
  (crit/sampling.h: Philox counter-based draws usable in device code, a Sampler
  with a documented rule order), state (crit/state.h: a storage-agnostic ledger of KV
  pages and recurrent state slots with prefix reuse), files and checkpoints
  (crit/json.h, crit/file.h, crit/sha256.h, crit/safetensors.h,
  crit/model_config.h), host-memory admission (crit/host_memory.h), structured output
  (crit/constraints.h: token, choice, JSON-object and byte-grammar matchers), operator
  ranges (crit/trace.h, with an optional NVTX sink), and CUDA services (graph capture,
  shared device allocations, category timing, spare device memory).
• The crit_core Python package: model files, strict JSON and safetensors readers,
  config.json rules, tokenizer, Hugging Face chat templates, generation config,
  streaming detokenization and the JSON schema/regex/EBNF compiler for structured output.
• Inference (guide): 26 techniques with implementation code, runnable
  examples, ownership contracts and adaptation guides, grouped by category: attention
  (attention-state, flash-prefill-attention, fp8-kv-attention, partitioned-attention,
  rotated-kv-quantization), projections (quantized-projections, rmsnorm-bf16,
  stored-tile-gemm), experts (async-expert-contributions, device-expert-cache,
  expert-residency, heterogeneous-expert-dispatch, ssd-expert-materialization),
  speculation (device-speculative-round, draft-vocabulary-subset, mtp-decode,
  recurrent-state-recovery), sampling and constraints (token-constraints,
  token-sampling), execution (concurrent-request-execution, recorded-selection,
  session-scheduling) and memory and loading (artifact-loading, checkpoint-storage,
  host-memory-admission, model-import-audit).
• Kernels (guide): checks for outside kernels (native integration,
  the actual device, exactness across column counts), pinned dependency retention,
  artifact verification and preservation of third-party notices, with native
  integration examples.
• Eval (guide): one speed, quality and exactness suite that
  every engine runs the same way through a small eval-v1 Python adapter, with prompts
  frozen as token ids per model and a layer-streamed FP32 reference. It reports decode,
  prefill, time to first token and quality against physical limits, computes an engine's
  max context from device memory, checks capacity and long-context behavior, scores
  held-out prompts in eight languages plus code and SQL, and offers pinned decode text
  (crit eval pin), paired A/B runs (crit eval ab), a CUPTI kernel report
  (crit eval kernels) and device probes (crit eval probe).
• Serving (guide): crit serve serves a sealed candidate as an
  OpenAI-compatible API with /v1/models, /v1/chat/completions (JSON or streamed,
  stop strings, system messages, response_format JSON object and JSON schema), optional
  API keys and CORS, and /health; --write-launchers writes serve.cmd and serve.sh.
• Run records (guide): crit run keeps retained candidate
  source and packages, lineage, checks, measurements, candidate budgets, quality and
  exactness terms, a weights policy, process reservations with owner-aware leases,
  cleanup and immutable sealed outcomes. An optional untracked settings.toml holds
  machine settings.
• Validation and packaging: host and native fixtures, one installable crit
  distribution (crit, crit_api, crit_eval, crit_kernels and crit_core), an
  installable C++ SDK and Windows/Linux CI workflows.

Inference examples include host proofs of concept and native sources with explicit
validation scopes. Model arithmetic, formats, tokenization and sampling choices belong
to each engine; model quality and device performance require target validation.