jarvis v2.0.0

Overview

The second public release of jarvis: a compact, GPT-style causal language model focused primarily on Russian, implemented from scratch in PyTorch. This release ships a larger model, a new 32,768-token tokenizer, a 2.5× larger pretraining corpus, an optional system prompt, and a streaming corpus builder that keeps the whole data pipeline inside a single Kaggle notebook.

Highlights

• 138M-class decoder-only Transformer with 16 layers, 12 attention heads, a 1,024-token context, RoPE, RMSNorm, SwiGLU, tied embeddings and depth-scaled residual initialisation.
• New byte-level BPE tokenizer with a 32,768-token vocabulary, NFC normalisation, per-digit splitting and a <|system|> role token.
• Optional system prompt that is never hardcoded: --system "..." or JARVIS_SYSTEM_PROMPT. About half of the SFT dialogues carry it, so the model works both with and without one.
• Two-stage training recipe: 3.2B pretraining tokens (web 88%, Wikipedia 4%, news 3%, dialogue 5%) followed by 200M SFT tokens with assistant-only loss.
• Streaming data pipeline: prepare_stream.py filters and tokenizes web, wiki and news in one pass, so a multi-billion-token corpus fits on a 20 GB disk.
• Training utilities: mixed-precision training, multi-GPU DDP, resumable sessions, periodic checkpoints, validation and sample generation.
• Local inference and evaluation: KV-cache generation, interactive console chat, optional Telegram bot, and a BPC benchmark that can score several checkpoints on byte-identical text.

Changes since v1.0.0

	v1 (jarvis-112m)	v2 (jarvis-138m)
Parameters	111.7M	138.4M
Layers	14	16
Context	768	1,024
Vocabulary	16,384	32,768
Pretraining tokens	1.2B	3.2B
SFT tokens	150M	200M
System prompt	—	<|system|>

Model checkpoint

The GitHub source archive does not include model weights. Download the fine-tuned checkpoint and matching tokenizer from Eragon1/jarvis-138m on Hugging Face. The model card documents intended use, data sources, evaluation and limitations.

After cloning this repository and installing requirements.txt, download the two files with huggingface_hub and start a chat:

python - <<'PY'
from huggingface_hub import snapshot_download

snapshot_download(
    repo_id="Eragon1/jarvis-138m",
    local_dir="checkpoints/hf-jarvis-138m",
    allow_patterns=["jarvis-138m-sft-fp16.pt", "tokenizer.json"],
)
PY

python chat.py \
  --ckpt checkpoints/hf-jarvis-138m/jarvis-138m-sft-fp16.pt \
  --tokenizer checkpoints/hf-jarvis-138m/tokenizer.json \
  --system "Ты — Jarvis, дружелюбный русскоязычный помощник."

See README.md for setup, training, bot configuration and the full project layout.

Evaluation snapshot

BPC (bits per character; lower is better) on a 600,000-character held-out sample from each source, 768-token windows. All three checkpoints were scored with the same script on the same text, so the rows are directly comparable:

Model	web_ru	texts_ru	chat_ru
jarvis v2 SFT checkpoint	1.08	1.10	0.88
jarvis v2 pretraining checkpoint	1.02	1.01	0.91
jarvis v1 SFT checkpoint	1.14	1.14	0.92
gzip / xz	4.07 / 3.08	4.19 / 3.03	3.57 / 2.68

v2 improves on v1 on every source: about 5% on web, 3.5% on news and 4% on dialogue. SFT gives up a little web and news likelihood in exchange for better dialogue, as expected. The v1 release reported 1.22 / 1.07 / 0.91 on its own sample; those numbers are not comparable with this table. Full measurements are in results/bpc_v2.json.

At the end of pretraining the held-out perplexity was 16.6 (web_ru), 11.5 (chat_ru), 17.3 (wiki_ru) and 18.6 (texts_ru); after SFT it is 7.6 on dialogue.

This is a small, domain-specific language-modeling check, not a comprehensive evaluation of dialogue quality, factuality or safety.

Known limitations

jarvis is an experimental, relatively small model. It can produce incorrect, repetitive, biased or unsafe text and has not been comprehensively evaluated for factuality, reasoning or safety. Factual questions, arithmetic and multi-step instructions remain unreliable at 138M parameters. The model is intended primarily for Russian, has a 1,024-token context window and should not be used for high-stakes decisions. The training datasets have separate licenses and terms; review the linked dataset cards before use or redistribution.

License

The source code is licensed under Apache-2.0; see LICENSE. The model checkpoint is published separately on Hugging Face. Upstream dataset terms remain applicable.

🇷🇺 По-русски

Второй публичный релиз jarvis — экспериментальной GPT-подобной модели для русского языка примерно на 138 млн параметров. По сравнению с первой версией модель крупнее (16 слоёв, контекст 1024), корпус претрейна вырос до 3,2 млрд токенов, появился новый BPE-токенизатор на 32 768 токенов и опциональный системный промпт <|system|>, который не хранится в репозитории.

Веса и совместимый токенизатор находятся на Hugging Face; в GitHub-архив исходный код входит без весов. По BPC модель лучше первой версии на всех трёх источниках (1.08 / 1.10 / 0.88 против 1.14 / 1.14 / 0.92), но по-прежнему слабо знает факты и плохо считает: это исследовательский артефакт, а не ассистент. Лицензии датасетов нужно проверять отдельно.