Repository navigation
v0.3.0 — GaLore, Long-Context, Synthetic Data Pipeline
What's New
GaLore Optimizer Integration
Full-parameter training via gradient low-rank projection — an alternative to LoRA that updates all weights while using less optimizer memory. 6 optimizer variants supported.
training:
galore_enabled: true
galore_optim: "galore_adamw_8bit"
galore_rank: 128Long-Context Optimizations
Train on sequences >4K tokens with RoPE scaling, NEFTune noise injection, sliding window attention, and sample packing.
training:
rope_scaling: { type: "yarn", factor: 4.0 }
neftune_noise_alpha: 5.0Synthetic Data Pipeline
Generate training data from a teacher model (API or local) with --generate-data.
forgelm --config my_config.yaml --generate-dataGPU Cost Estimation
Auto-detection for 18 GPU models with per-run cost tracking.
Additional
- PyPI:
pip install forgelmnow works - Adversarial Prompts: 140 prompts across 6 categories (was 50/3)
- Nightly CI: Automated compatibility testing against latest deps
- Wizard: GaLore, long-context, NEFTune options added
- 19 code review fixes: Security (API key redaction, webhook URL masking), correctness (SLERP normalization, eos_token crash), consistency (version fallback, success definitions)
Stats
- 297 tests (up from 242), 0 failures
- 17 Python modules, ~3,500 lines
- 47 files lint-clean (ruff check + format)
Full Changelog: https://github.com/cemililik/ForgeLM/blob/main/CHANGELOG.md