Skip to content

v0.3.0 — GaLore, Long-Context, Synthetic Data Pipeline

Choose a tag to compare

@cemililik cemililik released this 28 Mar 00:30
· 806 commits to main since this release

What's New

GaLore Optimizer Integration

Full-parameter training via gradient low-rank projection — an alternative to LoRA that updates all weights while using less optimizer memory. 6 optimizer variants supported.

training:
  galore_enabled: true
  galore_optim: "galore_adamw_8bit"
  galore_rank: 128

Long-Context Optimizations

Train on sequences >4K tokens with RoPE scaling, NEFTune noise injection, sliding window attention, and sample packing.

training:
  rope_scaling: { type: "yarn", factor: 4.0 }
  neftune_noise_alpha: 5.0

Synthetic Data Pipeline

Generate training data from a teacher model (API or local) with --generate-data.

forgelm --config my_config.yaml --generate-data

GPU Cost Estimation

Auto-detection for 18 GPU models with per-run cost tracking.

Additional

  • PyPI: pip install forgelm now works
  • Adversarial Prompts: 140 prompts across 6 categories (was 50/3)
  • Nightly CI: Automated compatibility testing against latest deps
  • Wizard: GaLore, long-context, NEFTune options added
  • 19 code review fixes: Security (API key redaction, webhook URL masking), correctness (SLERP normalization, eos_token crash), consistency (version fallback, success definitions)

Stats

  • 297 tests (up from 242), 0 failures
  • 17 Python modules, ~3,500 lines
  • 47 files lint-clean (ruff check + format)

Full Changelog: https://github.com/cemililik/ForgeLM/blob/main/CHANGELOG.md