Skip to content

Releases

Release v2.0

Choose a tag to compare

@anxiangsir anxiangsir released this 06 Aug 12:39

Release v2.0

This is the first official release of LLaVA-OneVision-2, introducing a new unified visual architecture for image and video understanding, complete training and inference support, scalable data preprocessing, and reproducible checkpoint conversion and validation workflows.

Highlights

  • Introduced LLaVA-OneVision-2 model implementations for Hugging Face Transformers and Megatron training.
  • Added dense and MoE variants spanning 2B, 4B, 8B, and 30B-A3B configurations.
  • Added the OneVision Encoder and codec-aligned image/video processing for stronger spatial and temporal understanding.
  • Added alignment, mid-training, instruction tuning, quick-start, and checkpoint conversion workflows.
  • Reworked offline packing into a scalable multi-stage pipeline with WebDataset output and distributed execution support.
  • Added robust HF/Megatron checkpoint merge, validation, and consistency test suites.
  • Added Muon optimizer support and training/checkpointing reliability improvements.
  • Launched a dedicated project page with benchmarks, method visualizations, task demos, resources, and community links.

What's Changed

LLaVA-OneVision-2 Models and Training

  • Add the LLaVA-OneVision-2 architecture, training entry points, model configurations, and examples by @anxiangsir in #110
  • Fix image processor merge-size handling and update documentation by @anxiangsir in #111
  • Add Muon optimizer support by @fdcp in #81
  • Add training workflows and model variants for Qwen3 2B, 4B, 8B, and 30B-A3B backbones
  • Add dense and MoE Hugging Face implementations with image and video processors

Data Processing and Packing

  • Introduce a scalable offline packing pipeline covering sample splitting, token-length computation, bin packing, sample packing, and WebDataset conversion
  • Add distributed pipeline automation, smoke tests, and practical packing documentation
  • Add image validation, dataset inspection, frame extraction, and bounding-box quantization utilities
  • Improve JPEG handling, checkpoint configurations, training behavior, and documentation by @anxiangsir in #117

Checkpoint Conversion and Validation

  • Add checkpoint conversion scripts for all released model variants
  • Add a robust merge_ov2 CLI for dense and MoE checkpoints with explicit validators
  • Support Hugging Face Hub repository IDs in merge paths by @anxiangsir in #118
  • Replace the legacy consistency script with a pytest-based consistency suite by @anxiangsir in #119

Documentation, Evaluation, and Project Page

  • Add the LLaVA-OneVision-2 project site and GitHub Pages workflow by @anxiangsir in #121 and #122
  • Add interactive image/video task demonstrations and codec visualizations by @didizhu-judy and contributors in #138, #139, #142, and #148
  • Add benchmark results, model and dataset resources, technical report links, roadmap, and reproduction links
  • Add a streamlined 4B quick start by @anxiangsir in #164, #165, #166, and #167
  • Add automatic contributor and community metadata generation by @anxiangsir in #170 and #179

Fixes and Polish

  • Improve checkpoint, packing, training, and merge robustness
  • Improve mobile layouts, dark-mode figures, citations, resource links, and benchmark presentation
  • Add community links for Discord, WeChat, Xiaohongshu, vLLM, and Glint Lab

New Contributors

Thanks to @fdcp, @Feilong607, @didizhu-judy, @YunyaoYan, @tanhuajie, and all contributors who helped build and refine LLaVA-OneVision-2.

Full Changelog: 1.5...2.0

1.5

1.5

Choose a tag to compare

@anxiangsir anxiangsir released this 26 Dec 02:32

Release v1.5

This is the first official release of LLaVA-OneVision-1.5, focusing on documentation improvements, data packing & preprocessing reliability, training configuration defaults, evaluation additions, and multiple bug fixes.

Highlights

  • Improved documentation and onboarding experience (README, training TODOs, LICENSE updates)
  • More robust data packing / WebDataset conversion with support for mixed packing
  • Added SFT data preprocessing guidance and materials
  • Introduced evaluation-related content
  • Improved training defaults (e.g., dtype bfloat16, default 4B merge, auto config updates)
  • Numerous bug fixes across packing, filtering, and demo workflows

What’s Changed

Documentation & Project Maintenance

Data Processing / Packing / Filtering

Training Defaults & Configuration

  • set processor.image_processor.max_pixels = 1600*1600 by @yiyexy in #8
  • default merge 4b by @yiyexy in #13
  • auto update config by @yiyexy in #15
  • update default dtype to bfloat16 by @yiyexy in #18
  • Remove the hosts Settings for multiple machines and only support sing… by @chengzheng345 in #56

Evaluation

SFT Data

Bug Fixes & Robustness Improvements

  • fix a bug by @yiyexy in #30
  • fix_issue#31 by @killTheHostage in #33
  • Remove duplicate dependency py-cpuinfo libraries. by @Lornatang in #50
  • Fixed the issue that the model_path name does not correspond to the HF, causing the model to fail to load. by @Lornatang in #51
  • refactor(merge_model): Enhanced file implementation robustness by @Lornatang in #52
  • fix Stage 1.5 Mid-Training demo error. by @Lornatang in #57

New Contributors

Full Changelog: https://github.com/EvolvingLMMs-Lab/LLaVA-OneVision-1.5/commits/1.5