Hyper-Pretrained Transformers (HPT) is a novel multimodal LLM framework from HyperGAI, and has been trained for vision-language models that are capable of understanding both textual and visual inputs. HPT has achieved highly competitive results with state-of-the-art models on a variety of multimodal LLM benchmarks. This repository contains the open-source implementation of inference code to reproduce the evaluation results of HPT Air on different benchmarks.
- [5/03] 🔥 Releasing HPT 1.5 Air, our best open-sourced 8B Multimodal LLM with Llama 3. Built with Meta Llama 3, Our hyper capable HPT 1.5 Air packs a punch on real world understanding and complex reasoning. HPT Air 1.5 achieves the best results among <10B models across a wide range of challenging benchmarks (MMMU, POPE, SEED-I, and more). HPT 1.5 Air is publicly available on [HuggingFace Repository]. Please read our our [technical blog post] for more details.
- [3/16] HPT 1.0 Air is out, our most efficient model as a cost-effective solution that is capable of solving a wide range of vision-and-language tasks. HPT 1.0 Air is publicly available and achieves state-of-the-art results among all the open-source multimodal LLM models of similar or smaller sizes on the challenging MMMU benchmark. Please read our [technical blog post] and [HuggingFace Repository] for more details.
- Overview of Model Achitecture
- Quick Start
- Evaluations
- Benchmarks
- Pretrained Models Used
- Disclaimer and Responsible Use
- Contact Us
- License
- Acknowledgements
pip install -r requirements.txt
pip install -e .
You can download the model weights from HF into your [Local Path] and set the global_model_path
as your [Local Path] in the model config file:
git lfs install
git clone https://huggingface.co/HyperGAI/HPT [Local Path]
or directly set global_model_path
as the HF repo-id ('HyperGAI/HPT').
You can also set other strategies in the config file that are different from our default settings.
After setting up the config file, launch the model demo for a quick trial:
python demo/demo.py --image_path [Image] --text [Text] --model [Config]
Example:
python demo/demo.py --image_path demo/einstein.jpg --text 'Question: What is unusual about this image?\nAnswer:' --model hpt-air-demo
You can design different prompts here to boost the question.
Launch the model for evaluation:
torchrun --nproc-per-node=8 run.py --data [Dataset] --model [Config]
Example for HPT 1.5 Air:
torchrun --nproc-per-node=8 run.py --data MMMU_DEV_VAL --model hpt-air-1-5
For HPT 1.5 Air
HPT 1.5 Air
-
Pretrained LLM: Llama3-8B-Instruct
-
Pretrained Visual Encoder: siglip-so400m-patch14-384
HPT 1.0 Air
-
Pretrained LLM: Yi-6B-Chat
-
Pretrained Visual Encoder: clip-vit-large-patch14-336
Note that the HPT Air is a quick open release of our models to facilitate the open, responsible AI research and community development. It does not have any moderation mechanism and provides no guarantees on their results. We hope to engage with the community to make the model finely respect guardrails to allow practical adoptions in real-world applications requiring moderated outputs.
- Contact: HPT@hypergai.com
- Follow us on Twitter.
- Follow us on Linkedin.
- Visit our website to learn more about us.
This project is released under the Apache 2.0 license. Parts of this project contain code and models from other sources, which are subject to their respective licenses and you need to apply their respective license if you want to use for commercial purposes.
The evaluation code for running this demo was extended based on the VLMEvalKit project. We also thank OpenAI for open-sourcing their visual encoder models and 01.AI and Meta for open-sourcing their large language models.