Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds a new reward model, ArmoRM, to RewardBench.
Description: Arbitrary-Rating Multi-Objective Reward Model (ArmoRM) with Mixture-of-Experts (MoE) Aggregation of Reward Objectives
Authors (* indicates equal contribution)
Haoxiang Wang*, Wei Xiong*, Tengyang Xie, Han Zhao, Tong Zhang
Blog: To appear soon (with implementation details)
Tech Report: To be released in June 2024
Model: ArmoRM-Llama3-8B-v0.1
Architecture
RewardBench LeaderBoard
Demo Code
Modified Files
rewardbench/models/
__init__.py
: add config for ArmoRMarmorm.py
: add ArmoRM pipelinescripts/
configs/eval_config.yaml
: eval config for ArmoRMrun_rm.py
:torch_dtype
. Our ArmoRM is native totorch.bfloat16
(certainlytorch.float32
also works, but it takes larger GPU memory), and the Int-8 quantization leads to a very slow inference of ArmoRM (even slower than using FP32). This new config choice allows evaluation of ArmoRM undertorch.bfloat16
.Evaluation Commands