Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Please help add SeaLLM-7B-v2 with 7.54 on MT-bench on the leaderboard #3013

Open
nxphi47 opened this issue Feb 5, 2024 · 1 comment
Open

Comments

@nxphi47
Copy link

nxphi47 commented Feb 5, 2024

Hi Lmsys team,

We released SeaLLM-7B-v2 last week - a multilingual model that achieves 7.54 on the English MT-bench.

Can you please check and verify the results and add our model to the leaderboard!

Generation and GPT-4 ratings files:

https://huggingface.co/SeaLLMs/SeaLLM-7B-v2/tree/main/evaluation/mt_bench

Please find the files in the link above.

Steps to reproduce the results:

  1. Add the format to conversations.py
# Seallm
register_conv_template(
    Conversation(
        name="seallm",
        system_template="""<|im_start|>system
{system_message}""",
        system_message="""You are a helpful, intelligent and safe assistant.""",
        roles=("<|im_start|>user", "<|im_start|>assistant"),
        sep_style=SeparatorStyle.CHATML,
        sep="</s>",
        stop_token_ids=[0, 1],
    )
)
  1. Add the following to model_adapter.py
class SeaLLMAdapter(BaseModelAdapter):
    """The model adapter for SeaLLMAdapter"""

    use_fast_tokenizer = False

    def match(self, model_path: str):
        return "sea" in model_path.lower()

    def get_default_conv_template(self, model_path: str) -> Conversation:
        return get_conv_template("seallm")
@nxphi47
Copy link
Author

nxphi47 commented Feb 5, 2024

@merrymercy @infwinston Please help. Let me know if you have any questions.

if you can add it to the Chatbot Arena, that is even better. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant