Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Evaluate: Qwen/Qwen1.5-XX #26

Open
3 tasks done
ggbetz opened this issue Mar 23, 2024 · 2 comments
Open
3 tasks done

Evaluate: Qwen/Qwen1.5-XX #26

ggbetz opened this issue Mar 23, 2024 · 2 comments
Labels
bug Something isn't working eval_request

Comments

@ggbetz
Copy link
Contributor

ggbetz commented Mar 23, 2024

For {XX} in [0.5B, 1.8B, 4B, 7B, 14B, 32B, 72B]:

Check:

  • The model has not been evaluated yet and doesn't show up on the CoT Leaderboard.
  • There is no evaluation request issue for the model in the repo.
  • The parameters below have been adapted and shall be used.

Parameters:

NEXT_MODEL_PATH=Qwen/Qwen1.5-{XX}
NEXT_MODEL_REVISION=main
NEXT_MODEL_PRECISION=bfloat16
MAX_LENGTH=2048 
GPU_MEMORY_UTILIZATION=0.7
VLLM_SWAP_SPACE=8
@ggbetz ggbetz changed the title Evaluate: Qwen/Qwen1.5-14B Evaluate: Qwen/Qwen1.5-XX Apr 10, 2024
@ggbetz ggbetz assigned yakazimir and unassigned yakazimir Apr 10, 2024
@ggbetz ggbetz added the bug Something isn't working label Apr 11, 2024
@ggbetz
Copy link
Contributor Author

ggbetz commented Apr 11, 2024

@ggbetz
Copy link
Contributor Author

ggbetz commented Apr 17, 2024

Might however be related to #48, as I've been testing the smallest base model only...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working eval_request
Projects
None yet
Development

No branches or pull requests

2 participants