Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Simplify entry script #204

Merged
merged 2 commits into from
Aug 25, 2023
Merged

[Feature] Simplify entry script #204

merged 2 commits into from
Aug 25, 2023

Conversation

gaotongxiao
Copy link
Collaborator

@gaotongxiao gaotongxiao commented Aug 15, 2023

Motivation

Introduce two ways to launch an evaluation task, trying to get rid of the configuration file in some simple tasks.

Apart from running from a config, this PR enables users to launch tasks in two additional ways:

  1. Running evaluation from configs provided in configs/models and configs/datasets:

    python run.py --models hf_llama_7b --datasets mmlu_gen ceval_gen
  2. Without a model configuration file, users can specify the specs in CLI (HF models only)

     python run.py --hf-path huggyllama/llama-7b \
     --tokenizer-kwargs padding_side="left" truncation="left" use_fast="False" \
      --max-out-len 100 \
      --max-seq-len 2048 \
     --batch-size 8 \
     --model-kwargs device_map="auto" \
     --no-batch-padding \
     --num-gpus=2 \
     --datasets ceval_gen mmlu_gen

The documentation will be updated in another PR.

Modifications

  1. Implement the aforementioned feature in run.py
  2. Move helper functions in run.py to utils/run.py, letting run.py focus on the logic and argument parsing only
  3. Add list_configs.py, which allows fuzzy config search in configs/models and configs/datasets
  4. Enhance HuggingFace wrapper such that it automatically decide the best model loader between AutoModelCausalLM and AutoModel
  5. Delete GLM-related datasets which distract the configuration matching system

@gaotongxiao gaotongxiao changed the title [Feature] Simply entry script [Feature] Simplify entry script Aug 15, 2023
run.py Show resolved Hide resolved
@gaotongxiao gaotongxiao marked this pull request as ready for review August 21, 2023 08:11
@Leymore Leymore merged commit 9058be0 into open-compass:main Aug 25, 2023
1 check passed
go-with-me000 pushed a commit to go-with-me000/opencompass that referenced this pull request Oct 9, 2023
* [Feature] Simply entry script

* update
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants