Skip to content
This repository was archived by the owner on Oct 25, 2024. It is now read-only.

Conversation

@changwangss
Copy link
Contributor

@changwangss changwangss commented Dec 18, 2023

Type of Change

Fix llm models extension test issue

  1. phi transformers version.
  2. add requirements for baichuan/qwen/codegen.

Support codegen sq workflow, due to codegen use the remote tokenizer, so add trust_remote_code to get the correct tokenizer. transformers version requests 4.35.2.
fp32 with limit 20

numactl -m 0 -C 0-55 python run_generation.py     --model Salesforce/codegen25-7b-multi    --accuracy     --batch_size 20     --n_samples 20     --allow_code_execution     --temperature 0.2     --do_sample     --tasks "humaneval" --trust_remote_code True
Evaluating generations...
{
  "humaneval": {
    "pass@1": 0.5725,
    "pass@10": 0.649981867977224
  },
  "config": {
    "model": "llama",
    "temperature": 0.2,
    "n_samples": 20
  }
}
{'humaneval': {'pass@1': 0.5725, 'pass@10': 0.649981867977224}, 'config': {'model': 'llama', 'temperature': 0.2, 'n_samples': 20}}

int8 sq with alpha 0,5 calib_iters 5 limit 20

python run_generation.py     --model Salesforce/codegen25-7b-multi        --alpha 0.5       --accuracy     --batch_size 20     --n_samples 20     --allow_code_execution     --temperature 0.2     --do_sample     --tasks "humaneval" --limit 20 --trust_remote_code True --sq --int8 --calib_iters 5
Evaluating generations...
{
  "humaneval": {
    "pass@1": 0.3575,
    "pass@10": 0.4497261252679209
  },
  "config": {
    "model": "llama",
    "temperature": 0.2,
    "n_samples": 20
  }
}
{'humaneval': {'pass@1': 0.3575, 'pass@10': 0.4497261252679209}, 'config': {'model': 'llama', 'temperature': 0.2, 'n_samples': 20}}

Description

detail description
JIRA ticket: xxx

Expected Behavior & Potential Risk

the expected behavior that triggered by this PR

How has this PR been tested?

how to reproduce the test (including hardware information)

Dependency Change?

any library dependency introduced or removed

Signed-off-by: changwangss <chang1.wang@intel.com>
@VincyZhang
Copy link
Contributor

@changwangss
Copy link
Contributor Author

@VincyZhang
Copy link
Contributor

VincyZhang commented Dec 19, 2023

Signed-off-by: changwangss <chang1.wang@intel.com>
Signed-off-by: changwangss <chang1.wang@intel.com>
Signed-off-by: changwangss <chang1.wang@intel.com>
Signed-off-by: changwangss <chang1.wang@intel.com>
Signed-off-by: changwangss <chang1.wang@intel.com>
Signed-off-by: changwangss <chang1.wang@intel.com>
Signed-off-by: changwangss <chang1.wang@intel.com>
Signed-off-by: changwangss <chang1.wang@intel.com>
Signed-off-by: Wang, Chang <chang1.wang@intel.com>
VincyZhang and others added 2 commits December 21, 2023 14:09
@changwangss
Copy link
Contributor Author

changwangss commented Dec 21, 2023

local test qwen

python -u ./run_generation.py --model Qwen/Qwen-7B --alpha 0.9 --output_dir saved_results_qwen --trust_remote_code True --int8 --benchmark --batch_size 1  --_commit_hash  f7bc352f27bb1c02ee371a4576942a7d96c8bb97 --calib_iter 10  --sq

image

Signed-off-by: VincyZhang <wenxin.zhang@intel.com>
@VincyZhang
Copy link
Contributor

CI all pass, ready for merge.

@VincyZhang VincyZhang merged commit 1967445 into main Dec 21, 2023
@VincyZhang VincyZhang deleted the wangchang/req branch December 21, 2023 10:38
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants