[RFC] Ease-of-use Task Register for Autotune API

[xin3he]

Ease-of-use Task Register for Autotune API

Principles

• Only require setting the target task name for autotune evaluation.
• Easy to extend for customized task and welcome contribution to INC.

Requirement

• Provides task name when registering.
• Requires model as input and float as output.

@register_task(name="text-generation")
def eval_func(model, model_name=None):
    eval_dataset = init_dataset("xxx")
    accuracy = Accuracy()
    for data, label in eval_dataset:
        output = model(data)
        accuracy.update(output, label)
    return accuracy

Repo Architecture

• torch
  • tasks # make sure all algos here have user interface.
    • text_generation # lm_eval, lm_code_eval
    • image_classification # cifar, imagenet
    • ... # welcome for contribution

Examples

lm_eval

@register_task(name="lm_eval")
def eval_func(model, model_name, tasks=["lambada_openai"]):
    from intel_extension_for_transformers.llm.evaluation.lm_eval import evaluate
    results = evaluate(
        model="hf-causal",
        model_args="pretrained=" + model_name + ",tokenizer=" + model_name + ",dtype=float32",
        user_model=model,
        batch_size=32,
        tasks=tasks,
    )
    return results["accuracy"]

lm_code_eval

@register_task(name="lm_code_eval")
def eval_func(model, model_name, tasks=None):
    from intel_extension_for_transformers.llm.evaluation.lm_code_eval import evaluate
    from transformers import AutoTokenizer
    tokenizer = AutoTokenizer.from_pretrained(model_name)
    results = evaluate(
        model=user_model,
        tokenizer=tokenizer,
        tasks=",".join(tasks),
        batch_size=args.batch_size,
    )
    return results["accuracy"]

Usage

model = autotune(
    model,
    conf,
    example_inputs, # example_inputs for jit.trace
    run_fn, # calibration function
    task="lm_eval", # registered evaluation task
    task_args={
        "model_name": "facebook/opt-125m",
        "tasks": ["lambada_openai", "hellaswag", "winogrande", "piqa", "wikitext"],
    }
)

[thuang6]

One general comments: Given current pile calibration dataset, some Chinese evaluation tasks like CEval/CMMLU highly likely perform poor. If we consider to support different types of evaluation (Chinese, Math, Code), we need to have a new better (most likely mixed) calibration dataset.

[xin3he]

I see, I'm also considering whether letting the run_fn(calibration) re-use the task name register.

[xin3he]

I also consider adding task_name, like lm_eval_small by limit the sample numbers, or args, like limit=100, for a quick autotune.

[changwangss]

May I know what's the purpose of autotune ?, if you want to fallback op1, op2 to fp32 dtype one by one, I don't suggest use lm-eval as evaluation func, because lm-eval is usually used by LLM models, calibration and evaluation also need much time, if you want to tune many times, it will OOM.

[xin3he]

Do you mean the memory will accumulate when tuning each time?

[changwangss]

yes，(torchscript mode)jit model for ipex int8 will be released until the program is finished running.

[xin3he]

Yes, it's a known issue of IPEX int8, and we can research to resolve it by only converting the model (w/o jit.trace) for accuracy validation.
From my perspective, this RFC is focusing on the ease-of-use of autotune, not the autotune shortages. The issue you mentioned can be discussed later. Thanks for raising that.

[xin3he]

Decision:

• Not to implement

Reasons:

1. For CV tasks, the imagenet dataset is not available for auto downloading, so it's not easy to use. (from Yi)
2. For LLM tasks, LLM customized eval_func should be defined in ITREX, INC keeps current API. (from Feng)