BetterTransformer not supporting CodeGen2

[ganler]

System Info

Name: optimum
Version: 1.8.4
Summary: Optimum Library is an extension of the Hugging Face Transformers library, providing a framework to integrate third-party libraries from Hardware Partners and interface with their specific functionality.
Home-page: https://github.com/huggingface/optimum
Author: HuggingFace Inc. Special Ops Team
Author-email: hardware@huggingface.co
License: Apache
Location: /home/jiawei/.conda/envs/g/lib/python3.8/site-packages
Requires: coloredlogs, datasets, huggingface-hub, numpy, packaging, sympy, torch, torchvision, transformers
Required-by: 
Ubuntu 20.04; Python 3.8; 4xA6000 GPUs

Who can help?

@fx

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

from transformers import AutoTokenizer, AutoModelForCausalLM
from optimum.bettertransformer import BetterTransformer

tokenizer = AutoTokenizer.from_pretrained("Salesforce/codegen2-16B")
model = AutoModelForCausalLM.from_pretrained(
    "Salesforce/codegen2-16B", trust_remote_code=True, revision="main"
)
model = BetterTransformer.transform(model, keep_original_model=True)

text = "def hello_world():"
input_ids = tokenizer(text, return_tensors="pt").input_ids
generated_ids = model.generate(input_ids, max_length=512)
print(tokenizer.decode(generated_ids[0], skip_special_tokens=True))

Expected behavior

It should work as this is basically the codegen architecture which is stated to be supported in https://huggingface.co/docs/optimum/v1.8.3/bettertransformer/overview

However, it will fail by throwing an exception saying None has no attribute of ...

[IlyasMoutawwakil]

it seems like the problem is in the modeling for CodeGen2 which is different than the original CodeGen (for some reason CodeGenAttention donn't have embed_positions). I will try to investigate why.
For now you can use one of the supported codegen- models (without the 2).

[IlyasMoutawwakil]

The modeling of CodeGen2 here is not part of the official transformers library (hence the trust_remote_code) and different from that of CodeGen.

[ganler]

I see. Thanks for the input. I am curious if it will be easy to support the next generation of CodeGen as it supports infilling. Thanks!

[IlyasMoutawwakil]

@ganler It should be easy since all the necessary code is already in there. But it also depends on the authors' choice. This discussion gives more details about adding native support huggingface/transformers#22290.

[michaelfeil]

I got a similar error:

optimum-cli export onnx --model=Salesforce/codegen2-1B --trust-remote-code onnx/

returns

│ .py:919 in from_pretrained                                                                       │
│                                                                                                  │
│   916 │   │   config_dict, unused_kwargs = PretrainedConfig.get_config_dict(pretrained_model_n   │
│   917 │   │   if "auto_map" in config_dict and "AutoConfig" in config_dict["auto_map"]:          │
│   918 │   │   │   if not trust_remote_code:                                                      │
│ ❱ 919 │   │   │   │   raise ValueError(                                                          │
│   920 │   │   │   │   │   f"Loading {pretrained_model_name_or_path} requires you to execute th   │
│   921 │   │   │   │   │   " repo on your local machine. Make sure you have read the code there   │
│   922 │   │   │   │   │   " set the option `trust_remote_code=True` to remove this error."       │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: Loading Salesforce/codegen2-1B requires you to execute the configuration file in that repo on your local machine. Make sure
you have read the code there to avoid malicious use, then set the option `trust_remote_code=True` to remove this error.```

[fxmarty]

@michaelfeil This command with --trust-remote-code works fine for me with optimum main and transformers 4.29.2.