How to fine tune Codegen?

[smith-co]

I would like to fine tune the Codegen model. Can you provide any documentation in this regard?

[shmuelhizmi]

+1

[enijkamp]

The converted PyTorch models can be fine-tuned similarly to other causal LMs in HuggingFace.

See tutorials like http://reyfarhan.com/posts/easy-gpt2-finetuning-huggingface/.

[TheodoreGalanos]

Would you be releasing training code for the original models? Would be nice to try some on v3s (if possible).

[thisisanshgupta]

I think this script might help in finetuning:

https://colab.research.google.com/drive/13dZVYEOMhXhkXWfvSMVM1TTtUDrT6Aeh?usp=sharing#scrollTo=vCPohrZ-CTWu

[enijkamp]

@TheodoreGalanos Working on a release for the JAX coding. I trained the models on TPU-v4 and have to resolve a blocker for v3.

[smith-co]

@enijkamp @thisisanshgupta I am checking the link you have shared.

Still I think it would greatly help everyone if it is possible to provide fine tuning steps in the repo. 🙏

[Ontopic]

I for one would appreciate any code/directions needed to run things on a TPU-v4. Great work all!

[enijkamp]

@thisisanshgupta @Ontopic Yes, I'm working on the release of my training library for TPU-v3/v4 and will keep you posted.

[tlkh]

Hello @enijkamp thank you for your work. Looking forward to some fine-tuning instructions and code.

Currently, I have tried to fine-tune as if it is GPT-2, but I am running into issues where the model's quality degrades significantly.

Is there any particular way the data has to be structured for fine-tuning? Currently, I am just concatenating together the prompts and code as follows:

def xyz():
    """abc"""
    code()

def xyz():
    """abc"""
    code()

[enijkamp]

@smith-co @thisisanshgupta @tlkh

For torch, I wrote up a minimal example in deepspeed, which can train the 16B on a ~24 GB gpu. You would need to sanity test this, optimize the configuration, plug in the data loader, and save the weights to disk:
https://github.com/salesforce/CodeGen/blob/main/jaxformer/hf/train_deepspeed.py

For jax, the training library in is undergoing sanity checks on TPU-v3 and should be released soon.

[enijkamp]

@smith-co @thisisanshgupta @tlkh @Ontopic @TheodoreGalanos @shmuelhizmi A first release of the training code for TPU-v3/v4 is here:

https://github.com/salesforce/jaxformer

[zhangybuaa]

@enijkamp I want to fine-tune the model with my own code data, how should I build the dataset. Are there any requirements for the format of the dataset, whether the data needs to be labeled and what format should it be labeled in. Can some guidance or examples be given, thanks！

[calix]

@smith-co @thisisanshgupta @tlkh

For torch, I wrote up a minimal example in deepspeed, which can train the 16B on a ~24 GB gpu. You would need to sanity test this, optimize the configuration, plug in the data loader, and save the weights to disk: https://github.com/salesforce/CodeGen/blob/main/jaxformer/hf/train_deepspeed.py

For jax, the training library in is undergoing sanity checks on TPU-v3 and should be released soon.

Besides the VRAM, how much RAM would be required to train the model?

[glicerico]

@enijkamp , or anyone who has used jaxformer to fine-tune on TPU-v4, what is the approximate cost?

[enijkamp]

@glicerico Roughly speaking, cost is a function of the size of the model and data. How much data do you have? Which model do you want to fine-tune?

[glicerico]

@enijkamp , trying to reproduce the work by Shin and Van Durme, who used a few hundred (sentence, parse) pairs to fine tune codex for semantic parsing. I would like to do this with CodeGen. Seeing your results, I would probably want to fine tune the 16GB model.

[srnsrn120]

@enijkamp : I want to finetune mono model , Can you please share dataset format for python and details steps or notebook .

[watreyoung]

@glicerico Roughly speaking, cost is a function of the size of the model and data. How much data do you have? Which model do you want to fine-tune?

Is there any more easier code script template withouth deep-speed to fine-tune CodeGen(350M)?
Plus: Is the data format same as other pre-trained model like CodeT5 or CodeBERT?
Looking forward to the reply.