Fine-tuning TransCoder

[wasiahmad]

Hi,

We recently proposed a small-scale program translation dataset, called AVATAR. We want to fine-tune TransCoder on the translation task but we didn't find any documentation on that. Can you provide some guidelines on fine-tuning TransCoder?

[baptisteroziere]

Hi,
Since you have a parallel dataset, you probably want to finetune using the MT objective.
If you have two languages lang1 and lang2, you need to add this to the arguments when training

--mt_steps 'lang1-lang2,lang2-lang1`

For the dataset, you need .pth files with this format (assuming the string lang1 is < lang2 in lexicographical order): train.lang1-lang2.lang1.pth and train.lang1-lang2.lang2.pth or files with .i.pth if you are training on several GPUs. You can use the preprocessing pipeline to create the dataset and just add the right symbolic links in the end.

[wasiahmad]

• If I understand correctly, I need to follow the training steps as mentioned here. To confirm, the .pth files should contain BPE processed dataset (not just the tokenized dataset)? Also, do we need to use _sa suffix with the language ID (e.g., python_sa, java_sa) ?

• Furthermore, I see you have a concept called dataset mode. So, do I need to use Monolingual Functions (ex: Java functions) mode to individually convert the raw dataset into BPE codes?

• Can you also confirm, both TransCoder and DOBF models use fast_bpe, not roberta_bpe?

[baptisteroziere]

• Yes the .pth should contain the binarized BPE processed dataset.
• For the pipeline, the monolingual functions mode will also break your data into functions and I'm not sure that would work for your usecase. It seems like you got your parallel dataset from competitive programming websites so even though the files are parallel I guess the functions in the files will not necessarily be parallel.
  You can try the monolingual mode but if you encounter errors for some of your examples you may need to modify it a bit to ensure that you don't skip lines (without outputting anything in the .tok file) otherwise you may have mismatching elements in your python and java files...
  Since your dataset is small and you may already have done a train-test split, you probably want to generate the train/valid/test tokenized files yourself and run the pipeline after commenting these lines https://github.com/facebookresearch/CodeGen/blob/master/codegen_sources/preprocessing/preprocess.py#L86-L93. You could also just import and run the functions to apply a BPE model and binarize the data in a notebook or a small script.
  Don't forget to use the BPE model that we provide if you want to reload our model to finetune it.
• The TransCoder model uses fast_bpe. The DOBF models we link to in DOBF's ReadME use the roberta_bpe and the architecture of RoBERTa in order to make it comparable to CodeBERT and graphCodeBERT on downstream tasks. However we also trained a model with a fast bpe and the same architecture as TransCoder which I could also link to at the end of this ReadME.

PS: since your dataset contains only python and java code, you will probably get better results using the model we pre-trained using DOBF and which we linked in TransCoder's ReadME: translator_transcoder_size_from_DOBF
It uses the same architecture and BPE as TransCoder.

[wasiahmad]

Awesome! We used your TransCoder-DOBF model in our experiments.

We extracted functions from the programs we collected from G4G and only considered programs having 1 standalone function. We fine-tuned the baseline models and now want to do the same with TransCoder. So far, we have been considering TransCoder without fine-tuning. You can see the comparison between TransCoder and other models here.

Although not related to this discussion thread, I want to take your input on one thing. It may sound a bit weird to you.

We used the previous repo of TransCoder and got higher Computational Accuracy on the G4G dataset proposed in the TransCoder paper. But when I switched to the new codebase which is this repo, the performance dropped significantly. We used beam_size=5 (top 1) in our experiment. Did you notice any performance differences in TransCoder with your new codebase? I do not see any difference in tokenization or BPE encoding (not sure though). However, I saw many changes were made to include DOBF in the source code.

Here are the performances (beam_size=5 (top 1)):

TransCoder (using old repo) - Java to Python: 49.6% ; Python to Java: 35.1%
TransCoder (using this repo) - Java to Python: 41.2% ; Python to Java: 28.8%

Any help/pointer would be appreciated.

[baptisteroziere]

Ok cool. I assumed you were using the model from the first version of TransCoder since DOBF wasn't cited in your paper on arxiv but I guess you will update it at some point.

About your issue with the scores using the old repo vs this repo, it definitely shouldn't be happening, you should get almost exactly the same scores (except for the timeouts) with the old and the new repo if you are evaluating with the same model. The tokenization should be almost the same and the BPE encodings exactly the same, however the binarization should be different since we reserve 500 tokens extra tokens for the identifier tokens (VAR_i, FUNC_i, CLASS_i) in the new version.
A few things to check:

• check that you are reloading the encoder and the decoder correctly in the logs (search for ============ Model Reloading).
• check the detailed computational accuracy results. Especially that the total number of examples and number of script_not_found is the same on both sides for the same metric and that the number of timeouts didn't blow up (that could be due to external factors on your server)
• which command are you using to run the evaluation on each repo? Are they the same?
• You may want to run the evaluation with beam_size=1 on both sides and compare them to the table of the paper corresponding to your model. Which model did you use exactly ?

[wasiahmad]

Yes, we didn't include the updated result in the archived paper. Previously, we were using model_1.pth and model_2.pth and now we are using the models shared here.

I am using the translate.py file to perform the translation, therefore I guess there should be no issue regarding binarization. I actually made some adjustments in the code to serve my need and it was working fine till I was using model implementations from the old repo. After using the model implementation from the new repo, the performance drops. I checked for potential bugs but couldn't find any.

I checked identical_gold, failure, total_evaluated and they are roughly the same. The major difference is in error and in success. I did some analysis too based on error types. For example, in Java to Python translation:

[THIS REPO]

{   
    "AttributeError": 11,
    "IndexError": 18,
    "KeyError": 13,
    "NameError": 16,
    "RecursionError": 3,
    "SyntaxError": 36,
    "TypeError": 63,
    "UnboundLocalError": 10,
    "ZeroDivisionError": 1,
    "other": 11,
    "total": 182
}

[OLD REPO]

{   
    "AttributeError": 12,
    "IndexError": 18,
    "KeyError": 10,
    "NameError": 18,
    "RecursionError": 1,
    "SyntaxError": 11,
    "TypeError": 51,
    "UnboundLocalError": 9,
    "ZeroDivisionError": 0,
    "other": 12,
    "total": 142
}

As you can see, SyntaxError and TypeError increased with the use of code from this repo. You can find how we categorized the errors here.

Let me run the evaluation with beam_size=1 and see how the model performs.

[wasiahmad]

The following results are using the code from this repository.

Results with beam_size=1:

Java to Python: 40.3% [comparing to 41.2% with beam_size=5 (top 1)]
Python to Java: 27.2% [comparing to 28.8% with beam_size=5 (top 1)]

The numbers reported in the TransCoder paper are:

[beam_size=1]
Java to Python: 35.0%
Python to Java: 24.7%

[beam_size=10 (top 1)]
Java to Python: 49.0%
Python to Java: 36.6%

Seems like we are getting comparable performance using beam_size=1, but I guess beam decoding is not helping when I am using the code from this repository.

[baptisteroziere]

I ran the evaluation again on a clean repository and I'm getting higher computation accuracies than you on TransCoder's test set. With these models I'm getting higher scores than in the TransCoder paper with beam_size 1 and slightly lower scores with beam size 10. After writing the TransCoder paper, we solved a few detokenization bugs in the code so that could explain why the current version is better with beam size 1 but I don't know why the scores are lower with the beam size 10 (top 1) metric. I will need to investigate it further, I'm suspecting that our new tokenizer (based on tree-sitter instead of javalang) leads to slightly different tokenization in a few cases.

It is also surprising that your scores are lower than mine using the new repository. I looked at the code of translate.py and didn't find any bug, but there may be some differences in the way you perform the evaluation if you're not using our methods to do it. Using the train.py script, which calls evaluator.py would also be more efficient. We were thinking of translate.py as a small tool for demos and didn't optimize it as much as the rest (for instance it processes the data on the fly and doesn't batch the calls to generate translations). The fact that it works quite well, just not as well as it should, makes me think that there is a small issue with the preprocessing of the data. Maybe check that there is a token at the beginning and at the end of each input sequence and that the sequences look correct (e.g. not too many tokens).

PS: I ran it without the --retry_mismatching_types argument to make it faster while the results in the paper were obtained with this argument turned on. It uses a few heuristics to cast the input types to the right types and avoid type errors in cases when they are easy to avoid and can increase the score a little bit but it couldn't explain the differences we're seeing here.

beam size = 1:
Java to Python (TransCoder_model_1): 44.49 %
Python to Java (TransCoder_model_2): 32.64%

beam size = 10 (top 1)
Java to Python (TransCoder_model_1): 46.44 %
Python to Java (TransCoder_model_2): 36.0%

[wasiahmad]

Thanks a lot for doing the evaluation on your end. Since I need to fine-tune TransCoder, I fully migrate your codebase to perform the experiments, hopefully, I will get the results you mentioned.

I am particularly clueless because I can reproduce most of your results reported in the TransCoder paper with your old codebase (using translate.py). When I moved to the new codebase, I carefully adopted the code portion I needed. As I was able to successfully run experiments, I thought everything was ok. But with the new code, I have always observed degraded performances, and also with beam decoding performances didn't improve much.

[baptisteroziere]

Yes sorry about that, we tried to make the migration easier than that.
To be honest, I don't know why you got such a low score using translate.py and I wouldn't have expected you to get a lower score when using the new repository and the new version of the model.
Could you give me an example of input function that fails with the new codebase and succeeded with the previous codebase and with the output that failed ? I would try to reproduce the error and to understand what's happening.

[wasiahmad]

I adapted the whole source code of the new repo now and still getting the same performance. I guess we are operating on different input files. To confirm, I am using the transcoder_test.java.tok and transcoder_test.python.tok files from transcoder_test_set.zip.

While I used translate.py, I do not use any tokenization, only use BPE apply and then perform translation. Save it in a file and then perform the evaluation. Do you think that is okay to use the preprocessed tokenized files provided in this repo?

[wasiahmad]

@Brozi I have successfully started fine-tuning TransCoder. However, I want to confirm with you.

I have the bpe processed and binarized data at data_path directory. The directory includes the following files.

test.java-python.bpe.java    test.java-python.python.pth    train.java-python.java.pth    valid.java-python.bpe.python
test.java-python.bpe.python  train.java-python.bpe.java     train.java-python.python.pth  valid.java-python.java.pth
test.java-python.java.pth    train.java-python.bpe.python   valid.java-python.bpe.java    valid.java-python.python.pth

I am using the following command for fine-tuning.

python $TRAIN_SCRIPT \
--exp_name transcoder-mt \
--dump_path $SAVE_DIR \
--data_path $path_2_data/transcoder-bin \
--split_data_accross_gpu local \
--max_len 512 \
--mt_steps 'java-python,python-java' \
--encoder_only False \
--n_layers 0 \
--n_layers_encoder 6  \
--n_layers_decoder 6 \
--emb_dim 1024 \
--n_heads 16 \
--lgs 'java-python' \
--max_vocab 64000 \
--gelu_activation true \
--roberta_mode false \
--reload_model "$MODEL_PATH," \
--lgs_mapping 'java:java_sa,python:python_sa'  \
--amp 2 \
--fp16 true  \
--batch_size 4 \
--accumulate_gradients 4 \
--epoch_size 2000 \
--max_epoch 20 \
--optimizer 'adam_inverse_sqrt,warmup_updates=200,lr=0.0001,weight_decay=0.01' \
--eval_bleu true \
--validation_metrics 'valid_python-java_mt_bleu' \
2>&1 | tee ${SAVE_DIR}/finetune.log;

I am using the translator_transcoder_size_from_DOBF as the reload_model. I saw the number of parameters in encoder and decoder are:

INFO - 08/31/21 23:24:47 - 0:00:08 - Number of parameters (encoder): 143278592
INFO - 08/31/21 23:24:47 - 0:00:08 - Number of parameters (decoders): 168481280

[Note] I found some bugs in the current codebase.

1. In this line, we should use self.epoch_size = len(self.data).
2. It seems like warmup_updates cannot be 0 (it gives ZeroDivisionError at model/src/optim.py line 126).

[baptisteroziere]

I adapted the whole source code of the new repo now and still getting the same performance. I guess we are operating on different input files. To confirm, I am using the transcoder_test.java.tok and transcoder_test.python.tok files from transcoder_test_set.zip.

The new evaluation I ran was from the .pth files in this zip. I double checked and they correspond exactly to the .tok files.

While I used translate.py, I do not use any tokenization, only use BPE apply and then perform translation. Save it in a file and then perform the evaluation. Do you think that is okay to use the preprocessed tokenized files provided in this repo?

Yes you should be getting exactly the same input as me when I ran the evaluation again if you just apply a fastBPE with the codes in data/bpe/cpp-java-python/codes on each line in the transcoder_test.java.tok and transcoder_test.python.tok files.

The structure of your files seems fine if you don't want to split your data across GPUs (they will all read from the same file here).
I can see a few issues with your command:

• --reload_model "$MODEL_PATH," reloads only the encoder and initializes the decoder randomly. I think that for your usecase you'd rather use --reload_model "$MODEL_PATH,$MODEL_PATH" to reload both.
• TransCoder doesn't use gelu activations so you should probably remove --gelu_activation true or set it to false (default value).

Suggestions:

• TransCoder uses --n_heads 8 and you use --n_heads 16. The extra heads will just be initialized randomly and it shouldn't really be an issue, just be aware that there will be some warnings about the extra weights not being found in the checkpoint when you reload the model
• For machine translation, we generally use --tokens_per_batch to choose the batch size for each batch so that the total number of tokens in the batch is close to --tokens_per_batch and it would probably be more efficient for you to do the same. Try --tokens_per_batch 5000 for instance if you have 32GB GPUs.
• if you're evaluating on TransCoder's validation and test datasets, it would probably be nice to compute the computational accuracy by adding --eval_computation True
• max 20 epochs of size 2000 each seems quite short.

You should also check if there are any warnings when you reload the model (e.g. if the lang embeddings and weights are loaded correctly). Maybe search for "random" in the logs to check that nothing was initialized randomly.
Here is the number of parameters in transcoder, I guess the difference comes from the extra attention heads

INFO - 08/31/21 06:21:05 - 0:00:18 - Number of parameters (encoder): 143279616
INFO - 08/31/21 06:21:05 - 0:00:18 - Number of parameters (decoders): 168482304

[wasiahmad]

With n_heads=8, I get this (which does not match your number):

INFO - 09/01/21 11:20:06 - 0:00:15 - Number of parameters (encoder): 143278592
INFO - 09/01/21 11:20:06 - 0:00:15 - Number of parameters (decoders): 168481280

Can you clarify what is the values of the following flags for TransCoder and translator_transcoder_size_from_DOBF?

--n_layers_encoder ?  \
--n_layers_decoder ? \
--emb_dim ? \
--n_heads ? \

[baptisteroziere]

We probably have the same parameters. I was just comparing to a model running on C++ as well so the extra parameters were the lang embeddings for C++. I checked and we have the same number of parameters when I run transcoder on only two languages.

--n_layers_encoder 6  \
--n_layers_decoder 6 \
--emb_dim 1024 \
--n_heads 8 \