Preprocess step is completing but pth files are not generating as expected in XLM folder

[Prathameshwar]

Hello,
After running preprocess steps from below command:

python -m codegen_sources.preprocessing.preprocess /path/data/mydata2 --langs java cpp --mode monolingual_functions --bpe_mode=fast --local=True --train_splits=1 --fastbpe_code_path=/path/data/bpe/cpp-java-python/ --fastbpe_vocab_path=/path/data/bpe/cpp-java-python/

XLM-syml folder is getting generated with file name like:
test.cpp_cl.pth
test.cpp_sa.pth
test.java_cl.pth
....
train.cpp_cl.0.pth
train.cpp_sa.0.pth
...

But when using these folder/files in Training step(MLM) is giving error like:

XLM-syml/train.java.pth not found
XLM-syml/valid.java.pth not found
XLM-syml/test.java.pth not found
XLM-syml/train.cpp.pth not found
XLM-syml/valid.cpp.pth not found
XLM-syml/test.cpp.pth not found

Train command:
python train.py --exp_name mlm --dump_path '/path/CodeGen/data/models' --data_path '/path/data/mydata2/XLM-syml' --split_data_accross_gpu local --mlm_steps 'java,cpp' --add_eof_to_stream true --word_mask_keep_rand '0.8,0.1,0.1' --word_pred '0.15' --encoder_only true --n_layers 12 --emb_dim 768 --n_heads 12 --lgs 'java-cpp' --max_vocab 64000 --gelu_activation true --roberta_mode false --amp 2 --fp16 true --batch_size 8 --bptt 512 --epoch_size 1000 --max_epoch 2000 --split_data_accross_gpu global --optimizer 'adam_inverse_sqrt,warmup_updates=10000,lr=0.0001,weight_decay=0.01' --save_periodic 0 --validation_metrics _valid_mlm_ppl --stopping_criterion '_valid_mlm_ppl,10'

I think files are getting generated with suffix like _cl.pth or _sa.pth which is not being considered in training step? OR I am doing something wrong?

Thanks

[skye95git]

Have you successfully run the pre-processing steps？ When I run:

python -m codegen_sources.preprocessing.preprocess /data2/linjiayi/CodeGen-master/data --langs python --mode monolingual_functions --bpe_mode=fast --local=True --train_splits=1

There is an error: there is no json in {str(self.folder)}.
But I have downloaded the source code from Google BigQuery and put them into self.folder.

Do you know what went wrong?
I suspect I had a problem saving the file name when downloading data from Google BigQuery.
When I export the data to Google storage, I set:

1. export format: JSON
2. compression: GZIP
3. filename: python-*.json

Is there something wrong with my operation?
Files downloaded from Google Storage cannot be opened directly. For example, the file name is python_python-000000000000.json. I just open it and it's garbled. I need to decompress it first. But the file type of the extracted file is no longer JSON.

The contents of the file also don't seem to be in standard JSON format. Is that what you downloaded? Can I preprocess this data directly?

[lhendre]

I am having issues as well, while I am not positive on your issue and we have different, I had a lot more luck with roberta bpe then fast bpe. that being said, while everything generated I am also running into issues with training

[skye95git]

I am having issues as well, while I am not positive on your issue and we have different, I had a lot more luck with roberta bpe then fast bpe. that being said, while everything generated I am also running into issues with training

Thank you. What is the suffix of the json you download from Google BigQuery? Is it .json.gz? Can you share a sample of your JSON format?

[lhendre]

You definitely need/want the GZ compressed file archive format. I am not trying the google BigQuery portion yet, I am using the test data they provide right now just to insure it works before I try anything else

[skye95git]

You definitely need/want the GZ compressed file archive format. I am not trying the google BigQuery portion yet, I am using the test data they provide right now just to insure it works before I try anything else

Thank you very much! I didn't notice the test data in the data\test_dataset folder. I will try it.

[lhendre]

No problem, let me know how that goes. Im just trying to do that to verify everything else works currently.

[skye95git]

No problem, let me know how that goes. Im just trying to do that to verify everything else works currently.

I have run successfully run the pre-processing steps on the the google BigQuery data. I only changed the filename python-000000000000.json.gz to python-000.json.gz. But when I run the pre-processing steps on the test data in the data\test_dataset folder, there is an error: AssertionError: failed to apply bpe on /home/linjiayi/CodeGen-master/data/test_dataset/cpp.train.sa.0.tok. I don't know why.

I have another question. Whether CodeGen's preprocessing steps can extract comments? I find there is a parameter --keep_comments in TransCoder. But CodeGen doesn't have.

[lhendre]

Yeah cant help you there, I didnt have much luck with BPE, switched to Roberta

[malachaux]

Hello,
After running preprocess steps from below command:

python -m codegen_sources.preprocessing.preprocess /path/data/mydata2 --langs java cpp --mode monolingual_functions --bpe_mode=fast --local=True --train_splits=1 --fastbpe_code_path=/path/data/bpe/cpp-java-python/ --fastbpe_vocab_path=/path/data/bpe/cpp-java-python/

XLM-syml folder is getting generated with file name like:
test.cpp_cl.pth
test.cpp_sa.pth
test.java_cl.pth
....
train.cpp_cl.0.pth
train.cpp_sa.0.pth
...

But when using these folder/files in Training step(MLM) is giving error like:

XLM-syml/train.java.pth not found
XLM-syml/valid.java.pth not found
XLM-syml/test.java.pth not found
XLM-syml/train.cpp.pth not found
XLM-syml/valid.cpp.pth not found
XLM-syml/test.cpp.pth not found

Train command:
python train.py --exp_name mlm --dump_path '/path/CodeGen/data/models' --data_path '/path/data/mydata2/XLM-syml' --split_data_accross_gpu local --mlm_steps 'java,cpp' --add_eof_to_stream true --word_mask_keep_rand '0.8,0.1,0.1' --word_pred '0.15' --encoder_only true --n_layers 12 --emb_dim 768 --n_heads 12 --lgs 'java-cpp' --max_vocab 64000 --gelu_activation true --roberta_mode false --amp 2 --fp16 true --batch_size 8 --bptt 512 --epoch_size 1000 --max_epoch 2000 --split_data_accross_gpu global --optimizer 'adam_inverse_sqrt,warmup_updates=10000,lr=0.0001,weight_decay=0.01' --save_periodic 0 --validation_metrics _valid_mlm_ppl --stopping_criterion '_valid_mlm_ppl,10'

I think files are getting generated with suffix like _cl.pth or _sa.pth which is not being considered in training step? OR I am doing something wrong?

Thanks

You are training on java cpp languages - so monolingual data - and you run the pipeline to get monolingual functions. you should run the pipeline with monolingual mode if you want to train something on java-cpp. If you want to train on functions then you languages and step in training command should be on java_sa cpp_sa for instance

[dowenk]

Have you successfully run the pre-processing steps？ When I run:

python -m codegen_sources.preprocessing.preprocess /data2/linjiayi/CodeGen-master/data --langs python --mode monolingual_functions --bpe_mode=fast --local=True --train_splits=1

There is an error: there is no json in {str(self.folder)}.
But I have downloaded the source code from Google BigQuery and put them into self.folder.

Do you know what went wrong?
I suspect I had a problem saving the file name when downloading data from Google BigQuery.
When I export the data to Google storage, I set:

1. export format: JSON

2. compression: GZIP

3. filename: python-*.json

Is there something wrong with my operation?
Files downloaded from Google Storage cannot be opened directly. For example, the file name is python_python-000000000000.json. I just open it and it's garbled. I need to decompress it first. But the file type of the extracted file is no longer JSON.

The contents of the file also don't seem to be in standard JSON format. Is that what you downloaded? Can I preprocess this data directly?

Your JSON file looks correct. Each line in the file is a JSON object representing python source code with key/value pairs "repo_name", "ref", "path" and "contents". The files should remain gzipped for processing.

[skye95git]

Have you successfully run the pre-processing steps？ When I run:
python -m codegen_sources.preprocessing.preprocess /data2/linjiayi/CodeGen-master/data --langs python --mode monolingual_functions --bpe_mode=fast --local=True --train_splits=1
There is an error: there is no json in {str(self.folder)}.
But I have downloaded the source code from Google BigQuery and put them into self.folder.
Do you know what went wrong?
I suspect I had a problem saving the file name when downloading data from Google BigQuery.
When I export the data to Google storage, I set:

1. export format: JSON

2. compression: GZIP

3. filename: python-*.json

Is there something wrong with my operation?
Files downloaded from Google Storage cannot be opened directly. For example, the file name is python_python-000000000000.json. I just open it and it's garbled. I need to decompress it first. But the file type of the extracted file is no longer JSON.

The contents of the file also don't seem to be in standard JSON format. Is that what you downloaded? Can I preprocess this data directly?

Your JSON file looks correct. Each line in the file is a JSON object representing python source code with key/value pairs "repo_name", "ref", "path" and "contents". The files should remain gzipped for processing.

Thanks for your reply. I have successfully run the pre-processing steps to extract functions.

[baptisteroziere]

Closing this issue as you managed to run the pipelines.
@skye95git you can still use the --keep_comments parameter like in TransCoder's pipeline.

Don't hesitate to create a new issue if you or @lhendre still have problems running the pipeline with fastbpe. With your parameters (not giving any pre-trained bpe codes and vocabs to the pipeline), it should train a new bpe model and use it to break the tokens. If you want to use our models on your datasets, you should use the BPE tokens and vocabs available at this link (look at the ReadME for more details). https://github.com/facebookresearch/CodeGen/tree/master/data/bpe/cpp-java-python