TypeError when testing DOBF

[SusanTan]

Hi,

I am new to this project and just playing around with your pertained models. I am able to obfuscate and deobfuscate a one liner code with commenting out https://github.com/facebookresearch/CodeGen/blob/main/codegen_sources/preprocessing/obfuscation/bobskater_obfuscator.py#L427-L436. I wonder if I am missing some flag that's not included in the help manual to turn off this code.

command: python -m codegen_sources.model.deobfuscate --lang python --model_path ~/xcellent-ml/trained_models/dobf.pth --beam_size 1 < ~/xcellent-ml/dataset_python/doall.py
The error message is:
adding to path /home/zujunt/xcellent-ml/CodeGen
INFO - 12/02/21 15:36:29 - 0:00:02 - ============ Model Reloading
INFO - 12/02/21 15:36:29 - 0:00:02 - Reloading encoder from /home/zujunt/xcellent-ml/trained_models/dobf.pth ...
WARNING - 12/02/21 15:36:33 - 0:00:06 - Lang java_dictionary matched to pretrained java_dictionary lang embedding.
WARNING - 12/02/21 15:36:33 - 0:00:06 - Lang java_obfuscated matched to pretrained java_obfuscated lang embedding.
WARNING - 12/02/21 15:36:33 - 0:00:06 - Lang python_dictionary matched to pretrained python_dictionary lang embedding.
WARNING - 12/02/21 15:36:33 - 0:00:06 - Lang python_obfuscated matched to pretrained python_obfuscated lang embedding.
INFO - 12/02/21 15:36:35 - 0:00:07 - Reloading decoders from /home/zujunt/xcellent-ml/trained_models/dobf.pth ...
WARNING - 12/02/21 15:36:35 - 0:00:08 - Lang java_dictionary matched to pretrained java_dictionary lang embedding.
WARNING - 12/02/21 15:36:35 - 0:00:08 - Lang java_obfuscated matched to pretrained java_obfuscated lang embedding.
WARNING - 12/02/21 15:36:35 - 0:00:08 - Lang python_dictionary matched to pretrained python_dictionary lang embedding.
WARNING - 12/02/21 15:36:35 - 0:00:08 - Lang python_obfuscated matched to pretrained python_obfuscated lang embedding.
INFO - 12/02/21 15:36:37 - 0:00:10 - Number of parameters (encoder): 125677911
INFO - 12/02/21 15:36:37 - 0:00:10 - Number of parameters (decoders): 97334103
INFO - 12/02/21 15:36:37 - 0:00:10 - Number of decoders: 1

INFO - 12/02/21 15:36:37 - 0:00:10 - Roberta BPE mode use Roberta pretrained codes and vocab /home/zujunt/xcellent-ml/CodeGen/data/bpe/roberta-base-vocab.
Original Code:
fruits = ["apple", "banana", "cherry"]
Traceback (most recent call last):
File "/home/zujunt/python/python-3.7.0/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/home/zujunt/python/python-3.7.0/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/zujunt/xcellent-ml/CodeGen/codegen_sources/model/deobfuscate.py", line 235, in
input, lang=params.lang, beam_size=params.beam_size,
File "/home/zujunt/xcellent-ml/CodeGen/codegen_sources/model/deobfuscate.py", line 140, in deobfuscate
input = obfuscator(input)[0]
File "/home/zujunt/xcellent-ml/CodeGen/codegen_sources/preprocessing/lang_processors/python_processor.py", line 195, in obfuscate_code
res, dico = obfuscateString(code, obfuscateNames=True, removeDocstrings=False)
File "/home/zujunt/xcellent-ml/CodeGen/codegen_sources/preprocessing/obfuscation/bobskater_obfuscator.py", line 457, in obfuscateString
sAst = transformer.visit(sAst)
File "/home/zujunt/python/python-3.7.0/lib/python3.7/ast.py", line 262, in visit
return visitor(node)
File "/home/zujunt/xcellent-ml/CodeGen/codegen_sources/preprocessing/obfuscation/bobskater_obfuscator.py", line 441, in generic_visit
super().generic_visit(node)
File "/home/zujunt/python/python-3.7.0/lib/python3.7/ast.py", line 317, in generic_visit
value = self.visit(value)
File "/home/zujunt/python/python-3.7.0/lib/python3.7/ast.py", line 262, in visit
return visitor(node)
File "/home/zujunt/xcellent-ml/CodeGen/codegen_sources/preprocessing/obfuscation/bobskater_obfuscator.py", line 441, in generic_visit
super().generic_visit(node)
File "/home/zujunt/python/python-3.7.0/lib/python3.7/ast.py", line 317, in generic_visit
value = self.visit(value)
File "/home/zujunt/python/python-3.7.0/lib/python3.7/ast.py", line 262, in visit
return visitor(node)
File "/home/zujunt/xcellent-ml/CodeGen/codegen_sources/preprocessing/obfuscation/bobskater_obfuscator.py", line 435, in generic_visit
+ "]"
TypeError: can only concatenate str (not "NoneType") to str

After commenting out the above mentioned code, I think I got the correct output:

Thank you for your help!

[baptisteroziere]

Hi,
This kind of debug messages should probably be deactivated by default, especially if they can cause bugs. I will do that and also cast the error message to str to avoid this error when running with this debug message on.
Thank you for noticing that and for telling us!