Why GPT-CC is lower than CodeX significantly.

[BitcoinNLPer]

This is Codex results

The following result is GPT-CC

• Is it caused by the quality of the pre-trained corpus data?

Thanks

[Symbolk]

Also curious, as I know, the HumanEval dataset contains 164 problems, according to the latest result in README, the model does not even pass any one of them!

[taisazero]

Ya that's correct we discovered issues with the pre-training corpus data. We fixed the issue and released a new pertaining corpus and are in the process of processing the dataset further and pre-training a new GPT-CC.

In the meantime, check out these awesome models: https://huggingface.co/spaces/codeparrot/code-generation-models