add subtsring decontamination #16
Conversation
There was a problem hiding this comment.
Great thank you for adding this! I just left a comment on a missing file from which you import humaneval reader. It would also be helpful if you can add a readme with what is needed to run the code.
@ChenghaoMou we can probably add instructions to execute your code too in that readme (forgot to mention it in your PR)
| from tqdm import tqdm | ||
| from multiprocessing import Pool | ||
|
|
||
| from human_eval.data import read_problems |
There was a problem hiding this comment.
I think the file human_eval is missing (btw we also have HumanEval on datasets for the future :) )
There was a problem hiding this comment.
Oh actually I directly installed https://github.com/openai/human-eval as a package. I could modify the code to use HF datasets instead yes.
| assert len(data) == 500 | ||
|
|
||
| # Checksum / version issues here | ||
| # dataset = load_dataset("mbpp", split="test") |
There was a problem hiding this comment.
I also noticed that, you will need to upgrade datasets version (if the error still happens there is the ignore_verifications=True argument)
Exact-substring match for decontamination #13
This removes 336 files from the
python-permissivedataset and 292 from thepython-permissive-dedupdataset. All these removals match HumanEval samples.