Skip to content

zanwenok/bigcode-dataset

 
 

Repository files navigation

BigCode Dataset

This repository gathers all the code used to build the BigCode datasets such as The Stack as well as the preprocessing necessary used for model training.

Contents

  • language_selection: notebooks and file with language to file extensions mapping used to build the Stack v1.1.
  • pii: code for running PII detection and anonymization on code datasets.
  • preprocessing: code for filtering code datasets based on:
    • line length and percentage of alphanumeric characters.
    • number of stars.
    • comments to code ratio.
    • tokenizer fertility
  • decontamination: script to remove files that match test-samples from code generation benchmarks.

About

No description, website, or topics provided.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Jupyter Notebook 84.4%
  • Python 15.6%