docker run -it --rm --shm-size=2g --gpus all gardner/gsd:latest --server https://q.llm.nzGSD (GPU Swarm for Datasets) is a FOSS initiative designed to democratize the improvement of datasets for broader R&D. This project brings together volunteers who contribute their GPU time to enhance text datasets, making them more useful for the community at large. Inspired by the paper "Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling" by Maini et al. (January 29, 2024) paper, GSD is a community approach to processing and refining data efficiently and at scale.
GSD is similar to a decentralized version of Facebook's cc_net.
Participants (You!) contribute by running a Docker container, which connects to a work queue to fetch work packets, processes the text locally using vLLM, and submits the results. The output is collated and batched before being published to HuggingFace.
- Docker 🐳
- A stable internet connection 🌐
- A GPU that can run Mistral 7B (~12GB of VRAM) We are accepting pull requests for CPU workers!
- A commitment to Getting Shit Done ✅
To join the swarm, simply run the docker container:
docker run -it --rm --gpus all gardner/gsd:latest --server https://q.llm.nz- Clone the Repository:
git clone https://github.com/gardner/gsd.git- Run the Docker Container: In the repository's directory, execute:
docker-compose up --buildA huge thank you to every volunteer and to the pioneers who inspired GSD, especially the authors of the influential work by Maini et al. Your contributions drive this project forward.
GSD is under the GNU General Public License v3.0 - see the LICENSE file for details.