Best practice for keeping a YOLO training dataset with my code on GitHub?

[Zephyron-afk]

🏷️ Discussion Type

Question

Body

Hey, I'm a solo dev working on a small YOLO object detection project. Right now my images and labels are just sitting in the repo next to the code, and it's getting big (a few GB).

What's the best practice here?

• Use Git LFS for the images?
• Keep the dataset out of GitHub and use something like DVC, or just a download script?
• Or only commit a small sample set and keep the rest somewhere else?

Also, should trained weights (.pt files) go in the repo, or in Releases?

Just want to set it up properly before it gets messy. Thanks!

[onetwothird]

[Zephyron-afk]

This is pretty much the approach I'd recommend too. Keeping a small sample in the repo and hosting the full dataset externally is a good balance, especially for a solo project. DVC can always be added later when dataset versioning becomes necessary.

[kaluarachchiteshan]

Hi,

Short answer: keep the code in Git, keep the big data out of normal Git history, and put trained weights in Releases or a model hub. Your options, in order of how I'd pick them:

1. Dataset: DVC or a plain download script (my recommendation for a few GB)

• DVC versions the dataset with small .dvc pointer files in Git, while the actual images live in remote storage (S3, Google Drive, Azure, an SSH server, etc.). You get reproducible "this code version = this dataset version" without bloating the repo.

pip install dvc
dvc init
dvc add data/images data/labels
git add data/images.dvc data/labels.dvc .gitignore
git commit -m "Track dataset with DVC"
dvc remote add -d storage <your-remote-url>
dvc push

• If DVC feels like too much, a scripts/download_data.py that fetches a zip from cloud storage or a Hugging Face dataset repo, plus a README line, is perfectly fine for a solo project.
• Hosting the dataset on Hugging Face Datasets or Kaggle is also common for vision data, and it gives you a public, versioned download link.

2. Git LFS: works, but watch the limits

• GitHub's free LFS quota is small (about 10 GiB storage and 10 GiB bandwidth per month at the time I last checked, so verify the current numbers), and every clone or CI run uses bandwidth. A few GB of images can burn through it quickly.
• LFS is a good fit only if the dataset is modest and you want everything in one place.

git lfs install
git lfs track "*.jpg" "*.png"
git add .gitattributes

• Note that LFS must be set up before the files are committed. Files already in history stay in normal Git objects unless you migrate them (git lfs migrate import --include="*.jpg,*.png" --everything), which rewrites history.

3. Commit only a small sample set

Yes, do this regardless of which option you choose. Keep 10 to 50 images and labels in the repo (for example data/sample/) so that tests, CI and new contributors can run the pipeline without downloading everything.

4. Trained weights (.pt)

• Don't commit them to the repo, since every retrained version adds a new full-size copy to history.
• Attach them to GitHub Releases (each file can be up to 2 GiB), or host them on Hugging Face Hub, with the version and training config in the release notes.
• If you use DVC, you can track the weights there too.

5. Fix what you already have

Since the images are already in the repo, you'll want to remove them from history, not only delete them:

pip install git-filter-repo
git filter-repo --path data/ --invert-paths

Back up the repo first. This rewrites history and needs a force-push, which is fine for a solo project. Then add a .gitignore:

data/
runs/
*.pt
*.onnx

(Adjust paths to your layout, and keep data/sample/ out of the ignore rule with !data/sample/.)

Suggested setup

• main repo: code, configs (data.yaml, hyperparameters), sample data, README with a "get the data" step
• Dataset: DVC remote or Hugging Face/Kaggle
• Weights: Releases or Hugging Face Hub
• Also record the dataset version and training command in each release so experiments are reproducible.