Update fraction arg to employ a random selection

[johnnynunez]

better if you have datasets unified

🛠️ PR Summary

_{Made with ❤️ by Ultralytics Actions}

🌟 Summary

Improved image selection for diverse dataset sampling 📸✨

📊 Key Changes

• Introduced randomness in selecting a subset of image files for datasets when not using the full dataset.

🎯 Purpose & Impact

• Purpose: This change aims to make dataset samples more diverse by randomly selecting images instead of taking the first portion of the dataset. This can help in training models to be more robust and perform better across varied data.
• Impact: Users employing a fraction of the dataset for model training or evaluation will now get a more diverse and possibly more representative sample of the whole dataset. This could lead to improvements in model accuracy and generalization.

[codecov]

Codecov Report

Attention: 2 lines in your changes are missing coverage. Please review.

Comparison is base (d8b8b71) 75.75% compared to head (ece9c1f) 75.70%.

Files	Patch %	Lines
ultralytics/data/base.py	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8234      +/-   ##
==========================================
- Coverage   75.75%   75.70%   -0.05%     
==========================================
  Files         117      117              
  Lines       14673    14674       +1     
==========================================
- Hits        11115    11109       -6     
- Misses       3558     3565       +7     

Flag	Coverage Δ
Benchmarks	36.28% <0.00%> (-0.01%)	⬇️
GPU	39.04% <0.00%> (-0.01%)	⬇️
Tests	70.68% <0.00%> (-0.17%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[johnnynunez]

#8233

[Burhan-Q]

@johnnynunez I believe I understand the change you're proposing. I suspect that the difference is slightly nuanced in what happens now vs what you're proposing. Your proposed change would randomize the selections for the train/val split. The existing code uses the shuffle argument in the Dataloader but I'm not certain that this would accomplish the same result, since the randomization is within the Dataloader which is probably specific to either train or val set (I'm not 100% certain).

I suspect that in most cases, people who care enough to want randomized train and val sampling, will probably split the data before passing in for training. Anyone else using fraction either doesn't care about randomizing or has a large enough dataset that splitting it this way wouldn't matter all that much.

Out of curiosity, have you tested to measure how much process time would be added using random.sample to split the dataset? I suspect this would be one of the most important considerations when evaluating this change.

[glenn-jocher]

@johnnynunez yes agree with @Burhan-Q here, the function is working as intended, it's simply using the first X percent of the dataset rather than sampling it randomly.

I suppose the only conceivable drawback is that the images in this first percent of the dataset can be replaced by changing the seed. @Burhan-Q what do you think? With the default settings, i.e. same seed the randomization is almost the same as the slice op.

My main usage for this was for training on the first i.e. 1% or 10% of huge datasets like OIV7 and Objects365 while developing and debugging.

[Burhan-Q]

@glenn-jocher I agree that this good as-is, especially given the argument description in the docs does not mention anything about randomly splitting the data:

Specifies the fraction of the dataset to use for training. Allows for training on a subset of the full dataset, useful for experiments or when resources are limited.

There's also this

ultralytics/ultralytics/data/utils.py

Lines 615 to 647 in 2f22ff4

	def autosplit(path=DATASETS_DIR / "coco8/images", weights=(0.9, 0.1, 0.0), annotated_only=False):
	"""
	Automatically split a dataset into train/val/test splits and save the resulting splits into autosplit_*.txt files.
	Args:
	path (Path, optional): Path to images directory. Defaults to DATASETS_DIR / 'coco8/images'.
	weights (list | tuple, optional): Train, validation, and test split fractions. Defaults to (0.9, 0.1, 0.0).
	annotated_only (bool, optional): If True, only images with an associated txt file are used. Defaults to False.
	Example:
	```python
	from ultralytics.data.utils import autosplit
	autosplit()
	```
	"""
	path = Path(path) # images dir
	files = sorted(x for x in path.rglob("*.*") if x.suffix[1:].lower() in IMG_FORMATS) # image files only
	n = len(files) # number of files
	random.seed(0) # for reproducibility
	indices = random.choices([0, 1, 2], weights=weights, k=n) # assign each image to a split
	txt = ["autosplit_train.txt", "autosplit_val.txt", "autosplit_test.txt"] # 3 txt files
	for x in txt:
	if (path.parent / x).exists():
	(path.parent / x).unlink() # remove existing
	LOGGER.info(f"Autosplitting images from {path}" + ", using *.txt labeled images only" * annotated_only)
	for i, img in TQDM(zip(indices, files), total=n):
	if not annotated_only or Path(img2label_paths([str(img)])[0]).exists(): # check label
	with open(path.parent / txt[i], "a") as f:
	f.write(f"./{img.relative_to(path.parent).as_posix()}" + "\n") # add image to txt file

which I commented about in the referenced issue, which can be used for randomly sampling/splitting the dataset. Maybe it could be worth while to reference this function in the docs table in the fraction argument description for anyone who may want something more randomized.

[johnnynunez]

For example in my case I have put together 4 datasets and converted them into coco format. As I am doing experiments to try to improve the architecture of Yolov8 I make 50 epochs but then I saw the slice so I could be eliminating 3/4 datasets, and this is a serious mistake. With my selection I may be randomly picking items from all dataset.

[glenn-jocher]

@johnnynunez hi there! It sounds like you're doing some interesting work combining multiple datasets for your YOLOv8 experiments. If you're concerned about the dataset slicing potentially leaving out some of your data, you might want to consider shuffling your combined dataset before training. This way, you can ensure that your training and validation sets are randomly sampled from across all your datasets.

Here's a quick example of how you could shuffle your dataset in Python:

from ultralytics.data.utils import autosplit

autosplit("path/to/dataset", weights=(0.9, 0.1, 0.0))

This way, you're randomly selecting from all your datasets for both training and validation. Hope this helps, and best of luck with your experiments! 🚀

[glenn-jocher]

@johnnynunez I've decided to merge this in as I believe it makes sense. Thank you for your contributions to YOLO!