Data loading and preparation slow and repeated for each iteration in hyperparameters evolution

[Witek-]

Search before asking

• I have searched the YOLOv5 issues and discussions and found no similar questions.

Question

I am running hyperparameters evolution exactly as stated in the documentation:

python train.py --epochs 10 --data coco128.yaml --weights yolov5s.pt --cache --evolve

and it is slow in the initial phase of each iteration. Below is the output with the key fragment. I marked two places that take exceptionally long time to run. Other operations run fast. Even the very training is fast when compared to these two time-consuming, over 1 minute-long parts.

Transferred 342/349 items from yolov5s.pt
AMP: checks passed
optimizer: SGD(lr=0.01154) with parameter groups 57 weight(decay=0.0), 60 weight(decay=0.00035), 60 bias
train: Scanning C:\Users\Witek\datasets\coco128\labels\train2017.cache... 126 images, 2 backgrounds, 0 corrupt: 100%|██████████| 128/128 [00:00<?, ?it/s]
train: Caching images (0.1GB ram): 100%|██████████| 128/128 [00:00<00:00, 1999.78it/s]

25 seconds pass here

val: Scanning C:\Users\Witek\datasets\coco128\labels\train2017.cache... 126 images, 2 backgrounds, 0 corrupt: 100%|██████████| 128/128 [00:00<?, ?it/s]

37 seconds pass here

AutoAnchor: 0.07 anchors/target, 0.024 Best Possible Recall (BPR). Anchors are a poor fit to dataset , attempting to improve...

Why is it so slow? Why scanning and casching speed is 00:00<?, ?it/s?
Am I doing something wrong? It's just 128 images and it takes over a minute to load and prepare them!
Am I wrong to assume that the dataset is loaded from scratch and scaled and augmented before each iteration of hyperparameter evolution? Can't it be loaded and preprocessed just once and than used for all iterations?

My CPU is AMD Ryzen 5 3600, 32 GB RAM, GPU is 3060Ti/8GB RAM. The dataset is automatically downloaded to my SSD drive. No other haevy processes are running on the computer.

Additional

No response

[github-actions]

👋 Hello @Witek-, thank you for your interest in YOLOv5 🚀! Please visit our ⭐️ Tutorials to get started, where you can find quickstart guides for simple tasks like Custom Data Training all the way to advanced concepts like Hyperparameter Evolution.

If this is a 🐛 Bug Report, please provide a minimum reproducible example to help us debug it.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset image examples and training logs, and verify you are following our Tips for Best Training Results.

Requirements

Python>=3.7.0 with all requirements.txt installed including PyTorch>=1.7. To get started:

git clone https://github.com/ultralytics/yolov5  # clone
cd yolov5
pip install -r requirements.txt  # install

Environments

YOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all YOLOv5 GitHub Actions Continuous Integration (CI) tests are currently passing. CI tests verify correct operation of YOLOv5 training, validation, inference, export and benchmarks on MacOS, Windows, and Ubuntu every 24 hours and on every commit.

Introducing YOLOv8 🚀

We're excited to announce the launch of our latest state-of-the-art (SOTA) object detection model for 2023 - YOLOv8 🚀!

Designed to be fast, accurate, and easy to use, YOLOv8 is an ideal choice for a wide range of object detection, image segmentation and image classification tasks. With YOLOv8, you'll be able to quickly and accurately detect objects in real-time, streamline your workflows, and achieve new levels of accuracy in your projects.

Check out our YOLOv8 Docs for details and get started with:

pip install ultralytics

[glenn-jocher]

@Witek- the slow scanning and caching can be caused by a combination of factors such as HDD speed, CPU processing power and image augmentations. However, it is strange that even the small dataset takes such a long time to load. One possible reason could be that during the initial iteration, Hyperparameter Evolution is actually adding augmentations, hence the extra loading time.

To speed up the process, you can try turning off augmentations and disabling caching by adding --noautoanchor --nosave flags to your command. Additionally, you can try to resize your images to a smaller dimension or use a higher-end CPU or GPU to improve performance.

Regarding your question about loading and pre-processing the dataset, YOLOv5 does preprocess images once and caches them for subsequent epochs, but this is not applicable to the Hyperparameter Evolution process as it is starting each iteration at the beginning. Hope this helps!

[Witek-]

@glenn-jocher Thank you. I ran the evolution on 3 different machines with very similar results. During data loading and preprocessing neither the CPU nor the SSD are working heavily. The CPU is at 15-25% and the SSD is at 1% (no wonder, it's just 128 images!). The memory, though, is being used heavily.
The flags you suggested, unfortunately, did not help :(
Can you suggest an easy way to load and pre-process the dataset ONCE and use the same pre-processed dataset in consecutive trainings without repeating this time consuming part each time I train the net?

[glenn-jocher]

@Witek-, if the flags I suggested did not help, one possible approach to preprocess the dataset only once and then use the same preprocessed data in consecutive trainings without repeating this time-consuming part could be to save the preprocessed images and labels as an HDF5 file.

You can use OpenCV or PIL library to preprocess your images and adjust their size, aspect ratio, brightness, and contrast. Then, you can save these preprocessed images as an HDF5 file, along with their corresponding labels. When you start training the model, you can simply read the HDF5 file and feed the images and labels provided to the code.

Another approach could be to preprocess the data on a remote server and then transfer the preprocessed data to your local machine. This would alleviate the burden on your local machine and enable you to train your models more quickly.

Hope this helps!

[Witek-]

It looks like it's kinda slow with this extremaly small set. With larger sets it looks like it works normally.

[glenn-jocher]

@Witek- thank you for bringing your issue to our attention. The slow scanning and caching speed with this small dataset could be related to a combination of factors such as hardware performance, dataset size, or image augmentations.

As you mentioned, with larger datasets, the operation speed is normal, indicating a possible dependency on the dataset size. Therefore, it's possible that the performance issue is related to a time inefficiency when processing small datasets.

Please note that YOLOv5 is an object detection algorithm that typically performs well on large datasets with many object classes. Training on small datasets may result in overfitting or reduced accuracy. We recommend using a larger dataset for better results if possible.

If you need further assistance, please don't hesitate to ask!