Creating a `cache-dataset` for Video classification.

[yakhyo]

Hello, now I am trying to test the video classification model R(2+1)D on Kinetics400. However the speed of loading data is so slow. I believe the loading speed can be improved by caching the data but I am not sure how to cache video files. In the code also, it is mentioned. I want to know to cache video files? is cache dataset creating feature also included in future updates?
Thank you !

cc @datumbox

[datumbox]

@yakhyo Thanks for the issue.

You can produce a cache by passing the --cache-dataset. It's a boolean flag that will tell the script to create or read from the cache if available. The cache is automatically produced for you the first time you parse the dataset, so it does require to wait through the process once.

A few tips to speed it up:

• The Kinetics class supports a num_workers set to 1 by default. You can change it here and here and that's going to boost it significantly.
• First time you create a cache, it's worth running on a machine without GPU (or a single GPU) to avoid producing concurrently the same cache across the processes and using all the available CPUs/threads to chunk down your dataset.

One thing to keep in mind while dealing with caches is that if you modify some of the parameters of the dataset, you might need to invalidate (delete) the cache. We have a PR that should soon fix this at #6234 but if you face any weird issues while switch models that require different framerates etc then remember to drop the cache and retry.

I'm closing the issue as I believe I answered your question but if I didn't please feel free to reopen.