UCF101: Dataloader Fail on assertion

[YanCote]

🐛 Bug

When loading UCF101 with different value of frames_per_clip and step_between_clips, it often yield frames_per_clip + 1 images which results in the following assertion to fail:

assert len(video) == self.num_frames, "{} x {}".format(
    video.shape, self.num_frames

Code To Reproduce

import torch
from torchvision.datasets import UCF101

ucf_loc = "/dataset/ucf"
ucf_data_dir = f"{ucf_loc}/UCF101/UCF-101"
ucf_label_dir = f"{ucf_loc}/ucfTrainTestlist"
frames_per_clip = 5
step_between_clips = 1
num_workers = 4

def custom_collate(batch):
    filtered_batch = []
    for video, _, label in batch:
        filtered_batch.append((video, label))
    return torch.utils.data.dataloader.default_collate(filtered_batch)

if __name__ == '__main__':
    # create train loader (allowing batches and other extras)
    test_dataset = UCF101(ucf_data_dir, ucf_label_dir, frames_per_clip=frames_per_clip,
                           step_between_clips=step_between_clips, train=False, transform=None, num_workers=num_workers)
    test_loader = torch.utils.data.DataLoader(test_dataset, batch_size=8, shuffle=True,
                                               collate_fn=custom_collate,num_workers=num_workers)

    for i, (video, label) in enumerate(test_loader):
        print(video.size())
        print(label)

Steps to reproduce the behavior:

1. Download UCF101
2. Modify ucf_loc and run the code included above

stack trace

Original Traceback (most recent call last):
File "/opt/miniconda3/envs/p38_ucf/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
data = fetcher.fetch(index)
File "/opt/miniconda3/envs/p38_ucf/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/opt/miniconda3/envs/p38_ucf/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/opt/miniconda3/envs/p38_ucf/lib/python3.8/site-packages/torchvision/datasets/ucf101.py", line 102, in getitem
video, audio, info, video_idx = self.video_clips.get_clip(idx)
File "/opt/miniconda3/envs/p38_ucf/lib/python3.8/site-packages/torchvision/datasets/video_utils.py", line 382, in get_clip
assert len(video) == self.num_frames, "{} x {}".format(
AssertionError: torch.Size([6, 240, 320, 3]) x 5

Expected behavior

The return tensor data should be composed of a total of frames_per_clip images consistently no matter the properties of the input video and the parameter values provided to UCF101 class.

Environment

• PyTorch =1.9.0 and TorchVision=0.10.0.
• OS (e.g., Linux): MacOs BigSur
• How you installed PyTorch / torchvision (conda, pip, source): conda
• Build command you used (if compiling from source): NA
• Python version: 3.8.10
• CUDA/cuDNN version: NA
• GPU models and configuration: CPU
• Any other relevant information:

Additional context

Preliminary Investigation:

My guess is that with the provided argument, _read_from_stream function return +/- 1 frames in video.py: read_video(). I did not dig deeper to understand why

[fmassa]

This is probably related to #3791, as now when using sec to index into the video, there are rounding errors which leads to the error.

cc @prabhat00155 @bjuncek I've brought this potential problem during our call a few weeks ago, we should fix it

[YanCote]

Thank you for following up.
A quick question, Is there a possible workaround, not sure how to modify torchvision/io/video.py:_read_from_stream() to address this issue.

[bjuncek]

@YanCote there is an easy workaround - I've submitted the PR and it's readily usable.
I think I might wanna change the defaults and update other datasets just in case, but this should patch up the issue for the time being

[xianyuanliu]

Hi guys, I have the same problem loading ucf101 with frames_per_clip=16. May I know if it is resolved now, please?

[bjuncek]

Yup, it's now resolved on the latest main.

[shehan360]

@bjuncek How can I get these changes to my environment? By installing the nightly build?

ex -

pip3 install --pre torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/nightly/cpu

[bjuncek]

Hi @shehan360

Nightly build should work provided that you have pyav installed as well.
Otherwise, I'd suggest building from source (and having pyav and ffmpeg installed if you want the full video-reader support.

[xiaoguan1206]

How can I solve this problem? PyTorch =1.10.1 and TorchVision= 0.11.2 .