ValueError: Caught ValueError in DataLoader worker process 0. #53

Raiseku · 2023-01-26T14:18:48Z

Search before asking

I have searched the Roboflow Notebooks issues and found no similar bug report.

Notebook name

https://colab.research.google.com/github/roboflow-ai/notebooks/blob/main/notebooks/train-yolov8-instance-segmentation-on-custom-dataset.ipynb

Bug


/content
Downloading https://github.com/ultralytics/assets/releases/download/v0.0.0/yolov8s-seg.pt to yolov8s-seg.pt...
100% 22.8M/22.8M [00:00<00:00, 150MB/s]

Ultralytics YOLOv8.0.11 🚀 Python-3.8.10 torch-1.13.1+cu116 CUDA:0 (Tesla T4, 15110MiB)
yolo/engine/trainer: task=segment, mode=train, model=yolov8s-seg.pt, data=/content/datasets/TicketDetection-3/data.yaml, epochs=15, patience=50, batch=16, imgsz=640, save=True, cache=False, device=, workers=8, project=None, name=None, exist_ok=False, pretrained=False, optimizer=SGD, verbose=False, seed=0, deterministic=True, single_cls=False, image_weights=False, rect=False, cos_lr=False, close_mosaic=10, resume=False, overlap_mask=True, mask_ratio=4, dropout=0.0, val=True, save_json=False, save_hybrid=False, conf=None, iou=0.7, max_det=300, half=False, dnn=False, plots=True, source=None, show=False, save_txt=False, save_conf=False, save_crop=False, hide_labels=False, hide_conf=False, vid_stride=1, line_thickness=3, visualize=False, augment=False, agnostic_nms=False, retina_masks=False, format=torchscript, keras=False, optimize=False, int8=False, dynamic=False, simplify=False, opset=17, workspace=4, nms=False, lr0=0.01, lrf=0.01, momentum=0.937, weight_decay=0.0005, warmup_epochs=3.0, warmup_momentum=0.8, warmup_bias_lr=0.1, box=7.5, cls=0.5, dfl=1.5, fl_gamma=0.0, label_smoothing=0.0, nbs=64, hsv_h=0.015, hsv_s=0.7, hsv_v=0.4, degrees=0.0, translate=0.1, scale=0.5, shear=0.0, perspective=0.0, flipud=0.0, fliplr=0.5, mosaic=1.0, mixup=0.0, copy_paste=0.0, cfg=None, hydra={'output_subdir': None, 'run': {'dir': '.'}}, v5loader=False, save_dir=runs/segment/train
Downloading https://ultralytics.com/assets/Arial.ttf to /root/.config/Ultralytics/Arial.ttf...
100% 755k/755k [00:00<00:00, 87.0MB/s]
Overriding model.yaml nc=80 with nc=1

                   from  n    params  module                                       arguments                     
  0                  -1  1       928  ultralytics.nn.modules.Conv                  [3, 32, 3, 2]                 
  1                  -1  1     18560  ultralytics.nn.modules.Conv                  [32, 64, 3, 2]                
  2                  -1  1     29056  ultralytics.nn.modules.C2f                   [64, 64, 1, True]             
  3                  -1  1     73984  ultralytics.nn.modules.Conv                  [64, 128, 3, 2]               
  4                  -1  2    197632  ultralytics.nn.modules.C2f                   [128, 128, 2, True]           
  5                  -1  1    295424  ultralytics.nn.modules.Conv                  [128, 256, 3, 2]              
  6                  -1  2    788480  ultralytics.nn.modules.C2f                   [256, 256, 2, True]           
  7                  -1  1   1180672  ultralytics.nn.modules.Conv                  [256, 512, 3, 2]              
  8                  -1  1   1838080  ultralytics.nn.modules.C2f                   [512, 512, 1, True]           
  9                  -1  1    656896  ultralytics.nn.modules.SPPF                  [512, 512, 5]                 
 10                  -1  1         0  torch.nn.modules.upsampling.Upsample         [None, 2, 'nearest']          
 11             [-1, 6]  1         0  ultralytics.nn.modules.Concat                [1]                           
 12                  -1  1    591360  ultralytics.nn.modules.C2f                   [768, 256, 1]                 
 13                  -1  1         0  torch.nn.modules.upsampling.Upsample         [None, 2, 'nearest']          
 14             [-1, 4]  1         0  ultralytics.nn.modules.Concat                [1]                           
 15                  -1  1    148224  ultralytics.nn.modules.C2f                   [384, 128, 1]                 
 16                  -1  1    147712  ultralytics.nn.modules.Conv                  [128, 128, 3, 2]              
 17            [-1, 12]  1         0  ultralytics.nn.modules.Concat                [1]                           
 18                  -1  1    493056  ultralytics.nn.modules.C2f                   [384, 256, 1]                 
 19                  -1  1    590336  ultralytics.nn.modules.Conv                  [256, 256, 3, 2]              
 20             [-1, 9]  1         0  ultralytics.nn.modules.Concat                [1]                           
 21                  -1  1   1969152  ultralytics.nn.modules.C2f                   [768, 512, 1]                 
 22        [15, 18, 21]  1   2770931  ultralytics.nn.modules.Segment               [1, 32, 128, [128, 256, 512]] 
YOLOv8s-seg summary: 261 layers, 11790483 parameters, 11790467 gradients, 42.7 GFLOPs

Transferred 411/417 items from pretrained weights
optimizer: SGD(lr=0.01) with parameter groups 66 weight(decay=0.0), 77 weight(decay=0.0005), 76 bias
train: Scanning /content/datasets/TicketDetection-3/train/labels... 150 images, 0 backgrounds, 0 corrupt: 100% 150/150 [00:00<00:00, 1433.15it/s]
train: New cache created: /content/datasets/TicketDetection-3/train/labels.cache
albumentations: Blur(p=0.01, blur_limit=(3, 7)), MedianBlur(p=0.01, blur_limit=(3, 7)), ToGray(p=0.01), CLAHE(p=0.01, clip_limit=(1, 4.0), tile_grid_size=(8, 8))
val: Scanning /content/datasets/TicketDetection-3/valid/labels... 14 images, 0 backgrounds, 0 corrupt: 100% 14/14 [00:00<00:00, 637.08it/s]
val: New cache created: /content/datasets/TicketDetection-3/valid/labels.cache
Image sizes 640 train, 640 val
Using 2 dataloader workers
Logging results to runs/segment/train
Starting training for 15 epochs...

      Epoch    GPU_mem   box_loss   seg_loss   cls_loss   dfl_loss  Instances       Size
  0% 0/10 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/usr/local/bin/yolo", line 8, in <module>
    sys.exit(entrypoint())
  File "/usr/local/lib/python3.8/dist-packages/ultralytics/yolo/cli.py", line 148, in entrypoint
    cli(cfg)
  File "/usr/local/lib/python3.8/dist-packages/ultralytics/yolo/cli.py", line 84, in cli
    func(cfg)
  File "/usr/local/lib/python3.8/dist-packages/hydra/main.py", line 79, in decorated_main
    return task_function(cfg_passthrough)
  File "/usr/local/lib/python3.8/dist-packages/ultralytics/yolo/v8/segment/train.py", line 153, in train
    model.train(**cfg)
  File "/usr/local/lib/python3.8/dist-packages/ultralytics/yolo/engine/model.py", line 203, in train
    self.trainer.train()
  File "/usr/local/lib/python3.8/dist-packages/ultralytics/yolo/engine/trainer.py", line 185, in train
    self._do_train(int(os.getenv("RANK", -1)), world_size)
  File "/usr/local/lib/python3.8/dist-packages/ultralytics/yolo/engine/trainer.py", line 285, in _do_train
    for i, batch in pbar:
  File "/usr/local/lib/python3.8/dist-packages/tqdm/std.py", line 1195, in __iter__
    for obj in iterable:
  File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/dataloader.py", line 628, in __next__
    data = self._next_data()
  File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/dataloader.py", line 1333, in _next_data
    return self._process_data(data)
  File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/dataloader.py", line 1359, in _process_data
    data.reraise()
  File "/usr/local/lib/python3.8/dist-packages/torch/_utils.py", line 543, in reraise
    raise exception
ValueError: Caught ValueError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/_utils/worker.py", line 302, in _worker_loop
    data = fetcher.fetch(index)
  File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/_utils/fetch.py", line 58, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/_utils/fetch.py", line 58, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.8/dist-packages/ultralytics/yolo/data/base.py", line 179, in __getitem__
    return self.transforms(self.get_label_info(index))
  File "/usr/local/lib/python3.8/dist-packages/ultralytics/yolo/data/augment.py", line 48, in __call__
    data = t(data)
  File "/usr/local/lib/python3.8/dist-packages/ultralytics/yolo/data/augment.py", line 48, in __call__
    data = t(data)
  File "/usr/local/lib/python3.8/dist-packages/ultralytics/yolo/data/augment.py", line 361, in __call__
    i = self.box_candidates(box1=instances.bboxes.T,
  File "/usr/local/lib/python3.8/dist-packages/ultralytics/yolo/data/augment.py", line 375, in box_candidates
    return (w2 > wh_thr) & (h2 > wh_thr) & (w2 * h2 / (w1 * h1 + eps) > area_thr) & (ar < ar_thr)  # candidates
ValueError: operands could not be broadcast together with shapes (3,) (4,)

Environment

Google Colab

Minimal Reproducible Example

!nvidia-smi
import os
HOME = os.getcwd()
print(HOME)
!pip install ultralytics==8.0.11

from IPython import display
display.clear_output()

import ultralytics
ultralytics.checks()
from ultralytics import YOLO

from IPython.display import display, Image
!mkdir {HOME}/datasets
%cd {HOME}/datasets
!pip install roboflow
display.clear_output()
from roboflow import Roboflow
rf = Roboflow(api_key="#####")
project = rf.workspace("muni").project("ticketdetection")
dataset = project.version(3).download("yolov5")
%cd {HOME}

!yolo task=segment mode=train model=yolov8s-seg.pt data={dataset.location}/data.yaml epochs=15 imgsz=640

Additional

The other important error i think is:
ValueError: operands could not be broadcast together with shapes (3,) (4,)

Can someone help me?

Are you willing to submit a PR?

Yes I'd like to help by submitting a PR!

The text was updated successfully, but these errors were encountered:

github-actions · 2023-01-26T14:19:29Z

👋 Hello @Raiseku, thank you for leaving an issue on Roboflow Notebooks.

🐞 Bug reports

If you are filing a bug report, please be as detailed as possible. This will help us more easily diagnose and resolve the problem you are facing. To learn more about contributing, check out our Contributing Guidelines.

If you require support with custom code that is not part of Roboflow Notebooks, please reach out on the Roboflow Forum or on the GitHub Discussions page associated with this repository.

💬 Get in touch

Do you have more questions about Roboflow that we haven't responded to yet? Feel free to ask them on the Roboflow Discuss forum. Our developer advocates and community team actively respond to questions there.

To ask questions about Notebooks, head over to the GitHub Discussions section of this repository.

SkalskiP · 2023-01-26T15:11:34Z

@Raiseku could you try changing:

!yolo task=segment mode=train model=yolov8s-seg.pt data={dataset.location}/data.yaml epochs=15 imgsz=640

to:

!yolo task=segment mode=train model=yolov8s-seg.pt data={dataset.location}/data.yaml epochs=15 imgsz=640 v5loader=True

?

Raiseku · 2023-01-26T15:41:07Z

Thanks for the answer!
i changed the code to:
!yolo task=segment mode=train model=yolov8s-seg.pt data={dataset.location}/data.yaml epochs=15 imgsz=640 v5loader=True

but now its giving me this error:

/content
Downloading https://github.com/ultralytics/assets/releases/download/v0.0.0/yolov8s-seg.pt to yolov8s-seg.pt...
100% 22.8M/22.8M [00:01<00:00, 17.2MB/s]

Ultralytics YOLOv8.0.11 🚀 Python-3.8.10 torch-1.13.1+cu116 CUDA:0 (Tesla T4, 15110MiB)
yolo/engine/trainer: task=segment, mode=train, model=yolov8s-seg.pt, data=/content/datasets/TicketDetection-3/data.yaml, epochs=15, patience=50, batch=16, imgsz=640, save=True, cache=False, device=, workers=8, project=None, name=None, exist_ok=False, pretrained=False, optimizer=SGD, verbose=False, seed=0, deterministic=True, single_cls=False, image_weights=False, rect=False, cos_lr=False, close_mosaic=10, resume=False, overlap_mask=True, mask_ratio=4, dropout=0.0, val=True, save_json=False, save_hybrid=False, conf=None, iou=0.7, max_det=300, half=False, dnn=False, plots=True, source=None, show=False, save_txt=False, save_conf=False, save_crop=False, hide_labels=False, hide_conf=False, vid_stride=1, line_thickness=3, visualize=False, augment=False, agnostic_nms=False, retina_masks=False, format=torchscript, keras=False, optimize=False, int8=False, dynamic=False, simplify=False, opset=17, workspace=4, nms=False, lr0=0.01, lrf=0.01, momentum=0.937, weight_decay=0.0005, warmup_epochs=3.0, warmup_momentum=0.8, warmup_bias_lr=0.1, box=7.5, cls=0.5, dfl=1.5, fl_gamma=0.0, label_smoothing=0.0, nbs=64, hsv_h=0.015, hsv_s=0.7, hsv_v=0.4, degrees=0.0, translate=0.1, scale=0.5, shear=0.0, perspective=0.0, flipud=0.0, fliplr=0.5, mosaic=1.0, mixup=0.0, copy_paste=0.0, cfg=None, hydra={'output_subdir': None, 'run': {'dir': '.'}}, v5loader=True, save_dir=runs/segment/train
Downloading https://ultralytics.com/assets/Arial.ttf to /root/.config/Ultralytics/Arial.ttf...
100% 755k/755k [00:00<00:00, 135MB/s]
Overriding model.yaml nc=80 with nc=1

                   from  n    params  module                                       arguments                     
  0                  -1  1       928  ultralytics.nn.modules.Conv                  [3, 32, 3, 2]                 
  1                  -1  1     18560  ultralytics.nn.modules.Conv                  [32, 64, 3, 2]                
  2                  -1  1     29056  ultralytics.nn.modules.C2f                   [64, 64, 1, True]             
  3                  -1  1     73984  ultralytics.nn.modules.Conv                  [64, 128, 3, 2]               
  4                  -1  2    197632  ultralytics.nn.modules.C2f                   [128, 128, 2, True]           
  5                  -1  1    295424  ultralytics.nn.modules.Conv                  [128, 256, 3, 2]              
  6                  -1  2    788480  ultralytics.nn.modules.C2f                   [256, 256, 2, True]           
  7                  -1  1   1180672  ultralytics.nn.modules.Conv                  [256, 512, 3, 2]              
  8                  -1  1   1838080  ultralytics.nn.modules.C2f                   [512, 512, 1, True]           
  9                  -1  1    656896  ultralytics.nn.modules.SPPF                  [512, 512, 5]                 
 10                  -1  1         0  torch.nn.modules.upsampling.Upsample         [None, 2, 'nearest']          
 11             [-1, 6]  1         0  ultralytics.nn.modules.Concat                [1]                           
 12                  -1  1    591360  ultralytics.nn.modules.C2f                   [768, 256, 1]                 
 13                  -1  1         0  torch.nn.modules.upsampling.Upsample         [None, 2, 'nearest']          
 14             [-1, 4]  1         0  ultralytics.nn.modules.Concat                [1]                           
 15                  -1  1    148224  ultralytics.nn.modules.C2f                   [384, 128, 1]                 
 16                  -1  1    147712  ultralytics.nn.modules.Conv                  [128, 128, 3, 2]              
 17            [-1, 12]  1         0  ultralytics.nn.modules.Concat                [1]                           
 18                  -1  1    493056  ultralytics.nn.modules.C2f                   [384, 256, 1]                 
 19                  -1  1    590336  ultralytics.nn.modules.Conv                  [256, 256, 3, 2]              
 20             [-1, 9]  1         0  ultralytics.nn.modules.Concat                [1]                           
 21                  -1  1   1969152  ultralytics.nn.modules.C2f                   [768, 512, 1]                 
 22        [15, 18, 21]  1   2770931  ultralytics.nn.modules.Segment               [1, 32, 128, [128, 256, 512]] 
YOLOv8s-seg summary: 261 layers, 11790483 parameters, 11790467 gradients, 42.7 GFLOPs

Transferred 411/417 items from pretrained weights
optimizer: SGD(lr=0.01) with parameter groups 66 weight(decay=0.0), 77 weight(decay=0.0005), 76 bias
albumentations: Blur(p=0.01, blur_limit=(3, 7)), MedianBlur(p=0.01, blur_limit=(3, 7)), ToGray(p=0.01), CLAHE(p=0.01, clip_limit=(1, 4.0), tile_grid_size=(8, 8))
train: Scanning /content/datasets/TicketDetection-3/train/labels... 150 images, 0 backgrounds, 0 corrupt: 100% 150/150 [00:00<00:00, 1414.79it/s]
train: New cache created: /content/datasets/TicketDetection-3/train/labels.cache
val: Scanning /content/datasets/TicketDetection-3/valid/labels... 14 images, 0 backgrounds, 0 corrupt: 100% 14/14 [00:00<00:00, 422.50it/s]
val: New cache created: /content/datasets/TicketDetection-3/valid/labels.cache
Image sizes 640 train, 640 val
Using 2 dataloader workers
Logging results to runs/segment/train
Starting training for 15 epochs...

      Epoch    GPU_mem   box_loss   seg_loss   cls_loss   dfl_loss  Instances       Size
  0% 0/10 [00:07<?, ?it/s]
Traceback (most recent call last):
  File "/usr/local/bin/yolo", line 8, in <module>
    sys.exit(entrypoint())
  File "/usr/local/lib/python3.8/dist-packages/ultralytics/yolo/cli.py", line 148, in entrypoint
    cli(cfg)
  File "/usr/local/lib/python3.8/dist-packages/ultralytics/yolo/cli.py", line 84, in cli
    func(cfg)
  File "/usr/local/lib/python3.8/dist-packages/hydra/main.py", line 79, in decorated_main
    return task_function(cfg_passthrough)
  File "/usr/local/lib/python3.8/dist-packages/ultralytics/yolo/v8/segment/train.py", line 153, in train
    model.train(**cfg)
  File "/usr/local/lib/python3.8/dist-packages/ultralytics/yolo/engine/model.py", line 203, in train
    self.trainer.train()
  File "/usr/local/lib/python3.8/dist-packages/ultralytics/yolo/engine/trainer.py", line 185, in train
    self._do_train(int(os.getenv("RANK", -1)), world_size)
  File "/usr/local/lib/python3.8/dist-packages/ultralytics/yolo/engine/trainer.py", line 303, in _do_train
    self.loss, self.loss_items = self.criterion(preds, batch)
  File "/usr/local/lib/python3.8/dist-packages/ultralytics/yolo/v8/segment/train.py", line 45, in criterion
    return self.compute_loss(preds, batch)
  File "/usr/local/lib/python3.8/dist-packages/ultralytics/yolo/v8/segment/train.py", line 91, in __call__
    masks = batch["masks"].to(self.device).float()
KeyError: 'masks'
Sentry is attempting to send 2 pending error messages
Waiting up to 2 seconds
Press Ctrl-C to quit

SkalskiP · 2023-01-26T16:25:11Z

From what I can see YOLOv8 team is working on solution for that problem right now: ultralytics/ultralytics#548 (comment)

Laughing-q · 2023-01-31T01:54:24Z

@Raiseku It has been solved in ultralytics/ultralytics#598 please upgrade you ultralytics package. :)

SkalskiP · 2023-01-31T10:37:10Z

@Laughing-q thanks for keeping us updated

Raiseku added the bug Something isn't working label Jan 26, 2023

SkalskiP self-assigned this Jan 26, 2023

SkalskiP added the yolov8 The problem originates in the YOLOv8 codebase label Jan 26, 2023

Raiseku closed this as completed Jan 31, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ValueError: Caught ValueError in DataLoader worker process 0. #53

ValueError: Caught ValueError in DataLoader worker process 0. #53

Raiseku commented Jan 26, 2023 •

edited

github-actions bot commented Jan 26, 2023

SkalskiP commented Jan 26, 2023

Raiseku commented Jan 26, 2023

SkalskiP commented Jan 26, 2023

Laughing-q commented Jan 31, 2023 •

edited

SkalskiP commented Jan 31, 2023

ValueError: Caught ValueError in DataLoader worker process 0. #53

ValueError: Caught ValueError in DataLoader worker process 0. #53

Comments

Raiseku commented Jan 26, 2023 • edited

Search before asking

Notebook name

Bug

Environment

Minimal Reproducible Example

Additional

Are you willing to submit a PR?

github-actions bot commented Jan 26, 2023

🐞 Bug reports

💬 Get in touch

SkalskiP commented Jan 26, 2023

Raiseku commented Jan 26, 2023

SkalskiP commented Jan 26, 2023

Laughing-q commented Jan 31, 2023 • edited

SkalskiP commented Jan 31, 2023

Raiseku commented Jan 26, 2023 •

edited

Laughing-q commented Jan 31, 2023 •

edited