Unable to use annotation with different number of keypoints

[MaxTeselkin]

Search before asking

• I have searched the YOLOv8 issues and discussions and found no similar questions.

Question

Hi! I have a custom dataset annotated for pose estimation task. But number of keypoints is not static - maximum number of keypoints is 3, but sometimes I annotate my objects with only 1 or 2 keypoints.

In my .yaml file I set the following keypoints shape:

kpt_shape: [3, 2] - my maximum number of keypoints is 3 and I have only x and y coordinates (no visibility parameter).

When I am trying to train my pose estimation model, I am getting the following error:

File "/app/repo/./train/src/main.py", line 1120, in start_training
    model.train(
  File "/usr/local/lib/python3.8/dist-packages/ultralytics/yolo/engine/model.py", line 371, in train
    self.trainer.train()
  File "/usr/local/lib/python3.8/dist-packages/ultralytics/yolo/engine/trainer.py", line 192, in train
    self._do_train(world_size)
  File "/usr/local/lib/python3.8/dist-packages/ultralytics/yolo/engine/trainer.py", line 273, in _do_train
    self._setup_train(world_size)
  File "/usr/local/lib/python3.8/dist-packages/ultralytics/yolo/engine/trainer.py", line 255, in _setup_train
    self.train_loader = self.get_dataloader(self.trainset, batch_size=batch_size, rank=RANK, mode='train')
  File "/usr/local/lib/python3.8/dist-packages/ultralytics/yolo/v8/detect/train.py", line 54, in get_dataloader
    dataset = self.build_dataset(dataset_path, mode, batch_size)
  File "/usr/local/lib/python3.8/dist-packages/ultralytics/yolo/v8/detect/train.py", line 28, in build_dataset
    return build_yolo_dataset(self.args, img_path, batch, self.data, mode=mode, rect=mode == 'val', stride=gs)
  File "/usr/local/lib/python3.8/dist-packages/ultralytics/yolo/data/build.py", line 74, in build_yolo_dataset
    return YOLODataset(
  File "/usr/local/lib/python3.8/dist-packages/ultralytics/yolo/data/dataset.py", line 39, in __init__
    super().__init__(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/ultralytics/yolo/data/base.py", line 72, in __init__
    self.labels = self.get_labels()
  File "/usr/local/lib/python3.8/dist-packages/ultralytics/yolo/data/dataset.py", line 132, in get_labels
    len_cls, len_boxes, len_segments = (sum(x) for x in zip(*lengths))
ValueError: not enough values to unpack (expected 3, got 0)

The cause of this problem is the fact that ultralytics expects my annotations to always have 3 points, but as I mentioned above sometimes my objects annotated with only 1 or 2 points.

What should I do? How can I train YOLOv8-pose model on dataset with dynamic number of keypoints?

P.S. My ultralytics version is 8.0.112

Additional

No response

[glenn-jocher]

@MaxTeselkin hello! Thank you for your detailed issue report. You provide very good insight into the problem you're facing with a variable number of keypoints in your custom annotations while training your YOLOv8 model for pose estimation.

From your description, I understand you have maximum of 3 keypoints, but sometimes fewer - either 1 or 2 keypoints are annotated. Due to the static configuration (kpt_shape: [3, 2]), the YOLOv8 indeed would expect your annotations to always have 3 points.

However, YOLOv8 expects a static number of keypoints for its pose estimation module and it can't handle a varying number of keypoints natively. As such, trying to vary the number of keypoints for a particular set of boxes will almost certainly cause dimension mismatches and errors.

To manage this, I would recommend you to still maintain kpt_shape: [3, 2] in your .yaml file and adjust your dataset to always have 3 keypoints. For the cases where you have less keypoints, you can add pseudo-keypoints with easily identifiable dummy values or coordinates that won't show up in your images or interfere with your training process. This way you maintain a consistent annotation structure while working around the constraints of the model.

I understand this is not a perfect solution for dynamic keypoints, and we appreciate your patience as we continue to iterate on our software. Let me know if you have any other questions or issues!

[MaxTeselkin]

I understood, thanks for response

[glenn-jocher]

@MaxTeselkin you're welcome! I'm glad I could provide you with the information you needed. If you have any more questions or run into any issues as you continue working on your project, please don't hesitate to reach out. The community and I are here to help. Happy coding!