Get segmentation from yolov5

[manoj-samal]

Search before asking

• I have searched the YOLOv5 issues and discussions and found no similar questions.

Question

How to get segmentations in form of [ [670,35][6,305] [60,3]] from segments model of yolov5

Additional

Thanks in advance

[github-actions]

👋 Hello @manoj-samal, thank you for your interest in YOLOv5 🚀! Please visit our ⭐️ Tutorials to get started, where you can find quickstart guides for simple tasks like Custom Data Training all the way to advanced concepts like Hyperparameter Evolution.

If this is a 🐛 Bug Report, please provide a minimum reproducible example to help us debug it.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset image examples and training logs, and verify you are following our Tips for Best Training Results.

Requirements

Python>=3.8.0 with all requirements.txt installed including PyTorch>=1.8. To get started:

git clone https://github.com/ultralytics/yolov5  # clone
cd yolov5
pip install -r requirements.txt  # install

Environments

YOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all YOLOv5 GitHub Actions Continuous Integration (CI) tests are currently passing. CI tests verify correct operation of YOLOv5 training, validation, inference, export and benchmarks on macOS, Windows, and Ubuntu every 24 hours and on every commit.

Introducing YOLOv8 🚀

We're excited to announce the launch of our latest state-of-the-art (SOTA) object detection model for 2023 - YOLOv8 🚀!

Designed to be fast, accurate, and easy to use, YOLOv8 is an ideal choice for a wide range of object detection, image segmentation and image classification tasks. With YOLOv8, you'll be able to quickly and accurately detect objects in real-time, streamline your workflows, and achieve new levels of accuracy in your projects.

Check out our YOLOv8 Docs for details and get started with:

pip install ultralytics

[glenn-jocher]

Hello!

Thank you for reaching out and for your interest in using YOLOv5 for segmentation tasks. To obtain segmentation results in the format you described (e.g., [[670,35],[6,305],[60,3]]), you can follow these steps:

1. Ensure You Have the Latest Version: First, make sure you are using the latest version of YOLOv5 and torch. You can update YOLOv5 by running:

   git pull
   pip install -U -r requirements.txt

2. Run Inference with Segmentation Model: Use the YOLOv5 segmentation model to run inference. Here is an example of how you can do this:

   import torch
   
   # Load the YOLOv5 segmentation model
   model = torch.hub.load('ultralytics/yolov5', 'custom', path='path/to/your/best.pt', force_reload=True)
   
   # Perform inference
   img = 'path/to/your/image.jpg'
   results = model(img)
   
   # Extract segmentation results
   segments = results.pandas().xyxy[0]['segments']

3. Format the Segmentation Results: The segmentation results can be formatted into the desired list of coordinate pairs. Here is an example of how you can achieve this:

   formatted_segments = []
   for segment in segments:
       formatted_segment = [[int(point[0]), int(point[1])] for point in segment]
       formatted_segments.append(formatted_segment)
   
   print(formatted_segments)

This will give you the segmentation results in the format [[670,35],[6,305],[60,3]].

If you encounter any issues or need further assistance, please provide a minimum reproducible code example so we can better understand the problem and help you more effectively. You can refer to our guide on creating a minimum reproducible example here: Minimum Reproducible Example.

Thank you for your cooperation, and happy coding! 😊

[manoj-samal]

results= model(im, augment=augment, visualize=visualize)
i'm getting output in tuple format

results = (
tensor[[[[ 2.55792e-01, -8.02494e-01, -1.52477e-02, ..., 1.31591e-01, 7.27416e-01, -7.37097e-01],
[ 4.26252e-01, -5.88908e-01, -2.77992e-01, ..., 5.57948e-02, 4.44748e-01, -7.22842e-01],
[-3.02036e-01, -2.36281e-01, -5.49053e-01, ..., -3.83684e-02, 3.37734e-01, -6.16241e-01]]]]], device='cuda:0'),

(tensor([[[[[ 0.74621, -0.36439, 0.65680, ..., -0.58979, 0.57147, -0.56773],
[-0.17386, -0.39769, 0.95084, ..., -0.61642, 0.48401, -0.48086],
[ 0.01340, -0.13087, 1.09881, ..., -0.53496, 0.38888, -0.53385]
]]]], device='cuda:0'),
[[[[[-5.41013e-02, -2.62636e-02, -4.65444e-02, ..., -5.09703e-02, -4.06154e-02, -7.81466e-02],
[-8.16091e-02, -5.18114e-02, -4.76743e-02, ..., -4.39999e-02, -4.33309e-02, -5.73979e-02],
[-8.58061e-02, -5.21048e-02, -5.23798e-03, ..., -5.07386e-02, -5.69061e-02, -5.13552e-02]]]], device='cuda:0' ))]

   we can seperate it  
   pred , out = results
   
   
   where len(pred) is num of detections
   and len(out) is 2 i dont know what it meant

[glenn-jocher]

@manoj-samal hello!

Thank you for reaching out and providing details about the issue you're encountering. Let's work together to resolve this.

From your description, it looks like you're getting the output in a tuple format when running inference with the YOLOv5 model. The tuple contains tensors that represent different parts of the model's output. Here's a breakdown of what you might be seeing:

1. Predictions (pred): This part of the output contains the detection results, including bounding boxes, confidence scores, and class predictions.
2. Additional Outputs (out): This part might include intermediate features or other relevant information from the model.

To help you better understand and utilize these outputs, let's go through a step-by-step example:

Step-by-Step Example

1. Run Inference:

   import torch
   
   # Load the YOLOv5 model
   model = torch.hub.load('ultralytics/yolov5', 'yolov5s', pretrained=True)
   
   # Perform inference
   img = 'path/to/your/image.jpg'
   results = model(img, augment=False, visualize=False)
   
   # Separate the results
   pred, out = results

2. Understanding the Outputs:

   • pred: Contains the detection results.
   • out: Contains additional outputs, which might include intermediate layers or other relevant data.

3. Extracting and Using Predictions:

   # Extract predictions
   predictions = pred[0]  # Assuming batch size of 1
   
   # Iterate through predictions
   for detection in predictions:
       # Extract bounding box coordinates, confidence score, and class prediction
       x1, y1, x2, y2, conf, cls = detection[:6]
       print(f"Bounding Box: ({x1}, {y1}, {x2}, {y2}), Confidence: {conf}, Class: {cls}")

Request for Additional Information

To further assist you, could you please provide a minimum reproducible code example? This will help us understand the context better and provide a more accurate solution. You can refer to our guide on creating a minimum reproducible example here: Minimum Reproducible Example.

Additionally, please ensure you are using the latest versions of torch and YOLOv5. You can update them using the following commands:

pip install --upgrade torch
git pull
pip install -U -r requirements.txt

If you have any more questions or need further clarification, feel free to ask. We're here to help! 😊

[manoj-samal]

pred: Contains the detection results.
out: Contains additional outputs, which might include intermediate layers or other relevant data.

after prediction is done the pred which has masks and out which has protos it undergoes process mask

def process_mask(protos, masks_in, bboxes, shape, upsample=False):
    
    Crop before upsample.
    proto_out: [mask_dim, mask_h, mask_w]
    out_masks: [n, mask_dim], n is number of masks after nms
    bboxes: [n, 4], n is number of masks after nms
    shape:input_image_size, (h, w)

    return: h, w, n
   

    c, mh, mw = protos.shape  # CHW
    ih, iw = shape
    masks = (masks_in @ protos.float().view(c, -1)).sigmoid().view(-1, mh, mw)  # CHW

    downsampled_bboxes = bboxes.clone()
    downsampled_bboxes[:, 0] *= mw / iw
    downsampled_bboxes[:, 2] *= mw / iw
    downsampled_bboxes[:, 3] *= mh / ih
    downsampled_bboxes[:, 1] *= mh / ih

    masks = crop(masks, downsampled_bboxes)  # CHW
    if upsample:
        masks = F.interpolate(masks[None], shape, mode='bilinear', align_corners=False)[0]  # CHW
    return masks.gt_(0.5)
masks = process_mask(proto[i], det[:, 6:], det[:, :4], im.shape[2:], upsample=True)  # HWC

now this masks is in binary

from this masks which is binary i need [ [670,35][6,305] [60,3]] segments

[glenn-jocher]

Hello @manoj-samal,

Thank you for providing detailed information about your issue. Let's work through this step-by-step to help you extract the segmentation coordinates from the binary masks.

Step-by-Step Solution

1. Ensure Latest Versions: First, please ensure you are using the latest versions of torch and YOLOv5. You can update them using:

   pip install --upgrade torch
   git pull
   pip install -U -r requirements.txt

2. Process Masks: You already have a function process_mask that converts the protos and masks into binary masks. Now, we need to extract the coordinates from these binary masks.

3. Extract Segmentation Coordinates: We can use OpenCV to find contours in the binary masks and extract the coordinates. Here’s how you can do it:

   import cv2
   import numpy as np
   
   def extract_segments(masks):
       segments = []
       for mask in masks:
           mask_np = mask.cpu().numpy().astype(np.uint8)  # Convert to numpy array
           contours, _ = cv2.findContours(mask_np, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
           for contour in contours:
               segment = contour.squeeze().tolist()  # Convert to list of coordinates
               segments.append(segment)
       return segments
   
   # Assuming `masks` is the binary mask tensor from `process_mask`
   segments = extract_segments(masks)
   print(segments)

Explanation

• Binary Mask to Numpy: Convert the binary mask tensor to a numpy array for processing with OpenCV.
• Find Contours: Use cv2.findContours to find the contours in the binary mask.
• Extract Coordinates: Convert the contours to a list of coordinates.

Request for Minimum Reproducible Example

If you encounter any issues or the solution does not work as expected, please provide a minimum reproducible code example. This will help us understand the problem better and provide a more accurate solution. You can refer to our guide on creating a minimum reproducible example here: Minimum Reproducible Example.

By ensuring we can reproduce the issue, we can investigate it more effectively and provide a timely solution.

Conclusion

I hope this helps you extract the segmentation coordinates from the binary masks. If you have any further questions or need additional assistance, please feel free to ask. We're here to help! 😊