tasks/pose/

[giscus[bot]]

tasks/pose/

Learn how to use Ultralytics YOLOv8 for pose estimation tasks. Find pretrained models, learn how to train, validate, predict, and export your own.

https://docs.ultralytics.com/tasks/pose/

[KingAsperin]

Do you already have a documentation on how to implement yolov8 landmark detection on android development? if none is it posible for you to make a thread to teach us how to implement yolov8 model into android development please

[pderrenger]

@KingAsperin hey there! 👋

We don't currently have a dedicated guide for implementing YOLOv8 landmark detection on Android, but it's a great suggestion! 🚀

In the meantime, you can export a YOLOv8 pose model to ONNX or TensorFlow formats, which are more compatible with Android development. For example, you can use the following command to export a model:

yolo export model=yolov8n-pose.pt format=onnx

Once you have the ONNX model, you can use tools like ONNX Runtime Mobile to run it on Android devices. For TensorFlow, you can convert the model to TensorFlow Lite and use the TensorFlow Lite Android Support Library.

I hope this helps! If you have more questions or need further assistance, feel free to ask. Happy coding! 😊

[leonidasavila]

How can i calculate angle position between three points? Thanks

[pderrenger]

@leonidasavila hey there! 👋 To calculate the angle between three points, you can use the law of cosines. Here's a quick example in Python:

import numpy as np

def calculate_angle(p1, p2, p3):
    # p1, p2, p3 are the points in format [x, y]
    # Calculate the vectors
    v1 = np.array(p1) - np.array(p2)
    v2 = np.array(p3) - np.array(p2)
    
    # Calculate the angle in radians
    angle_rad = np.arccos(np.dot(v1, v2) / (np.linalg.norm(v1) * np.linalg.norm(v2)))
    
    # Convert to degrees
    angle_deg = np.degrees(angle_rad)
    
    return angle_deg

# Example points
p1 = [0, 0]
p2 = [1, 0]
p3 = [1, 1]

angle = calculate_angle(p1, p2, p3)
print(f"The angle is {angle} degrees")

This calculates the angle at p2 formed by the line segments p1p2 and p2p3. Hope this helps! Let me know if you have any other questions. 😊

[leonidasavila]

Thanks @pderrenger, for your quickly response! I´m trying to migrate our startup app from mediapipe to yolo 😍

Do you have any tutorial or short example to define specifics landmarks to make calculates? (4 example push-up)

Best regards!
Leonidas

[pderrenger]

Hey Leonidas! 🌟

It's awesome to hear about your transition from MediaPipe to YOLO for your startup app! For specific landmark detection like push-ups, you can leverage our YOLOv8 pose models. Here's a quick example to get you started:

from ultralytics import YOLO

# Load a pretrained pose model
model = YOLO('yolov8n-pose.pt')

# Predict keypoints on an image
results = model('path/to/your/image.jpg')

# Access keypoints
keypoints = results[0].keypoints

# Now, you can use these keypoints to calculate angles or any other metrics you need

This snippet loads a pretrained YOLOv8 pose model and predicts keypoints on your image. You can then use these keypoints for further calculations, like angles between joints during push-ups.

For more detailed guidance, check out our Pose Estimation page. If you have any more questions or need further assistance, feel free to ask. Happy coding! 😊

[ozankavcu38]

By the way sir after loading the model how could we dissapear the pose estimatet lines like boxes=False argument

[pderrenger]

Hey there! 😊 To hide the pose estimation lines (similar to using boxes=False for bounding boxes), you can simply not plot them. When you're working with the results from the pose estimation model, you have control over what gets visualized. Here's a quick example on how to display the keypoints without drawing the connecting lines:

from ultralytics import YOLO

# Load your pose model
model = YOLO('yolov8n-pose.pt')

# Run prediction
results = model('path/to/image.jpg')

# Access keypoints
keypoints = results[0].keypoints

# Now, you can visualize keypoints without drawing lines between them
# This part depends on how you're visualizing. For example, using OpenCV to plot keypoints:
import cv2
image = cv2.imread('path/to/image.jpg')
for kpt in keypoints.xy:
    x, y = int(kpt[0]), int(kpt[1])
    cv2.circle(image, (x, y), radius=3, color=(0, 255, 0), thickness=-1)

cv2.imshow("Keypoints without lines", image)
cv2.waitKey(0)
cv2.destroyAllWindows()

This way, you'll see the keypoints on your image without the estimated pose lines. Hope this helps, and good luck with your app migration! If you have any more questions, feel free to ask. 🚀

[ozankavcu38]

Sir i just wanna ask you something about live training . I was making some calculations and getting some results that is okay by the way yolov8 is very good about pose estimation issues. I just wanna ask when i am training mp4 files ( my main is live training however i am just using for the testing this ) . The model is drawing everything like boxes poses conf values etc. I dont want to see nothing on the video however i just want to keep operating the results by the way . In addition to this i just want to write something on video like i have counted 3 people on the top of right corner of that video how could i do this i did not see any answer . when i turn off results = model(source="C://Users//Hp//Desktop//videolar//operation_room_1.mp4", show=True, conf=0.3, save=False,
verbose=False, stream=True,show_conf=True). like show false i did not see anything about video it is completely dissapeared or when i turn off stream yeah my calculations is continue however the video is dissapeared . I want to show the video appeared however there must be nothing as image , poses , boxes .

[glenn-jocher]

Hello! It sounds like you're doing some interesting work with live training and pose estimation using YOLOv8. To address your concerns about visualizing the video without drawing boxes, poses, or confidence values, while still being able to operate on the results and add custom text annotations, here's a simplified approach:

1. To process the video without visualizing detections: You can set show=False to prevent displaying the video with detections. This will still allow you to access the detection results programmatically.

2. To add custom text annotations: You can use OpenCV to write text on the video frames. Here's a small code snippet to help you get started:

import cv2
from ultralytics import YOLO

# Load your model
model = YOLO('path/to/your/model.pt')

# Open the video
cap = cv2.VideoCapture("C://Users//Hp//Desktop//videolar//operation_room_1.mp4")

# Process each frame
while cap.isOpened():
    ret, frame = cap.read()
    if not ret:
        break

    # Run model prediction
    results = model(frame, show=False)

    # Example: Count people and add text annotation
    people_count = sum(1 for *_, cls in results.xyxy[0] if cls == 0)  # Assuming class 0 is 'person'
    cv2.putText(frame, f"People Count: {people_count}", (50, 50), cv2.FONT_HERSHEY_SIMPLEX, 1, (255, 255, 255), 2)

    # Show the frame
    cv2.imshow('Frame', frame)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

cap.release()
cv2.destroyAllWindows()

This script processes each frame to count the number of people (assuming class 0 corresponds to 'person') and displays the count on the top right corner of the video. Adjust the class index as per your model's class labels.

Remember, the show=False parameter in the model() call ensures that the original video is processed without overlaying the detections, while cv2.putText() is used to add custom text annotations.

I hope this helps! Let me know if you have any more questions. 😊

[samverh]

Hi, is it possible to run this model on the OAK-D camera with VPU?

[pderrenger]

Hey there! 😊

After converting your YOLOv8 model for use with the OAK camera, the output format indeed changes, and parsing the results might require some adjustments. For handling the output layer and parsing keypoints correctly, I recommend checking out the Pose Estimation documentation on the Ultralytics Docs page. It provides insights into how keypoints are represented and might give you a good starting point.

For a quick example, if you're working with an ONNX model, you might use something like this to parse the output:

import numpy as np

# Assuming 'output' is the model's output tensor
keypoints = np.where(output > confidence_threshold)  # Apply confidence threshold
# Process 'keypoints' as needed

This is a simplified example, and the exact implementation will depend on your model's output structure. If you need more specific guidance, feel free to ask! 🚀

[samverh]

Hi, your help is really appreciated.

I am still struggling with parsing the output in the proper way. I inspected the model using Netron and the output layer should have a shape of (1, 56, 8400). The actual shape is (470400, ) so I perform the following conversion:

outputs = data.reshape(1, 56, 8400)
outputs = outputs.transpose(0, 2, 1)

Now I assume the bounding boxes can be extracted by iterating over the data as outputs[0][i][:4] where i is the row index. Then, I convert the boxes with the following function and filter the most relevant boxes using NMS.

def xywh2box(bbox_vals):

    center_x = bbox_vals[0]
    center_y = bbox_vals[1]
    w = bbox_vals[2] 
    h = bbox_vals[3]
    x = center_x - 0.5 * w # center_x to top_left_x
    y = center_y - 0.5 * h # center_y to top_left_y
    box = [x, y, x + w, y + h]

    return box

When plotting the results including keypoint coordinates I am getting very small boxes. It seems that my data conversion is incorrect or that I should apply scaling, but the image sizes and input shapes are both (640, 640).

What am I missing here?

[pderrenger]

@samverh hey there! 👋 It looks like you're on the right track with reshaping and transposing the output. However, if the boxes appear too small, it might indeed be a scaling issue. When working with models converted to OpenVINO and running on devices like the OAK-D, remember that the output coordinates are normalized. You might need to scale them back to the original image dimensions.

Here's a quick tip: Ensure you're scaling the bounding box coordinates by the original image width and height before converting them with your xywh2box function. For example:

def scale_boxes(outputs, img_width, img_height):
    scaled_outputs = []
    for i in range(outputs.shape[1]):  # Iterate over each detection
        bbox = outputs[0][i][:4]
        bbox[0] *= img_width  # Scale x
        bbox[1] *= img_height # Scale y
        bbox[2] *= img_width  # Scale width
        bbox[3] *= img_height # Scale height
        scaled_outputs.append(xywh2box(bbox))
    return scaled_outputs

Make sure to replace img_width and img_height with the actual dimensions of your input images. This should help correct the size of the bounding boxes. If you're still facing issues, double-check the model's input and output layer specifications and ensure they align with your preprocessing steps. Keep up the great work! 🚀

[samverh]

The problem was the input from the OAK-D camera. Frames from ColorCamera/MonoCamera are in U8 data type [0, 255]. To obtain [0, 1] values that can be passed to the model, rescaling is needed by setting mean=0 and scale=255. This can be done when converting the model to blob format:

import blobconverter

# convert onnx to blob
blobconverter.from_onnx(
    model=f"{yolo_modelname}.onnx", 
    output_dir=f"{folder}/",
    data_type="FP16",
    shaves=6,
    use_cache=False,

    optimizer_params=[
        "--mean_values=[0,0,0]",
        "--scale_values=[255,255,255]",
    ]
)

Thanks again for your help! 🙌

[pderrenger]

Hey there! 🙌 It looks like you've done a great job converting the model and reshaping the output for use with your OAK-D camera. The issue with small bounding boxes might indeed stem from not scaling the output coordinates back to the original image dimensions.

Given the output shape (1, 56, 8400) and after your reshaping to (1, 8400, 56), it's crucial to remember that the model outputs are normalized to [0, 1]. To map these back to the original image size, you'll need to scale the bounding box coordinates (x, y, width, height) by the width and height of the input image.

Here's a quick adjustment to your xywh2box function to include scaling:

def xywh2box(bbox_vals, img_width, img_height):
    center_x = bbox_vals[0] * img_width
    center_y = bbox_vals[1] * img_height
    w = bbox_vals[2] * img_width
    h = bbox_vals[3] * img_height
    x = center_x - 0.5 * w  # Convert center_x to top_left_x
    y = center_y - 0.5 * h  # Convert center_y to top_left_y
    box = [x, y, x + w, y + h]
    return box

Make sure to pass the img_width and img_height of your original image to this function. This should correct the size of your bounding boxes. If you're still encountering issues, double-check the output layer's interpretation to ensure you're extracting the bounding box and keypoint information correctly.

Keep up the great work, and don't hesitate to reach out if you have more questions! 😊

[shekharsingh8811]

Hi,
Could you please let me know which approach does the yolov8-pose model take? And, how does it output the bounding boxes?
Is it,

1. Top-down approach — which involves a person detector that outputs the bounding boxes and then a pose is calculated for each of these bounding boxes.
   OR
2. Bottom-up approach — which involves detecting all parts of each person within an image and then associating or grouping the parts that belong to each individual. Finally, the estimated pose is then used to calculate the bounding box thus encapsulating all the detected keypoints within it.

My motive here is to figure out, if there would be any difference between the bounding box output that I would get from yolov8 (detection) model and the yolov8-pose model. Would there be any?

Thank You!!

[glenn-jocher]

@shekharsingh8811 hey there! 😊

Absolutely, you can use bounding boxes from another model as input for YOLOv8-pose for pose estimation. While YOLOv8-pose is designed for a top-down approach, integrating external bounding boxes requires a bit of manual preprocessing. Here's a quick example on how you might do it:

from ultralytics import YOLO
import cv2

# Load your YOLOv8-pose model
model = YOLO('yolov8n-pose.pt')

# Assuming 'external_boxes' is a list of bounding boxes [x1, y1, x2, y2] from another model
# and 'image' is the image you're working with
for box in external_boxes:
    # Crop the person from the image using the bounding box
    x1, y1, x2, y2 = box
    person_crop = image[y1:y2, x1:x2]

    # Run pose estimation on the cropped image
    results = model(person_crop)

    # Now, results contain keypoints for the pose within the cropped image
    # You can plot these keypoints directly on the original image or the cropped image

This way, you're manually handling the bounding box to keypoint part of the top-down approach. The bounding boxes from YOLOv8 detection and those used for YOLOv8-pose might differ, but this method allows you to use any source of bounding boxes!

Hope this helps! Let me know if you have more questions.

[shekharsingh8811]

Hi @glenn-jocher,

I'm currently using the yolov8s-pose.engine model on my dataset which comprises of 640 x 640 frames / images. The person in the scene is at times at the edges of the frame and thus only partially visible, irrespective of that the model always outputs 17 keypoints and tries to squash the non-visible keypoints towards the edges of the frame. Is this expected?
On the other hand, I expected the keypoint coordinates to be in the range (0.0 , 639.0) but for these edge cases the model at times outputs the keypoint x or y coordinate value as 640.0 which it shouldn't right?

Could you please your thoughts on this.

Thank You!

[glenn-jocher]

Hi @shekharsingh8811,

Thank you for your detailed question! 😊

The behavior you're observing, where the YOLOv8-pose model outputs 17 keypoints even for partially visible persons and occasionally produces keypoint coordinates at the edge (640.0), is expected. This is because the model is designed to predict all keypoints regardless of visibility, and it attempts to place non-visible keypoints as close to the edge of the frame as possible.

Regarding the keypoint coordinates reaching 640.0, this can happen due to the model's bounding box normalization and rounding during inference. While the coordinates should ideally be within the image dimensions (0.0 to 639.0), slight deviations can occur.

To address this, you can post-process the keypoints to ensure they fall within the expected range. Here's a quick example in Python:

import numpy as np

# Assuming 'results' is the output from the model
for result in results:
    keypoints = result.keypoints.xy
    keypoints = np.clip(keypoints, 0, 639)  # Ensure keypoints are within the image bounds
    # Further processing or visualization

If you haven't already, please ensure you're using the latest versions of torch and ultralytics. You can update them using:

pip install --upgrade torch ultralytics

If the issue persists, could you provide a minimum reproducible example? This will help us investigate further. You can find guidelines for creating one here: Minimum Reproducible Example.

Feel free to reach out if you have more questions or need further assistance!

[shekharsingh8811]

Hi @glenn-jocher,

Thank you for the response!
I'm already clipping the keypoints for my use case, so this resolves the issue for now.

[glenn-jocher]

Hi @shekharsingh8811,

Thank you for your response! 😊

I'm glad to hear that clipping the keypoints resolves the issue for your use case. The behavior you described, where the model outputs keypoints even for partially visible persons and occasionally produces coordinates at the edge (640.0), is indeed expected due to the model's bounding box normalization and rounding during inference.

If you have any further questions or need additional assistance, feel free to ask. For more detailed information on pose estimation, you can refer to our Pose Estimation Documentation.

Happy coding! 🚀

[StavrosAnaniadis]

Hello! I am trying to output all the points detected by yolov8 into a excel or csv file in a video, and am having trouble with it. I am using the code:
from ultralytics import YOLO

model = YOLO('yolov8m-pose.pt')
video_path = #video path here

results = model(source=video_path, show=True, conf = 0.3, save=True)

Is there any way I can save all of the coordinates for the 2 people in the video to an excel file?

Thank you,
Stavros Ananiadis

[pderrenger]

@StavrosAnaniadis hi there! 🙌

For saving the keypoints to an Excel or CSV file, including their labels like "left arm", "right wrist" etc., you'll need to modify the approach slightly. Based on your requirement and the given structure of the Keypoints object, here's a concise example using pandas to save the data in a more structured format, including the labels:

import pandas as pd
from ultralytics import YOLO

# Define keypoint labels for your pose model (order should match the model's output)
labels = ["nose", "left_eye", "right_eye", "left_ear", "right_ear", "left_shoulder",
          "right_shoulder", "left_elbow", "right_elbow", "left_wrist", "right_wrist",
          "left_hip", "right_hip", "left_knee", "right_knee", "left_ankle", "right_ankle"]

model = YOLO('yolov8m-pose.pt')
results = model(source=video_path, show=True, conf=0.3, save=True)

# Initialize an empty list to store data
data = []

# Loop through results and store keypoints data
for frame_results in results:
    for person_id, person in enumerate(frame_results.keypoints):
        for label, (x, y, conf) in zip(labels, person.numpy()):
            data.append({
                "person_id": person_id,
                "label": label,
                "x": x,
                "y": y,
                "confidence": conf
            })

# Create a DataFrame and save it to an Excel file
df = pd.DataFrame(data)
df.to_excel("keypoints.xlsx", index=False)

This script runs through the keypoints detected on each person in every frame, collecting their labels, coordinates, and confidence levels into a structured format that's then written to an "keypoints.xlsx" Excel file.

Hope this serves your project well, especially with the fascinating application of machine learning in analyzing sword fights! If you need more info on the keypoint indexes and labels, referring to this Ultralytics docs page might be helpful. Good luck with your analysis! 🤺👀

[StavrosAnaniadis]

Thank you so much for your help.

When I run this code, I get the error:
for label, (x, y, conf) in zip(labels, person.numpy()):

ValueError: not enough values to unpack (expected 3, got 1)
Referring to line 26: for label, (x, y, conf) in zip(labels, person.numpy()):

Do you know why this would not be working?

Thank you,
Stavros Ananiadis

[pderrenger]

Hi Stavros,

It looks like the error stems from unpacking the keypoints incorrectly. The keypoints are structured as xy, xyn, and conf for the coordinates, normalized coordinates, and confidences respectively, according to the Ultralytics documentation. Here's how you can modify your loop for correct unpacking:

import pandas as pd

data = []
# Assuming 'results' is from your prediction
for frame_id, result in enumerate(results):
    for person_id, person in enumerate(result.keypoints):
        keypoints = person.xy  # Get keypoints
        confidences = person.conf  # Get confidences
        for kp_id, (xy, conf) in enumerate(zip(keypoints, confidences)):
            label = labels[kp_id]  # Get the label from predefined labels list
            data.append([frame_id, person_id, label, *xy, conf])

# Convert the data into a pandas DataFrame
df = pd.DataFrame(data, columns=['frame_id', 'person_id', 'label', 'x', 'y', 'confidence'])

# Save to an Excel file
df.to_excel('keypoints.xlsx', index=False)

Remember, for this to work, ensure you have labels list correctly ordered as per your model's keypoints output.

This adaptation should resolve the unpacking issue and store coordinates with their corresponding labels like "left arm", "right wrist", etc. 🚀

Best of luck with your project on providing insights to sword fighters. It sounds like a cutting-edge (pun intended) application of AI! If you have more questions or hit another snag, feel free to ask.

[StavrosAnaniadis]

Hello! Thank you for your continued assistance. On running the code, a new error appeared:
ValueError: 6 columns passed, passed data had 21 columns
My labels look like:

Define keypoint labels for your pose model (order should match the model's output)

labels = ["nose", "left_eye", "right_eye", "left_ear", "right_ear", "left_shoulder",
"right_shoulder", "left_elbow", "right_elbow", "left_wrist", "right_wrist",
"left_hip", "right_hip", "left_knee", "right_knee", "left_ankle", "right_ankle"]
Would you happen to know whats causing that?

Thank you,
Stavros Ananiadis

[pderrenger]

Hello Stavros,

It seems the issue arises because there’s a mismatch in the structure of the data being written to the CSV and the expected format. The error indicates that the DataFrame construction is attempting to handle more columns than specified.

Given the ValueError related to column mismatch and considering the structure of the keypoints in the YOLOv8 pose estimation model, it appears we need to adjust how the keypoints are unpacked before saving them. Specifically, the xy, conf values should be handled correctly.

The correct usage for processing keypoints and confidences involves accessing these attributes directly from the person object as .xy or .conf and ensuring each keypoint and its confidence are correctly represented:

import pandas as pd
from ultralytics import YOLO

model = YOLO('yolov8m-pose.pt')
results = model(source=video_path, conf=0.3)

data = []

for frame_id, (frame_results) in enumerate(results):
    for person_id, person in enumerate(frame_results.keypoints):
        xy = person.xy.numpy()  # keypoints coordinates
        conf = person.conf.numpy()  # confidence scores of keypoints
        for kp_id, (xy, kp_conf) in enumerate(zip(xy, conf)):
            label = labels[kp_id]  # Using the predefined labels list
            x, y = xy
            data.append({"frame_id": frame_id, "person_id": person_id, "label": label, "x": x, "y": y, "confidence": kp_conf})

df = pd.DataFrame(data)
df.to_csv('keypoints.csv', index=False)

Make sure the labels list matches your model's output order. This script aims to correctly handle the extraction of keypoint coordinates and confidence scores, complementing your innovative approach to analyzing sword fighting techniques with AI. Keep pushing the boundaries! 🤺🚀

If you have more questions or need further assistance, feel free to reach out.

[arav1ndb]

Where can I find the mapping for the xy coordinates obtained by the algorithm, as in which point pertains to which body part?

[arav1ndb]

I got it here, for anyone else searching for it

# Define keypoint labels for your pose model (order should match the model's output)
labels = ["nose", "left_eye", "right_eye", "left_ear", "right_ear", "left_shoulder",
          "right_shoulder", "left_elbow", "right_elbow", "left_wrist", "right_wrist",
          "left_hip", "right_hip", "left_knee", "right_knee", "left_ankle", "right_ankle"]

[glenn-jocher]

@arav1ndb great find! 🌟 For anyone else curious about the keypoint labels for pose estimation models, here they are:

labels = [
    "nose", "left_eye", "right_eye", "left_ear", "right_ear",
    "left_shoulder", "right_shoulder", "left_elbow", "right_elbow",
    "left_wrist", "right_wrist", "left_hip", "right_hip",
    "left_knee", "right_knee", "left_ankle", "right_ankle"
]

These labels correspond to the output keypoints of your pose model. When interpreting [x, y] coordinate predictions, you can match each point to the respective body part listed above. Happy coding! 😊

[dolfMo]

Hello! I would like to ask if it is possible to output the coordinate information of the specified key point, which is the coordinate point of the wrist

[glenn-jocher]

@dolfMo hello! 😊 Yes, it's definitely possible to get the coordinate information of specific keypoints, such as the wrist, using YOLOv8's pose estimation models. If you're working with a model like yolov8n-pose.pt, the keypoint coordinates, including those for the wrist, are outputted as part of the model's predictions.

Here's a quick Python example on how to access these keypoints:

from ultralytics import YOLO

# Load the trained pose estimation model
model = YOLO('yolov8n-pose.pt')

# Run predictions
results = model('path/to/your/image.jpg')

# Access keypoints for each detected person
for person in results.keypoints:
    wrist_coords = person.xy[person.names.index('wrist')]  # Assuming 'wrist' is part of your model's keypoints
    print(f'Wrist coordinates: {wrist_coords}')

Make sure to replace 'path/to/your/image.jpg' with the path to your image. The keypoint names (like 'wrist') and their exact order depend on your model and dataset, so you might need to adjust the person.names.index('wrist') part to fit your specifics.

Let me know if there's anything else I can help with! 🌟

[dolfMo]

Hello, I encountered the following error message after executing your code. I would like to know why?
for person in results.keypoints: AttributeError: 'list' object has no attribute 'keypoints'

[jirivchi1]

result = results[0]
Extract keypoints
keypoints = result.keypoints

It works for me in the previous code but I am got a new error

[glenn-jocher]

Hey there! 👋 It looks like you're encountering an attribute error because results is a list, not directly accessing attributes. For pose estimation with YOLOv8, you'll need to interact with the list of results to get the keypoints. Here's how you can correctly access keypoints:

from ultralytics import YOLO

# Load your model
model = YOLO('yolov8n-pose.pt') 

# Run your prediction
results = model('path/to/image.jpg')

# Correct way to access keypoints 
for result in results:
    if hasattr(result, 'keypoints'):
        for person in result.keypoints:
            wrist_coords = person.xy[person.names.index('wrist')]
            print(f'Wrist coordinates: {wrist_coords}')

This snippet checks if the keypoints attribute exists in each result and then iterates through each person detected to find the wrist coordinates. Let me know if that helps or if you have any more questions!

Keep on exploring! 🚀

[pyrat]

Hi,

I have trained a model based on YOLOv8l-pose, when I infer the model I get None for confidence and the keypoints just have x and y. Is there a reason why there is no confidence? here is contents of the keypoints object:

proto-proto-1 | Initializing LandmarkDetector
proto-proto-1 | YOLOv8l-pose summary (fused): 287 layers, 44464206 parameters, 0 gradients, 168.6 GFLOPs
proto-proto-1 |
proto-proto-1 | 0: 192x640 1 fish, 56.9ms
proto-proto-1 | Speed: 1.3ms preprocess, 56.9ms inference, 164.3ms postprocess per image at shape (1, 3, 192, 640)
proto-proto-1 | ultralytics.engine.results.Keypoints object with attributes:
proto-proto-1 |
proto-proto-1 | conf: None
proto-proto-1 | data: tensor([[[ 7.0377, 94.4488],
proto-proto-1 | [ 35.9414, 107.0425],
proto-proto-1 | [ 1.5750, 111.7595],
proto-proto-1 | [ 55.6107, 92.0907],
proto-proto-1 | [101.0829, 49.3980],
proto-proto-1 | [147.8248, 93.4525],
proto-proto-1 | [112.3623, 138.2682],
proto-proto-1 | [153.1546, 141.8253],
proto-proto-1 | [277.1523, 16.1814],
proto-proto-1 | [343.0081, 22.3252],
proto-proto-1 | [324.8737, 135.0928],
proto-proto-1 | [429.2652, 111.5511],
proto-proto-1 | [470.5245, 98.0572],
proto-proto-1 | [457.7650, 36.1689],
proto-proto-1 | [479.3454, 46.8262],
proto-proto-1 | [507.4113, 54.2151],
proto-proto-1 | [532.2663, 45.4843],
proto-proto-1 | [542.9807, 83.6471],
proto-proto-1 | [521.8376, 102.8704],
proto-proto-1 | [506.2361, 90.8436]]], device='cuda:0')
proto-proto-1 | has_visible: False
proto-proto-1 | orig_shape: (154, 576)
proto-proto-1 | shape: torch.Size([1, 20, 2])
proto-proto-1 | xy: tensor([[[ 7.0377, 94.4488],
proto-proto-1 | [ 35.9414, 107.0425],
proto-proto-1 | [ 1.5750, 111.7595],
proto-proto-1 | [ 55.6107, 92.0907],
proto-proto-1 | [101.0829, 49.3980],
proto-proto-1 | [147.8248, 93.4525],
proto-proto-1 | [112.3623, 138.2682],
proto-proto-1 | [153.1546, 141.8253],
proto-proto-1 | [277.1523, 16.1814],
proto-proto-1 | [343.0081, 22.3252],
proto-proto-1 | [324.8737, 135.0928],
proto-proto-1 | [429.2652, 111.5511],
proto-proto-1 | [470.5245, 98.0572],
proto-proto-1 | [457.7650, 36.1689],
proto-proto-1 | [479.3454, 46.8262],
proto-proto-1 | [507.4113, 54.2151],
proto-proto-1 | [532.2663, 45.4843],
proto-proto-1 | [542.9807, 83.6471],
proto-proto-1 | [521.8376, 102.8704],
proto-proto-1 | [506.2361, 90.8436]]], device='cuda:0')
proto-proto-1 | xyn: tensor([[[0.0122, 0.6133],
proto-proto-1 | [0.0624, 0.6951],
proto-proto-1 | [0.0027, 0.7257],
proto-proto-1 | [0.0965, 0.5980],
proto-proto-1 | [0.1755, 0.3208],
proto-proto-1 | [0.2566, 0.6068],
proto-proto-1 | [0.1951, 0.8978],
proto-proto-1 | [0.2659, 0.9209],
proto-proto-1 | [0.4812, 0.1051],
proto-proto-1 | [0.5955, 0.1450],
proto-proto-1 | [0.5640, 0.8772],
proto-proto-1 | [0.7453, 0.7244],
proto-proto-1 | [0.8169, 0.6367],
proto-proto-1 | [0.7947, 0.2349],
proto-proto-1 | [0.8322, 0.3041],
proto-proto-1 | [0.8809, 0.3520],
proto-proto-1 | [0.9241, 0.2954],
proto-proto-1 | [0.9427, 0.5432],
proto-proto-1 | [0.9060, 0.6680],
proto-proto-1 | [0.8789, 0.5899]]], device='cuda:0')

[glenn-jocher]

Hey there! 🌟 It looks like the keypoints are indeed present but the confidence associated with each keypoint isn't being returned directly in your output. This behavior depends on how the model was trained and how the inference script processes the outputs.

For YOLOv8l-pose, confidence scores are typically associated with the detection rather than individual keypoints. If you need per-keypoint confidence, consider modifying the model's architecture or post-processing script to include this information.

As for the keypoints, x and y in your output represent the normalized positions of each keypoint detected by your model. This is standard for pose estimation tasks, where the main goal is to locate key positional landmarks.

If your application requires confidence scores for each keypoint, you may explore customizing the post-processing step to calculate these based on the model's predictions or the detection's overall confidence score.

For now, you can work with the provided keypoints as they give you the essential information about the position of each landmark detected in the image. Happy coding! 😊

[PolinKun]

can we model pose estimation for excavator work?

[glenn-jocher]

@PolinKun absolutely! Pose estimation can be utilized for modeling excavator work by identifying and tracking keypoints on the excavator. For instance, keypoints could represent the excavator's arm joints, bucket, and other significant parts to analyze its movements and work dynamics.

Here's a quick start on doing so with a YOLOv8n-pose model:

from ultralytics import YOLO

# Load a pretrained pose estimation model
model = YOLO('yolov8n-pose.pt')

# Predict on an image
results = model('path/to/excavator/image.jpg')

# Display the keypoints
results.show()

This example loads a pretrained YOLOv8n-pose model and runs it on an excavator image to detect keypoints. You might want to train the model on a dataset specific to excavators for optimal results. Training on the excavator-specific dataset would involve collecting images of excavators, annotating them with keypoints, and following the training process as detailed in our docs.

Feel free to explore the Pose Estimation section for more insights and guidance on custom training for your specific needs. Happy modeling! 😊

[shukurullo2004]

hello, i hope you can help i am struggled to create logic human falling based on shoulder and knee distance if shoulder is below than knee that means human is fallen, but i can not get keypoints 😭of them and calculate distance, becouse it is first time to using yolov pose estimation. thank you

[shukurullo2004]

another question: can i find velocity of shoulder and knees using keypoints ?
if yes, how can i ?

[pderrenger]

Hello! 😊 Yes, you can calculate the velocity of shoulders and knees using keypoints from pose estimation. Velocity is essentially the change in position over time. If you have keypoints for shoulders and knees detected over sequential frames, you can calculate the distance they move between frames and divide by the time elapsed to get their velocity.

Here's a simple example on how you might approach this:

from ultralytics import YOLO

# Load your pose model
model = YOLO('yolov8n-pose.pt')

# Assume frame1 and frame2 are consecutive frames from your video
results_frame1 = model(frame1)
results_frame2 = model(frame2)

# Extract keypoint locations for shoulders and knees for both frames
# For demonstration, let's say the IDs for shoulders and knees are 2 and 3 respectively
shoulder_frame1 = results_frame1.keypoints[2]
knee_frame1 = results_frame1.keypoints[3]
shoulder_frame2 = results_frame2.keypoints[2]
knee_frame2 = results_frame2.keypoints[3]

# Calculate the distance moved by each keypoint (simple Euclidean distance for demonstration)
distance_shoulder = ((shoulder_frame2[0] - shoulder_frame1[0])**2 + (shoulder_frame2[1] - shoulder_frame1[1])**2)**0.5
distance_knee = ((knee_frame2[0] - knee_frame1[0])**2 + (knee_frame2[1] - knee_frame1[1])**2)**0.5

# Assuming you know the time elapsed between the two frames (time_elapsed), calculate velocity
velocity_shoulder = distance_shoulder / time_elapsed
velocity_knee = distance_knee / time_elapsed

Please adjust the keypoint IDs according to the specific model outputs and ensure the correct handling of the coordinate system used in your data. The velocity calculation here is a basic illustration. Depending on your exact needs, especially if you're dealing with video data, you might need a more robust approach to handle varying frame rates and more complex movement patterns. Good luck with your project!

[shukurullo2004]

thank you helping me, with your help i can detect delta distance but how can i measure time ?

i am doing project human fall detection with using yolov8-pose.pt model, so i can find keypoints but logical part is being difficult, my logic is that comparing velocity of shoulder if shoulder velocity > knee velocity AND if position of shoulder below than knee points that means human is fallen.

so can you help me write this logic

thank you a lot

[pderrenger]

@shukurullo2004 hey there! 😃 I'm glad to hear you've made progress with keypoint detection using the YOLOv8-pose model. For measuring the time, especially in videos, you can use the frame rate (FPS) of your video. If you're processing frame by frame, the time between frames would be 1/FPS. Here's a simplified way to include it in your logic:

fps = 30  # Example FPS of your video
time_elapsed = 1 / fps  # Time between each frame

# Your logic for comparing shoulder and knee velocities
# Assuming you already calculated velocities velocity_shoulder and velocity_knee
if (velocity_shoulder > velocity_knee) and (shoulder_position < knee_position):
    print("Human has fallen")

Remember to adjust fps according to your video's frame rate. This way, you can incorporate time measurement into your logic seamlessly. Best of luck with your project on human fall detection! Feel free to share any further questions. 🚀

[mihkriss]

Hello, I want to ask you for help
My task is to determine in the video whether there is a fight or not, using a model based on key points,
Could you tell me how to implement this using yolov8-pose ^_^

[pderrenger]

Hello there! 😄 It's awesome that you're looking to use YOLOv8-pose for detecting fights in videos based on key points. Here's a simple way to get started:

First, make sure you've installed the Ultralytics YOLOv8 package. If not, you can do so using pip:

pip install ultralytics

Next, you’ll want to use a YOLOv8 pose estimation model. Since your task is quite specific, I’d recommend starting with our pretrained yolov8n-pose.pt model to detect human poses and then apply your logic to determine if a fight is happening based on the poses/keypoints patterns observed.

Here’s how you can run predictions on a video file:

from ultralytics import YOLO

# Load a pretrained pose model
model = YOLO('yolov8n-pose.pt')

# Run predictions on a video file
results = model.predict('path/to/your/video.mp4', stream=True)

# Process your results here to analyze key points and infer fighting scenarios

You might need to develop a custom algorithm that analyzes the keypoints for each person detected in the video frames, looking for patterns that might indicate a fight (like proximity of people, rapid changes in pose, etc.).

Please note, identifying fights based on motion and pose in videos can be complex and may require further fine-tuning or training the model on a dataset specific to fighting scenarios if the results aren’t satisfactory.

Good luck with your project, and feel free to reach out if you have more questions! 👍

[mihkriss]

Do I understand correctly that I need to first determine the key points (their coordinates)? How to do this with video? and how will they be identified if there are several people there?

[pderrenger]

@mihkriss hey there! Yes, you've got the basic idea right. 😊 To work with videos and detect key points for multiple people, here's a streamlined approach using YOLOv8-pose models, like yolov8n-pose.pt:

1. Model Prediction on Video Frames: Pass each frame of your video to the model. It will automatically detect people and their keypoints.

   from ultralytics import YOLO
   
   # Load a pretrained model
   model = YOLO('yolov8n-pose.pt')
   
   # Predict on a video
   results = model.predict('path/to/your/video.mp4', stream=True)

2. Handling Multiple People: The model outputs keypoints for all detected people in a frame. Each person will have their own set of keypoints.

3. Iterating Through Results: Use a loop to iterate through each result's keypoints. You can differentiate between people by their unique set of keypoints.

Here's an example of accessing keypoints for further processing:

for result in results:
    for person in result.keypoints:
        keypoints = person.xy  # Access X, Y coordinates
        # Your logic to analyze keypoints here

YOLOv8's pose estimation models are pretrained on the COCO keypoints dataset and are optimized to handle multiple people in a scene efficiently. They provide the keypoints' locations along with confidence scores, making it easier to analyze complex scenes. 🚀

Hope this helps! Feel free to reach out if you have more questions.

[mihkriss]

Hello! I have a task to detect fights by key points, I extracted points from the dataset to train the model to detect a fight. I have several questions:

1. In what format should points be stored so that they can be trained later?
2. And once I save the points in the desired format, how can I train the model to determine the fighting pose?
3. And in the final output I need to display only a frame with a label on the video that there is a fight going on here
   Please help me deal with these questions :(

[glenn-jocher]

Hello! It sounds like an interesting project you're working on. Let's address your questions to help get you started with detecting fights using key points:

1. Point Format: The keypoints should be stored in a format that represents the x, y coordinates (and possibly the visibility) of each keypoint. For YOLOv8 pose estimation, keypoints are typically encoded in a list or array per object instance, each element holding the [x, y, visible] information.

2. Training for Fighting Pose: With your points appropriately formatted and saved, you can train a model like yolov8n-pose to recognize fighting poses. Start by organizing your data according to the YOLO pose dataset format, then train the model using command:

   yolo pose train data=your_dataset.yaml model=yolov8n-pose.pt epochs=100 imgsz=640

   Make sure your your_dataset.yaml points to your training and validation data, configured similarly to the coco-pose.yaml.

3. Displaying Frames with a Label: For the output, after prediction, you can process the results to check if a fighting pose is detected and then annotate the frame with a label indicating the fight. Here's a snippet:

   from ultralytics import YOLO
   model = YOLO('path/to/your_trained_model.pt')
   results = model('path/to/video_or_image')
   if 'fight' in results.pandas().xyxy[0]['name'].values:  # assuming 'fight' is a class name
       # This code assumes you're working with frames;
       # you can annotate the frame here to indicate a fight is detected.

I hope this helps clarify your workflow! If you have any more questions or need further assistance, don't hesitate to ask. Happy coding! 😊

[mihkriss]

Why when withdrawing
result.boxes.xyxy
displays
tensor([[ 85., 335., 755., 1521.]])
which does not correspond to the format from 0 to 1

[glenn-jocher]

Hey there! When you get result.boxes.xyxy, it actually returns the bounding boxes in pixel coordinates, not normalized ones. 😊 If you're looking to work with coordinates normalized between 0 and 1, you can do so by dividing by the image dimensions. Here's a quick example:

# Assuming 'result' is your prediction output and 'img' is your input image.
normalized_boxes = result.boxes.xyxy / torch.tensor([img.shape[1], img.shape[0], img.shape[1], img.shape[0]]).to(result.boxes.device)

This will give you the coordinates in the format you're looking for!

For pose estimation using YOLOv8, keypoints are usually the focus, but knowing how to handle bounding boxes is definitely useful too. If there's anything else you'd like to dive into, feel free to ask! 🌟

[gustavoortiz7]

hola buenas
estoy haciendo un proyecto de correción deportiva , remates de voley , y bueno para poder lograr esto intento usar yolov8-pose para , poder obtener los puntos clases de la persona y luego convertirlos en secuencias para una lstm , quería saber cómo puedo obtener los puntos claves pero solo de la persona que hace el remate , por que yolo de detecta a todas las personas que ve en el frame y los se quiere hacer es que en ves de mandarle un vídeo , mandarle una dirección para así poder leer todos los fotogramas de distintas acciones , bueno en este caso solo son dos (rematebueno y remate malo) no trabajo con video por que , saca muchos fotogramas repetidas aunque al final mi proyecto debe detectar en tiempo real el error que cometió , agradecería mucho si puedes proporcionarme como convertir los puntos clases en secuencias y si tiene algún concejos de cómo abordar el problema de mejor manera gracias

[pderrenger]

¡Hola! Para focalizar la detección de puntos clave únicamente en la persona que realiza el remate en vóley, una estrategia sería primero identificar a esa persona específica usando algún criterio (como la posición en el marco) y luego ejecutar la estimación de pose solamente en esa persona detectada.

Aquí un breve ejemplo de cómo podrías hacerlo:

from ultralytics import YOLO

# Carga tu modelo de estimación de pose
model = YOLO('yolov8n-pose.pt')

# Para procesar cada imagen de tu directorio
for imagen in tus_imagenes:
    # Realiza detecciones
    results = model(imagen)
    
    # De alguna manera determina cuál de los detectados es el jugador haciendo el remate
    # Esto podría implicar filtrar por ubicación si siempre rematan desde una posición similar
    jugador_remate = filtrar_jugador(results)
    
    if jugador_remate:
        # Convierte los puntos clave (keypoints) del jugador a secuencias para tu LSTM
        keypoints = jugador_remate.keypoints
        secuencia = convertir_a_secuencia(keypoints)
        # Usa secuencia para tu LSTM...

Este código es solo un esqueleto para darte una idea. La función filtrar_jugador sería personalizada para identificar al jugador correcto, y convertir_a_secuencia dependerá de cómo requieras estructurar los datos para tu LSTM.

Recuerda, el rendimiento efectivo dependerá mucho de la precisión de tu modelo YOLOv8-pose y cómo de bien puedes filtrar al jugador que realiza el remate. Podría serte útil recortar previamente la región de interés (en donde el jugador lleva a cabo el remate) antes de pasarle la imagen al modelo de pose.

Para leer los fotogramas desde una dirección específica, puedes usar algún método para iterar sobre los archivos en ese directorio. Python y las bibliotecas de manejo de archivos te permitirán hacer esto de manera bastante sencilla.

Finalmente, para el análisis en tiempo real, después de haber entrenado tu modelo LSTM con suficientes datos, podrás clasificar los remates en tiempo real procesando individualmente los frames de vídeo capturado.

Espero que esto te ayude en tu proyecto de corrección deportiva. ¡La combinación de YOLOv8-pose y una LSTM suena prometedora!🚀

[Bekbo01]

Hi there! Is it possible to train pose model for 2 or more classes? If it possible can you share with examples?

[pderrenger]

@Bekbo01 hey there! 🌟 Yes, you can have a .yaml configuration for training models with two or more classes. Here's a simplified example for a pose estimation task with two classes:

# custom-pose-dataset.yaml
train: /datasets/coco128/images/train2017/  # path to training images
val: /datasets/coco128/images/val2017/      # path to validation images

nc: 2  # number of classes
names: ['Class1', 'Class2']  # class names

# Define keypoints for each class
keypoints:
  - [nose, left_eye, right_eye, left_ear, right_ear, left_shoulder, right_shoulder, left_elbow, right_elbow, left_wrist, right_wrist, left_hip, right_hip, left_knee, right_knee, left_ankle, right_ankle]  # Class1 keypoints
  - [point1, point2, point3, point4]  # Class2 keypoints
  
# Example for specific keypoints count per class if different
num_keypoints:
  - 17  # Number of keypoints for Class1
  - 4   # Number of keypoints for Class2

Make sure the keypoints and num_keypoints sections reflect the actual structure of your dataset for each class. You'll need to customize paths and names accordingly. This configuration allows you to define different sets of keypoints for each class, which is especially useful in diverse datasets where different objects require different keypoints for accurate pose estimation. 🚀

[Bekbo01]

hey there! In wich version of yolo it works? I got error KeyError: 'kpt_shape' (if add just kpt_shape it will work inly for one class)

[pderrenger]

Hey there! 👋 It seems you encountered a KeyError: 'kpt_shape' because the kpt_shape parameter might be missing or improperly set for multiple classes in your dataset configuration. For pose estimation with multiple classes in YOLOv8, you should define num_keypoints for each class within your .yaml dataset configuration file, not just kpt_shape.

Here's a quick fix using an example .yaml config tailored for two classes:

# Your custom-pose-dataset.yaml
train: /path/to/train/images
val: /path/to/val/images

nc: 2  # number of classes
names: ['Class1', 'Class2']  # Class names

# Define keypoints for each class
num_keypoints:
  - 17  # Number of keypoints for Class1
  - 5   # Number of keypoints for Class2

Make sure to adjust the train, val paths, names, and num_keypoints according to your specific dataset and classes. This configuration will help you avoid the KeyError and work seamlessly with multiple classes. Hope this helps! Let me know if you have more questions! 🚀

[lilin19890401]

When we training multiple classes and the .yaml as you mentioned above，two classes with different number of keypoints. How do we modify nc and kpt_shape in yolov8n-pose.yaml ?

[pderrenger]

@lilin19890401 hello! For training a YOLOv8 pose model with multiple classes where each class has a different number of keypoints, you don't need to modify nc or kpt_shape directly in the yolov8n-pose.yaml. Instead, you should specify num_keypoints for each class in your dataset configuration YAML file. Here’s how you can structure it:

nc: 2  # Number of classes
names: ['Class1', 'Class2']  # Class names

# Specify keypoints for each class
num_keypoints:
  - 17  # Number of keypoints for Class1
  - 5   # Number of keypoints for Class2

This setup tells the model how many keypoints to expect for each class, allowing for different configurations per class. Make sure your dataset labels match this configuration! Let me know if you need further assistance. 😊

[maryaliza]

Hello,
I am trying to run the pose estimation on only certain detected objects. I am tracking a multi object video with custom detection model of yolov8 and I am aiming to run the pose estimation whenever a person is detected to classify them as walking and sitting person. I'm wondering if you can point me to the right direction.
I have tried extracting the label of the detected models from the results object but the results.keypoints showing none even though I see the results on the output video.

I was wondering if I should run the prediction model separately and then pose estimation whenever there is a person detected and then (label if someone is standing or walking) and then track model with modified labels?

[glenn-jocher]

@maryaliza hi there! To achieve your goal of running pose estimation on detected persons and classifying their actions, you can indeed run the detection and pose estimation models sequentially. First, use your custom YOLOv8 detection model to identify persons. When a person is detected, you can then run the pose estimation model to get the keypoints and classify the action (walking or sitting).

Regarding the issue with results.keypoints showing None, ensure you're using the latest version of the Ultralytics package. If the problem persists, you might need to check if the pose model is correctly loaded and applied.

Here's a brief workflow:

1. Run your detection model to identify persons.
2. For each detected person, crop the image around the bounding box.
3. Run the pose estimation model on the cropped image.
4. Use the keypoints to classify the action.

This approach should help you achieve accurate and efficient results. For more detailed guidance, you can refer to the Pose Estimation documentation. Good luck!

[Smith-jeff]

Hello，I'm trying to detect three objects include human face, cigarette and phone with yolov8-pose because we need 68 keypoints only in the faces;
I would like to know how to handle the other two types of undetected key points in the annotation file. Can you provide a reference? We look forward to your reply!

[pderrenger]

To handle the undetected keypoints for the cigarette and phone in your annotation file, you can set their coordinates to a default value, such as [0, 0, 0], indicating they are not present. This ensures the model focuses on the 68 keypoints for the face while ignoring the other objects. For more detailed information on dataset formatting, please refer to the Dataset Guide on our documentation page.

[DzanielCraig]

How can I get parameter "visible" for every keypoint in the results, if has_visible = True?

[glenn-jocher]

@DzanielCraig to get the "visible" parameter for each keypoint in the results, you need to ensure that your model and dataset support visibility flags. If has_visible=True in your dataset configuration, the model should output keypoints with visibility information. You can access this visibility data in the results object after running predictions. If you encounter any issues, please ensure you are using the latest version of the Ultralytics package and that your dataset is correctly formatted. For more details, refer to the Pose Estimation documentation.

[DzanielCraig]

Is confidence means visible?As far as I know, confidence indicates whether the point is valid but visibility indicates whether the point is covered in the picture. For instance, a point is not visible in the picture, but it is a valid keypoint, should it have a low confidence?

[glenn-jocher]

Confidence and visibility are distinct concepts in pose estimation. Confidence indicates the model's certainty about the keypoint's location, while visibility denotes whether the keypoint is visible in the image. A keypoint can have high confidence but be marked as not visible if it is occluded or outside the image frame. If has_visible=True in your dataset, the model should output visibility flags alongside confidence scores. Ensure your dataset and model are correctly configured to include visibility information. For more details, refer to the Pose Estimation documentation.

[1-zym]

WARNING ⚠️ Upgrade to torch>=2.0.0 for deterministic training.这个问题，对我的实验可复现性有影响？

[glenn-jocher]

@1-zym upgrading to torch>=2.0.0 is recommended for deterministic training, which can enhance the reproducibility of your experiments. If you encounter issues, please ensure you are using the latest versions of all relevant packages. For more details on pose estimation with YOLOv8, you can visit Ultralytics Pose Estimation.

[Anandukc]

I just want to know can pose estimation can be use to identify focused and non focused students in a classroom?

[pderrenger]

@Anandukc yes, pose estimation can be used to identify focused and non-focused students in a classroom. By analyzing keypoints such as head and eye positions, you can infer where students are looking and their overall posture. This data can help determine if a student is paying attention or distracted. For more details on how to implement this, you can refer to our pose estimation documentation.

[xhf3571]

Hi,

I am currently using yolov8n-pose.pt to validate on the COCO test-dev2017 dataset. I ran the following code:

```python
from ultralytics import YOLO

if __name__ == '__main__':
    # Load the model
    model = YOLO(r'yolov8n-pose.pt')
    # Validate the model
    metrics = model.val(
        data=r'D:\ultralytics-8.2.66\ultralytics\cfg\datasets\coco-pose.yaml',
        val=True,
        split='test',
        save_json=True,
        save_hybrid=False,
        conf=0.25
    )

This generated predictions.json, which I renamed to person_keypoints_test-dev2017_YOLOV8n-pose_results as required, and submitted to the COCO Keypoints evaluation website here. However, despite multiple attempts, my submission always scores 0.

Main Issue

The predictions.json generated by my val.py script contains the following keys: image_id, category_id, bbox, keypoints, score. In contrast, the COCO train and validation annotations (person_keypoints_train2017.json, person_keypoints_val2017.json) contain these additional keys: image_id, category_id, bbox, keypoints, segmentation, num_keypoints, area, iscrowd, id.The additional keys is：segmentation, num_keypoints, area, iscrowd,

Questions

1. Is the reason for my 0 score the missing keys in my predictions.json?（And I have tried using the PREDICT to generate JSON, but it has always failed）
2. If so, how can I correctly generate predictions.json with all the necessary keys?
3. If the missing keys are not the issue, what could be the reason for my 0 score and how can I resolve it?

Sample of my predictions.json

Here is a sample of my predictions.json:

[
    {
        "image_id": 255037,
        "category_id": 0,
        "bbox": [385.333, 60.101, 30.459, 48.111],
        "keypoints": [402.19720458984375, 59.434791564941406, 0.022939346730709076, ...],
        "score": 0.14029
    },
    {
        "image_id": 255037,
        "category_id": 0,
        "bbox": [400.003, 61.648, 18.764, 46.545],
        "keypoints": [405.827880859375, 59.67790222167969, 0.12535953521728516, ...],
        "score": 0.01377
    },
    {
        "image_id": 255037,
        "category_id": 0,
        "bbox": [385.287, 60.082, 18.392, 46.91],
        "keypoints": [397.5641174316406, 60.115478515625, 0.1089448630809784, ...],
        "score": 0.0131
    },
    {
        "image_id": 255037,
        "category_id": 0,
        "bbox": [412.931, 60.076, 14.57, 44.361],
        "keypoints": [413.3561706542969, 60.88957977294922, 0.14515842497348785, ...],
        "score": 0.00446
    }
    ...
]

I would appreciate any guidance on resolving this issue.

Thank you!

[pderrenger]

@xhf3571 thank you for your detailed explanation. The missing keys in your predictions.json are not the issue. The COCO evaluation server does not require segmentation, num_keypoints, area, iscrowd, or id for predictions. The problem might be related to the format or content of your predictions.json. Ensure that your category_id is set correctly (for COCO, it should be 1 for person). Also, verify that the keypoints are in the correct format and order. If the issue persists, please try updating to the latest version of the Ultralytics package and re-running your validation. For further details, you can refer to the Pose Estimation documentation.

[xhf3571]

I validated YOLOv8n-pose on the COCO evaluation website https://codalab.lisn.upsaclay.fr/competitions/7403#participate-submit_results. Why is the mAP50 0.779 and mAP50-95 0.485, which are lower than those published on the YOLOv8 official website?

[glenn-jocher]

@xhf3571 the discrepancy in mAP scores could be due to several factors, including differences in evaluation metrics, dataset splits, or preprocessing steps. Ensure you are using the latest version of the YOLOv8 package and that your evaluation setup matches the one used for the official benchmarks. If the issue persists, please provide more details about your evaluation process for further assistance. For more information on pose estimation, you can visit our Pose Estimation documentation.

[xhf3571]

Why is it that when I set the batch size to 128 and 64, the only difference is the increase in GPU memory usage, which is 22G and 10G respectively, but the training speed does not increase?

[pderrenger]

@xhf3571 the increase in GPU memory usage with larger batch sizes is expected, but the training speed may not always scale linearly due to various factors such as data loading bottlenecks, GPU utilization, and model architecture. Ensure you're using the latest version of the Ultralytics package and consider profiling your data pipeline to identify any potential bottlenecks. For more detailed insights, you can refer to the Pose Estimation documentation.

[xhf3571]

When my training is interrupted, I can resume it from where it left off by using resume=True, and the mAP will continue to increase from the interruption point. However, after a complete training session, if I use last.pt to train a new session, such as 300 epochs, the mAP does not start from the mAP of the last epoch of the previous training, but rather from 0, as if it is a completely new training session. If I want the mAP to continue growing from the last epoch of the previous training, what should I do?

[glenn-jocher]

@xhf3571 to continue improving mAP from the last epoch of a previous training session, you should use the resume=True argument instead of starting a new training session with last.pt. This ensures that the optimizer state and learning rate scheduler are also restored, allowing the model to continue training seamlessly from where it left off. If you start a new session with last.pt, it initializes a fresh training process, which is why the mAP starts from scratch.

[xhf3571]

Perhaps I wasn't clear enough, which led to a misunderstanding. For example, if I complete training for 300 epochs and I want to continue training from epoch 301 to 600, I can't use resume=True because it will give an error indicating there is no unfinished training. However, if I use the last.pt from the first 300 epochs to train the second 300 epochs, the mAP starts from zero. So, how can I continue training from epoch 301 to 600 without starting mAP from zero?

[pderrenger]

@xhf3571 to continue training from epoch 301 to 600 without resetting the mAP, you should load the last checkpoint from your initial training and start a new training session using that checkpoint. Ensure you specify the correct path to your last.pt file and set the epochs argument to the total number of epochs you want (600 in this case). This way, the training will continue from where it left off, preserving the mAP and other metrics. If you encounter any issues, please ensure you are using the latest version of the Ultralytics package.

[xhf3571]

After running the following code:

from ultralytics import YOLO

# Load a model
model = YOLO("yolov8n-pose.yaml")  # build a new model from YAML

# Train the model
results = model.train(data="my-pose.yaml", epochs=300, imgsz=640)

I found that the model hasn't converged yet. I need to continue training. Based on your suggestion, should I proceed with the following?

from ultralytics import YOLO

# Load a model
model = YOLO("/path/to/my/last.pt")  # load from the last checkpoint

# Train the model
results = model.train(data="my-pose.yaml", epochs=600, imgsz=640)

Please let me know if this approach is correct.

Thank you!

[pderrenger]

Your approach is correct. To continue training from epoch 301 to 600, you should load the last checkpoint and set the total epochs to 600. This way, the training will resume from where it left off, and the mAP will not reset. Ensure you use the latest version of the Ultralytics package to avoid any issues. If you encounter any problems, please let us know.

[xhf3571]

After running the code, I encountered the error 'nothing to resume'. However, if I set resume=False, the mAP doesn't start increasing from the last epoch of my first training but instead starts from a low value. In my scenario, after completing 300 epochs of training for the first time, I found that the model hadn't converged, so I thought 'it should continue training for another 400 epochs'. How should I modify my train.py?

[pderrenger]

@xhf3571 to continue training from epoch 301 to 600 without resetting the mAP, load the last checkpoint from your initial training and set the total epochs to 600. Here's how you can do it:

from ultralytics import YOLO

# Load the last checkpoint
model = YOLO("/path/to/last.pt")

# Continue training
results = model.train(data="my-pose.yaml", epochs=600, imgsz=640)

This will ensure the training resumes from where it left off, preserving the mAP and other metrics. If you encounter any issues, please ensure you are using the latest version of the Ultralytics package.

[Liuhangji]

How to output 3D coordinates after human pose estimation

[pderrenger]

@Liuhangji to output 3D coordinates after human pose estimation, you need to extend the 2D keypoints to include depth information. YOLOv8-pose models provide 2D coordinates by default. For 3D coordinates, you would typically need additional depth data from a stereo camera setup or a depth sensor. You can then combine this depth data with the 2D keypoints to obtain 3D coordinates. If you have further questions or need more details, please refer to our Pose Estimation documentation.

[rchlrq]

I followed the steps in the YouTube video about YOLOv8 pose estimation, exactly as said. I tested an image of a person as the input; however, it detects 300+ people in that picture. It also detects 300+ people in videos I input (1 person) and my webcam. Does anyone know how to resolve this issue? Thank you!

[glenn-jocher]

@rchlrq it sounds like the model might be misinterpreting the input data. First, ensure you're using the latest version of the Ultralytics package. If the issue persists, try adjusting the confidence threshold to filter out low-confidence detections. You can do this by setting a higher conf value in the prediction command. Additionally, verify that the input image size matches the model's expected input size. If these steps don't resolve the issue, please provide more details about your setup for further assistance.

[rchlrq]

I tried both suggestions by adjusting the code to this:

from ultralytics import YOLO 

model = YOLO('yolov8m-pose.pt')
results = model(source='person.jpg', show=True, conf=0.99, save=False, imgsz = 640)

It still displays that it detected 300 people when the image only has 1 person. Here is the output:

image 1/1 person.jpg: 640x640 300 persons, 2059.0ms Speed: 3.0ms preprocess, 2059.0ms inference, 4.0ms postprocess per image at shape (1, 3, 640, 640)

Is there anything else I can try to solve this issue?

[glenn-jocher]

Thank you for trying the suggestions. Since the issue persists, please ensure you are using the latest version of the Ultralytics package. If the problem continues, it might be helpful to check if the model weights are correctly loaded and not corrupted. Additionally, you could try using a different model size, such as yolov8s-pose.pt, to see if the issue is specific to the yolov8m-pose.pt model. If none of these steps resolve the issue, please provide more details about your environment and setup for further assistance.

[rchlrq]

I have tried yolov8m, yolov8n, and yolov8x models, and all of them give the same results (results posted below).

I am new to AI and DL, so I am using the exact same code from the video other than the confidence threshold:

from ultralytics import YOLO 

model = YOLO('yolov8x-pose.pt')
results = model(source='side.jpg', show=True, conf=0.99, save=False)

I used pip install ultralytics for installation, and the version of ultralytics is 8.2.71.

[glenn-jocher]

@rchlrq it seems like the model might be misinterpreting the input data. First, ensure you're using the latest version of the Ultralytics package. If the issue persists, try adjusting the confidence threshold to filter out low-confidence detections. You can do this by setting a higher conf value in the prediction command. Additionally, verify that the input image size matches the model's expected input size. If these steps don't resolve the issue, please provide more details about your setup for further assistance.

[Version1221]

Hi I would like to ask you regarding annotation for Pose Estimation. using the model I am easily able to predict and get the pose and confidence but for the Ground truth annotation, I know roboflow has Keypoint detection where we can do annotation. I am just confused how shall I annotate this data as I have around 60 images and each pedestrian has a different pose and to predefine one pose and work on it I am finding it difficult to do. Could you please explain how I can proceed with it.

Once the GT is done apart from confidence value, IOU and visual inspection of the pose is there any other way to validate the performance of the detection. I would like to have a in depth knowledge regarding this.

My 3rd question is regarding Auto annotation which roboflow offers I didn't find that option in Keypoint detection.

Looking forward to hear from you :)
Thanks

[glenn-jocher]

@Version1221 for annotating pose estimation data, you can use tools like Roboflow for keypoint detection. Ensure each keypoint is accurately labeled for each pedestrian. For validation, apart from confidence, IoU, and visual inspection, you can use metrics like PCK (Percentage of Correct Keypoints) and AUC (Area Under Curve) for a more comprehensive evaluation. Regarding auto-annotation, Roboflow's current offerings might not include keypoint detection, so manual annotation might be necessary. For more details, visit Ultralytics Pose Estimation Documentation.

[Version1221]

Hi Glenn, thankyou for your reply
I will use other metrics you told me in my use case.

I have a question regarding data we get from yolo model and groud truth from key point estimation using roboflow. I am not able to develop a sync where my GT value is similar to what I get from yolo model. For now I am able to get a csv with Bounding box and Confidence Value, but the GT values hold the keypoints (17 pre-defined points x & y coordinates)

Could you please help me with this constrain how I can perform IOU or other metrics, and what changes shall I bring to my code to get similar coordinates as I get from Ground Truth.

Hoping to get a positive feedback

[glenn-jocher]

@Version1221 to align your ground truth (GT) keypoints with the YOLO model predictions, ensure both datasets follow the same keypoint format and order. You can use the JSON2YOLO tool to convert your annotations to the YOLO format. For evaluating performance, consider using metrics like PCK (Percentage of Correct Keypoints) and AUC (Area Under Curve) in addition to IoU and confidence scores. Unfortunately, Roboflow's auto-annotation might not support keypoint detection, so manual annotation may be necessary. For more details, visit the Ultralytics Pose Estimation Documentation.

[actionTurtle]

Is there a way to use or processes with stereo 3D footage? Is that coming maybe? Seems like it would really help out. Thanks!

[glenn-jocher]

@actionTurtle currently, YOLOv8 does not support stereo 3D footage directly. However, you can process each frame separately and combine the results. For updates on new features, keep an eye on our GitHub releases.

[CH-Echo]

Many people have asked how I can use batch inference to improve performance

[glenn-jocher]

@CH-Echo to improve performance with batch inference, you can process multiple images simultaneously by setting the batch parameter in your inference call. This allows the model to handle several inputs at once, optimizing resource usage and reducing overall processing time.

[CH-Echo]

@CH-Echo为了提高批量推理的性能，您可以通过batch在推理调用中设置参数来同时处理多张图像。这允许模型一次处理多个输入，从而优化资源使用率并减少总体处理时间。

How to do it specifically? Can you give me a demo? Because yolo inputs a picture, but if there are multiple people in a picture, how to implement multi-person batch processing? Yes, there is another question. I have thought about using multi-threaded inference, but how to do it on the GPU? Implementation because when I look at inference the GPU usage is only 30%

[glenn-jocher]

@CH-Echo to implement batch inference, you can use the batch parameter in your inference call to process multiple images simultaneously. For multi-person detection within a single image, the model will automatically detect multiple keypoints. For multi-threaded inference on GPU, ensure each thread has its own model instance to maximize GPU utilization. For more details, please refer to the documentation on pose estimation.