RT-DETR (Realtime Detection Transformer) - Ultralytics YOLOv8 Docs

[giscus[bot]]

RT-DETR (Realtime Detection Transformer) - Ultralytics YOLOv8 Docs

Explore RT-DETR, a high-performance real-time object detector. Learn how to use pre-trained models with Ultralytics Python API for various tasks.

https://docs.ultralytics.com/models/rtdetr/

Tips for ValueError: matrix contains invalid numeric entries while training.

Why is this happening?

This is because there are NaN values produced in training process. The rtdetr training with amp=True is more sensitive than yolo models to custom datasets, probably it's related to the transformer architecture and nms-free design(we might figure out a better solution for this in the future).

Suggestion when you encounter this issue.

• Try lower learning rate.
• Try training with pretrained rt-detr weights instead of training from scratch.
• Try setting amp=False manually. (we set amp=True by default just like YOLO models to save cuda memory and speed up training.

[KainingYing]

Hi, How can I download this pretrained checkpoint file?

[AyushExel]

It will be auto downloaded when you use it for the first time

[eorroot]

Hope to support training mode soon!

[AyushExel]

We're working on it

[glenn-jocher]

We appreciate your enthusiasm! The team is actively working on enhancing the training mode for RT-DETR. Stay tuned for updates! 🚀

[turtoe]

I wonder will there be any guidance of exporting onnx model then convert to TensorRT? I tried to do so but failed.
What i tried to do is to convert the model into fp16. Thanks for the info~

ONNX with given opset: onnx 1.13.1 opset 17
pytorch version: 2.0.0+cu117

with code "model.export(device=0,format="tensorrt")"
With the following errors:

1. ONNX: simplifying with onnxsim 0.4.28...
   ONNX: simplifier failure: [ONNXRuntimeError] : 1 : FAIL : Type Error: Type parameter (T) of Optype (Where) bound to different types (tensor(float) and tensor(double) in node (/model.28/Where).

2. [05/23/2023-13:54:51] [TRT] [E] [selectLayer.cpp::inferOutputType::73] Error Code 2: Internal Error (Assertion builder::implicitlyConvertibleTypes(inputForms[1].type, inputForms[2].type) failed. )
   [05/23/2023-13:54:51] [TRT] [E] ModelImporter.cpp:771: While parsing node number 648 [Where -> "/model.28/Where_1_output_0"]:
   [05/23/2023-13:54:51] [TRT] [E] ModelImporter.cpp:772: --- Begin node ---
   [05/23/2023-13:54:51] [TRT] [E] ModelImporter.cpp:773: input: "/model.28/Cast_8_output_0"
   input: "/model.28/Concat_3_output_0"
   input: "/model.11/Constant_9_output_0"
   output: "/model.28/Where_1_output_0"
   name: "/model.28/Where_1"
   op_type: "Where"

[05/23/2023-13:54:51] [TRT] [E] ModelImporter.cpp:774: --- End node ---
[05/23/2023-13:54:51] [TRT] [E] ModelImporter.cpp:777: ERROR: ModelImporter.cpp:195 In function parseGraph:
[6] Invalid Node - /model.28/Where_1
[selectLayer.cpp::inferOutputType::73] Error Code 2: Internal Error (Assertion builder::implicitlyConvertibleTypes(inputForms[1].type, inputForms[2].type) failed. )

[Tigerold]

reported the same issure. any suggestion,please

---

[05/30/2023-14:20:03] [W] [TRT] onnx2trt_utils.cpp:374: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[05/30/2023-14:20:03] [W] [TRT] builtin_op_importers.cpp:4877: TensorRT is using FLOAT32 precision to run an INT32 TopK. Rounding errors may occur for large integer values
[05/30/2023-14:20:03] [W] [TRT] onnx2trt_utils.cpp:400: One or more weights outside the range of INT32 was clamped
[05/30/2023-14:20:03] [W] [TRT] ISelectLayer with inputs /input_layer/Cast_15_output_0 and (Unnamed Layer* 395) [Shuffle]_output and (Unnamed Layer* 394) [Constant]_output: thenInput input has type Int32 but elseInput has type Float.
[05/30/2023-14:20:03] [E] [TRT] [selectLayer.cpp::inferOutputType::73] Error Code 2: Internal Error (Assertion builder::implicitlyConvertibleTypes(inputForms[1].type, inputForms[2].type) failed. )
[05/30/2023-14:20:03] [E] [TRT] ModelImporter.cpp:771: While parsing node number 223 [Where -> "/input_layer/Where_2_output_0"]:
[05/30/2023-14:20:03] [E] [TRT] ModelImporter.cpp:772: --- Begin node ---
[05/30/2023-14:20:03] [E] [TRT] ModelImporter.cpp:773: input: "/input_layer/Cast_15_output_0"
input: "/input_layer/Reshape_4_output_0"
input: "/input_layer/Constant_78_output_0"
output: "/input_layer/Where_2_output_0"
name: "/input_layer/Where_2"
op_type: "Where"

[05/30/2023-14:20:03] [E] [TRT] ModelImporter.cpp:774: --- End node ---
[05/30/2023-14:20:03] [E] [TRT] ModelImporter.cpp:777: ERROR: ModelImporter.cpp:195 In function parseGraph:
[6] Invalid Node - /input_layer/Where_2
[selectLayer.cpp::inferOutputType::73] Error Code 2: Internal Error (Assertion builder::implicitlyConvertibleTypes(inputForms[1].type, inputForms[2].type) failed. )
Segmentation fault (core dumped)

[pderrenger]

@LinuxCup it seems you're encountering type conversion issues during the ONNX to TensorRT conversion process, particularly with the Where operation and INT64 to INT32 casting. Here are a few suggestions that might help resolve these issues:

1. Ensure Compatibility: Verify that the versions of PyTorch, ONNX, and TensorRT you're using are compatible with each other. Sometimes, using the latest versions of these libraries can introduce incompatibilities.

2. Type Casting: The warning about INT64 weights suggests that there might be a mismatch in the expected data types. You can try to modify the export script to cast these weights to INT32 before exporting to ONNX.

3. Simplify Model: Use the simplify flag during the export to ONNX to remove operations that might not be supported or cause issues in TensorRT.

4. Model Precision: When exporting the model to ONNX and converting to TensorRT, ensure that the precision (e.g., FP32, FP16) is consistent and supported by TensorRT.

5. Update Libraries: Make sure that you are using the latest stable versions of ONNX and TensorRT, as they may contain fixes for bugs present in earlier versions.

6. TensorRT Parser: Sometimes, the issue might be with the TensorRT parser. Look for any updates or patches that might address these issues.

7. ONNX Model Check: Use ONNX tools to check the validity of the ONNX model before attempting to convert it to TensorRT.

8. TensorRT Documentation: Consult the TensorRT documentation for any known issues with the Where operation and possible workarounds.

9. Community and Support: If the issue persists, consider reaching out to the TensorRT community or NVIDIA support for assistance, as they might have more insights into the problem.

10. Fallback to PyTorch: If immediate conversion to TensorRT is not critical, you can consider using the PyTorch model directly with GPU acceleration until the conversion issues are resolved.

Remember to back up your work before making changes, and consider testing each change individually to isolate the cause of the issue. If you continue to face difficulties, please provide detailed information about your environment, versions of the libraries used, and a minimal reproducible example, so the community can assist you further.

[ly510244856]

wait for training mode

[pderrenger]

@ly510244856 to wait for training mode in YOLOv8, you can use the patience argument during training. This argument specifies the number of epochs to wait for improvement in validation metrics before early stopping. Here's a concise example:

from ultralytics import YOLO

# Load a YOLOv8 model
model = YOLO('yolov8n.pt')

# Train the model with patience set to 50 epochs
results = model.train(data='coco128.yaml', epochs=100, patience=50)

In this example, if there's no improvement in validation metrics for 50 epochs, the training will stop early. Adjust the patience value as needed for your training process. 🚀

[Adeelbek]

Hi there!

Just a brief suggestion.
If you face problem while running inference of RT-DETR and see following error:

RT-DETR inference run is giving strange error (datatype error)

Please make sure that your PyTorch version is 2.0 or higher

Cheers

[pderrenger]

@Adeelbek hello!

Absolutely, ensuring that you're running PyTorch version 2.0 or higher is crucial for RT-DETR inference. If you're encountering datatype errors, updating PyTorch could resolve the issue. Let us know if you need further assistance!

Cheers 😊👍

[creativesalam]

Does it have model smaller than Large? Like RT-DETR Medium, RT-DETR Small etc? If pre-trained is not available that is alright but can we do that from scratch?

[Snarci]

If you want to train a model from scratch with a smaller size you can edit the model's yaml.
At the top of the yaml you can remove the following lines:

scales:

[depth, width, max_channels]

l: [1.00, 1.00, 1024]

And add something like this:

nc: 80 # number of classes
depth_multiple: your_depth # model depth multiple
width_multiple: your_width # layer channel multiple

[Snarci]

Sorry, for the wrong styling ;)

scales:
l: [1.00, 1.00, 1024]

And add something like this:

nc: 80 # number of classes
depth_multiple: your_depth # model depth multiple
width_multiple: your_width # layer channel multiple

[lyuwenyu]

We provide scaled-rtdetr models based on resnet-series.

https://github.com/PaddlePaddle/PaddleDetection/tree/develop/configs/rtdetr

[pderrenger]

@lyuwenyu currently, the RT-DETR models provided by Ultralytics are the Large (L) and Extra-Large (X) variants. If you're interested in training a smaller version, such as a Medium or Small RT-DETR, you can indeed train these models from scratch by customizing the model configuration file.

You can adjust the depth_multiple and width_multiple parameters in the model's YAML file to scale down the model size. These parameters control the depth (number of layers) and width (number of channels) of the network, respectively. Reducing these values will result in a smaller model.

For example, you might create a custom YAML file with the following content to define a smaller model:

nc: 80  # number of classes
depth_multiple: 0.33  # model depth multiple (smaller)
width_multiple: 0.50  # layer channel multiple (smaller)

By adjusting these parameters, you can control the size and capacity of the model to suit your requirements. Once you have defined your custom model configuration, you can proceed to train the model using the Ultralytics framework.

For more detailed instructions and the latest updates on available models and configurations, please refer to the official PaddlePaddle PaddleDetection repository and the Ultralytics documentation.

[lyuwenyu]

As the author of RT-DETR, I am very pleased to see this work in ultralytics. Also sharing the news about our latest scaled models (including but not limited to (s, m, l, x). See details.

[AyushExel]

Thanks for the update. This is super coool!

[glenn-jocher]

We're thrilled you're excited about the RT-DETR! 🚀 If you're interested in exploring more about the model and its capabilities, feel free to check out the detailed documentation and usage examples here. Happy detecting! 😊

[Adeelbek]

Hello there,

I was training rt-detr-l model with custom dataset. Initially training process went well till epoch 8.
After epoch 8, I received following error message.

ValueError: matrix contains invalid numeric entries

Is this some bug or did I do something wrong.
My dataset label format is darknet txt.

Thanks for your support~

[prathamgarg911]

hi i am facing the same issue but when i slow down the learning rate and change the optimizer it is resolved but no permanent solution.

[Laughing-q]

@prathamgarg911 the training of RTDETR is more sensitive than YOLO models, this error is because there are NaN values produced while trainign. In my experiments it's also related to training strategies, low the learning rate could also work. Please try amp=False, it'll occupy more cuda memory while training but it should solve the issue for now.

[prathamgarg911]

Okay i ll try it . thankyou so much @Laughing-q!

[prathamgarg911]

hi can you suggest some training strategy , i tried reducing the learning rate and setting the optimizer to AdamW and setting amp =False but still the results are not very good and also changing the learning rate is mandatory as only amp =False is not working and still gives the error.

[glenn-jocher]

@prathamgarg911 it's great to hear that you're experimenting with different strategies to improve your RT-DETR training. Here are a few additional suggestions you might consider:

1. Pretrained Weights: If you haven't already, start your training with pretrained weights. This can provide a more stable starting point and often leads to better convergence.

2. Gradual Warmup: Implement a learning rate scheduler with a gradual warmup phase. This can help the model adapt to your dataset more effectively before applying higher learning rates.

3. Data Normalization: Ensure that your input data is properly normalized. RT-DETR, like other transformer models, can be sensitive to the scale of the input data.

4. Data Augmentation: Review your data augmentation pipeline. Sometimes, overly aggressive augmentations can destabilize training. Consider reducing the intensity or frequency of certain augmentations.

5. Batch Size: If possible, try increasing the batch size. A larger batch size can help stabilize training by providing more diverse gradients at each step.

6. Gradient Clipping: Implement gradient clipping to prevent exploding gradients, which can cause NaN values during training.

7. Regularization: Experiment with different regularization techniques, such as weight decay or dropout, to prevent overfitting and help stabilize training.

8. Review Dataset: Double-check your dataset for any anomalies or corrupt images/annotations that could be causing issues during training.

9. Debugging: Use tools like TensorBoard to monitor the training process. Look for any sudden spikes in loss values that could indicate where the NaN values are originating.

10. Community Support: Engage with the Ultralytics community on GitHub or other forums. Other users may have encountered similar issues and could have specific advice for your scenario.

Remember, model training can be an iterative process, and finding the right combination of settings can take some experimentation. Keep trying different approaches, and you'll likely find a strategy that works well for your particular dataset. Good luck! 🚀

[sdreamforchen]

Hello， Hope to help.
If I want design a N or Tiny model like yolov8n or yolov8t.
How to modify the config of RT-DETR?

[pderrenger]

@sdreamforchen designing a smaller or "Tiny" version of a model like RT-DETR involves modifying the architecture to reduce the number of parameters and computational complexity. Here are some general steps you can follow to create a scaled-down version of RT-DETR:

1. Reduce the Number of Layers: The depth of the model can be reduced by decreasing the number of transformer encoder and decoder layers.

2. Narrow the Model: Reduce the width of the model by decreasing the dimensionality of the feature maps within the transformer blocks.

3. Simplify the Head: The detection head can be simplified by reducing the number of prediction layers or the complexity of the prediction heads.

4. Adjust the Input Resolution: Smaller models often operate on lower input resolutions to save on computation.

5. Prune the Model: Remove less important weights or filters from the model.

6. Knowledge Distillation: Train the smaller model using a larger, pre-trained model as a teacher to transfer knowledge.

7. Hyperparameter Tuning: After making architectural changes, you'll likely need to fine-tune hyperparameters such as the learning rate, weight decay, and others for optimal performance.

For RT-DETR specifically, you would need to delve into the model's configuration file and adjust the parameters that define the model's size and complexity. This could involve modifying the number of attention heads, the dimensionality of the feedforward network within the transformers, and other architecture-specific settings.

Remember to validate your new model design thoroughly to ensure that the performance meets your requirements. Smaller models may be faster and require less computational resources, but this often comes at the cost of reduced accuracy.

If you need further guidance on modifying the RT-DETR configuration for a smaller model, I recommend checking the Ultralytics documentation and potentially reaching out to the community for best practices on designing efficient architectures.

[MacJavellana]

hello!! if i would want to do knowledge distillation on RTDETR by using a trained model, of my own dataset, for its teacher and a modified rtdetr lightweight backbone (i.e. efficietnet backbone) for its student. how will i be doing this ? could you point me in the right direction or if there has been any discussion for this could you link them. im quite lost at the moment

any help is welcome thank you in advance!

[chuong98]

Hi, was you able to reproduce the training results for rt-detr ysing ultralitic code? Also, I found there is a config in YoloV8/yolov8-rtdetr.yaml, is there a pretrained weight on COCO for this model yet?
Thank you

[creativesalam]

You can try this:

from ultralytics import RTDETR

model = RTDETR("rtdetr-l.pt")
model.info() # display model information
model.train(data="coco8.yaml") # train
model.predict("path/to/image.jpg") # predict

It will download the pre-trained weights for rtdetr-l.pt, there is another one as well rtdetr-x.pt [model = RTDETR("rtdetr-x.pt")].

[chuong98]

Thanks for your reply.
But I mean can the training code reproduce the results? Not inference result using the pretrained weight from Paddle.

[Adeelbek]

Using a pre-trained weight option did not help me in my training with the custom dataset. It still gives the same issue which I raised above.
matrix contains invalid numeric entries
I thought the pre-trained weight may extend my training epochs at least near 100 but it had reached only 8 epochs and ended up with a matrix value issue.

[pderrenger]

@Adeelbek i'm sorry to hear that you're still encountering the ValueError: matrix contains invalid numeric entries issue during training with RT-DETR. Here are a few additional suggestions that might help you troubleshoot and potentially resolve the problem:

• Data Inspection: Ensure that your custom dataset is correctly annotated and does not contain any corrupt images or labels that could produce NaN values during training.
• Batch Size Adjustment: If you're using a very large batch size, try reducing it. A smaller batch size can sometimes help stabilize training.
• Gradient Clipping: Implement gradient clipping to prevent gradients from exploding, which can lead to NaN values.
• Learning Rate Scheduler: Use a learning rate scheduler to adjust the learning rate during training dynamically. Sometimes a learning rate that decays over time can help stabilize training.
• Normalization: Verify that your input data is properly normalized. Input data should typically be scaled to a range that the model expects, such as [0, 1] or [-1, 1].
• Debugging: Enable debugging to catch NaN values. PyTorch, for example, has utilities like torch.autograd.set_detect_anomaly(True) that can help identify where NaNs are being introduced.

If these steps do not resolve the issue, it might be beneficial to reach out to the Ultralytics team directly through the appropriate channels, such as their GitHub issues page, for further assistance. They may have more specific insights into the RT-DETR model's training nuances, especially with custom datasets.

[cqlp1314]

I got an error when training without mosaic

[Adeelbek]

What did you train and why did you train without mosaic?

[pderrenger]

@Adeelbek it seems you encountered an error while training without the mosaic augmentation. Mosaic is a data augmentation technique that combines four training images into one, allowing the model to learn from multiple contexts and scales, which can be beneficial for object detection tasks.

Training without mosaic might be a deliberate choice if you're aiming for a specific use case or dataset characteristic where mosaic does not improve performance or is not applicable. However, it's important to ensure that your dataset and training configuration are compatible with training without this augmentation.

Here are a few things to consider:

• Dataset Diversity: Without mosaic, your model relies solely on the diversity within your original dataset. Ensure your dataset is sufficiently varied in terms of object scales, orientations, and backgrounds.
• Hyperparameters: Adjust other hyperparameters to compensate for the lack of mosaic augmentation. This might include changes to learning rate, batch size, or other augmentations.
• Model Capacity: Ensure your model has the right capacity to learn from the dataset without the need for mosaic augmentation. Sometimes, a more complex model might be required.

If you're still facing issues, it might be helpful to:

• Check Dataset Annotations: Ensure that your dataset annotations are correct and there are no corrupt images or labels.
• Debugging: Use debugging tools or techniques to identify the exact source of the error. This might involve checking the loss values, gradients, or intermediate outputs for any anomalies.

If you need further assistance, please provide more details about the error message, your dataset, and the training configuration. This will help in diagnosing the issue more effectively. 🛠️

[Adeelbek]

I trained RT-DETR Small (ResNet-18 backbone) model with my single-class custom dataset in the PaddlePaddle platform.
Training has ended successfully without any errors or problems. The following screenshot demonstrates the final training results.
I trained from scratch and still was able to get the results. It means in Pytorch implementation has some issues.

[pderrenger]

@Adeelbek it's great to hear that you've successfully trained the RT-DETR Small model with a ResNet-18 backbone on your custom dataset using the PaddlePaddle platform! 🎉 Training from scratch and achieving good results is a significant accomplishment, especially with a single-class dataset.

Regarding the PyTorch implementation, if you're experiencing issues, it could be due to a variety of factors, such as differences in the dataset preprocessing, training hyperparameters, or even the underlying architecture adaptations between PaddlePaddle and PyTorch. Here are a few suggestions that might help resolve any issues you're facing with the PyTorch implementation:

• Double-check the dataset preprocessing steps to ensure they are consistent with the PaddlePaddle implementation.
• Review the training hyperparameters and try to align them as closely as possible with your successful PaddlePaddle training run.
• If you're using automatic mixed precision (amp=True), consider training with amp=False to see if it affects the stability of the training process.
• Ensure that the loss functions and optimizer settings are equivalent across both platforms.

If you continue to face challenges, you might want to reach out to the community or the maintainers of the repository for more targeted support. Sharing details about the specific issues you're encountering will help others provide more accurate advice.

Keep up the good work, and I hope you continue to achieve excellent results with your models! 🚀

[ExtReMLapin]

those GPT-4 api tier answers are getting out of hands

[sdreamforchen]

How about the time delay ，someone declare it's very slow 

…

---Original--- From: ***@***.***> Date: Wed, Jun 21, 2023 15:05 PM To: ***@***.***>; Cc: ***@***.******@***.***>; Subject: Re: [ultralytics/ultralytics] RT-DETR (Realtime DetectionTransformer) - Ultralytics YOLOv8 Docs (Discussion #2545) I trained RT-DETR Small (ResNet-18 backbone) model with my single-class custom dataset in the PaddlePaddle platform. Training has ended successfully without any errors or problems. The following screenshot demonstrates the final training results. I trained from scratch and still was able to get the results. It means in Pytorch implementation has some issues. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***>

[Adeelbek]

what time ?

[pderrenger]

@Adeelbek it seems like there's a concern about potential latency or slow performance with RT-DETR models. Here are a few points to consider:

• Model Complexity: RT-DETR models, especially larger ones, can be computationally intensive due to their transformer architecture, which may lead to longer inference times compared to more lightweight models.
• Hardware Utilization: The inference speed can greatly depend on the hardware being used. GPUs, particularly those with TensorRT optimization, can significantly speed up the process.
• Optimization: Using half-precision computation (FP16) can reduce latency without a substantial loss in accuracy. This is often a good trade-off for real-time applications.
• Batch Size: Adjusting the batch size during inference can also impact speed. Larger batch sizes can be more efficient on some hardware.

If you're experiencing slower than expected performance, you might want to:

• Check if you're using the most appropriate model size for your needs (e.g., 'RT-DETR-L' vs. 'RT-DETR-X').
• Ensure that your hardware is fully utilized and compatible with the optimizations used by RT-DETR.
• Experiment with different configurations, such as reducing the image resolution or turning off certain features that might not be necessary for your use case.

If you have specific metrics or scenarios where the time delay is an issue, providing more details can help in giving a more targeted response. 🕒🔧

[sdreamforchen]

Your trained RT-DETR Small module inference time 

…

---Original--- From: ***@***.***> Date: Wed, Jun 21, 2023 16:46 PM To: ***@***.***>; Cc: ***@***.******@***.***>; Subject: Re: [ultralytics/ultralytics] RT-DETR (Realtime DetectionTransformer) - Ultralytics YOLOv8 Docs (Discussion #2545) what time ? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***>

[Adeelbek]

If you look at my screenshot, you may see 52 FPS. So, for inference delay, you may calculate accordingly

[creativesalam]

No this FPS is different, when I changed the Evaluation batch size, this FPS is affected. I don't think it can accurately calculate the inference time.

[pderrenger]

@creativesalam the FPS (Frames Per Second) value you're seeing is indicative of the model's inference speed, but it's important to note that this can vary based on several factors, including the batch size during evaluation, the hardware specifications of the machine running the inference, and the specific configurations of the model.

If you're looking to get a more accurate measure of inference time for a single image (which is often what's needed for real-time applications), you should run the model inference with a batch size of 1 and measure the time taken for the model to process this single image. This will give you a better idea of the inference delay or latency you can expect in a real-time scenario.

To measure the inference time for a single image, you can use the following Python code snippet:

import time
from ultralytics import YOLO

# Load the model
model = YOLO('path/to/your/rtdetr-small.pt')

# Load an image
img = 'path/to/your/image.jpg'

# Run inference with a single image and measure the time taken
start_time = time.time()
results = model(img)
end_time = time.time()

# Calculate the inference time
inference_time = end_time - start_time
print(f"Inference time for a single image: {inference_time:.4f} seconds")

This will give you the inference time in seconds for a single image. Remember to run this test on the same hardware where you plan to deploy the model to get a realistic measure of inference time.

[sdreamforchen]

Thanks

…

---Original--- From: ***@***.***> Date: Wed, Jun 21, 2023 17:01 PM To: ***@***.***>; Cc: ***@***.******@***.***>; Subject: Re: [ultralytics/ultralytics] RT-DETR (Realtime DetectionTransformer) - Ultralytics YOLOv8 Docs (Discussion #2545) I you look at my screenshot, you may see 52 FPS. So, inference delay you may calculate accordingly — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***>

[glenn-jocher]

Let us know if you need any other help!

[RfailesDev]

Hello,

I want to train an RT-DETR model on my custom dataset (12k images). I used code from the documentation with slight modifications; however, it's behaving strangely. When it comes to caching, it starts executing code again, which seems to be causing the error:

RuntimeError: 
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

Here's my code:

# Load a COCO-pretrained RT-DETR-l model
model = RTDETR('rtdetr-l.pt')

# Train the model on the custom example dataset for 80 epochs
results = model.train(data=f'{dataset}/data.yaml', epochs=80, imgsz=1024, batch=-1, cache="disk", name='custom_model_12k')

If needed, this is the content of :data.yaml

train: ../train/images
val: ../valid/images
test: ../test/images

nc: 4
names: ['a', 'ah', 'b', 'bh']

I tried to solve this issue by adding if __name__ == '__main__', but after another code restart, it just got stuck in an infinite loop.
(Maybe it has some relevance, but I'm using PyCharm.)
Thank you in advance.

[glenn-jocher]

Hi there,

It seems like you're encountering a multiprocessing issue, which is common when using PyTorch on Windows with PyCharm. The error you're seeing typically occurs when the DataLoader tries to use more than one worker in a non-main thread.

Here's a quick suggestion that might help:

• Ensure that your dataset loading and model training code is under if __name__ == '__main__': to avoid issues with the way Python multiprocessing works, especially on Windows.

Your modified code should look something like this:

from ultralytics import RTDETR

def train():
    # Load a COCO-pretrained RT-DETR-l model
    model = RTDETR('rtdetr-l.pt')

    # Train the model on the custom example dataset for 80 epochs
    results = model.train(data=f'{dataset}/data.yaml', epochs=80, imgsz=1024, batch=-1, cache="disk", name='custom_model_12k')

if __name__ == '__main__':
    train()

If you're still facing issues, try reducing the number of workers to 0 by adding workers=0 in your train() function call. This will disable multiprocessing and might resolve the error.

Let me know if this helps or if you need further assistance! 😊

[leonokida]

Hello,
I am facing a problem when validating my RT-DETR model after I trained it:
Traceback (most recent call last): File "/lustre04/scratch/leonokid/rt-detr/train.py", line 6, in <module> metrics = model.val(split='test') ^^^^^^^^^^^^^^^^^^^^^^^ File "/home/leonokid/.local/lib/python3.11/site-packages/ultralytics/vit/rtdetr/model.py", line 122, in val validator(model=self.model) File "/home/leonokid/.local/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/leonokid/.local/lib/python3.11/site-packages/ultralytics/yolo/engine/validator.py", line 131, in __call__ raise FileNotFoundError(emojis(f"Dataset '{self.args.data}' for task={self.args.task} not found ❌")) FileNotFoundError: Dataset 'None' for task=detect not found ❌
My code is:
from ultralytics import RTDETR model = RTDETR('./rtdetr-l.pt') model.train(data="data.yaml", epochs=200, verbose=True, seed=0, amp=False, batch=8) metrics = model.val(split='test')
My data.yaml file is:
train: /home/leonokid/scratch/rt-detr/dataset/train val: /home/leonokid/scratch/rt-detr/dataset/validation test: /home/leonokid/scratch/rt-detr/dataset/test nc: 1 names: ['smoke']

[glenn-jocher]

Hello! It looks like the validation process is unable to locate your dataset because the data argument isn't being retained after training. When you call model.val(split='test'), the model expects the data argument to be set with the path to your dataset configuration file.

To fix this, you should pass the data argument again when calling model.val(). Here's how you can modify your code:

from ultralytics import RTDETR 

# Load your model and train
model = RTDETR('./rtdetr-l.pt')  
model.train(data="data.yaml", epochs=200, verbose=True, seed=0, amp=False, batch=8)  

# Validate your model
metrics = model.val(data="data.yaml", split='test')  # Pass the data argument here

Make sure that your data.yaml file is correctly formatted and the paths to your datasets are correct. Give this a try, and it should resolve the FileNotFoundError. If you encounter any more issues, feel free to reach out! 😊

[mohaliyet]

I'm facing an issue while training the model with custom data: some loss logs are showing 'nan' values. I've followed the documentation closely. Could this be a mistake on my part, or is it a problem with the implementation?

[glenn-jocher]

@mohaliyet RTDETR is inherently slower and less stable than YOLOv8 during training, so unless there's a specific reason you are training RTDETR I would simply train with YOLOv8 instead.

[bharath5673]

RT-DETR (Tracker) in simple lines of code @ https://github.com/bharath5673/RT-DETR/blob/main/test.py

[glenn-jocher]

@bharath5673 thanks for sharing!

[vgarciafer]

Is it possible to run hyperparameter tuning with RT-DETR?
I can succesfully run it with YOLOv8 but not with RT-DETR. It returns an warning without further explanation.

[glenn-jocher]

@vgarciafer yes this should be possible. If you are seeing a reproducible bug when tuning RTDETR models please raise a bug report directly at https://github.com/ultralytics/ultralytics/issues

[Xiao-Qi-Guai]

I'm facing an issue while validating the model. According to the training results, when validating during training, I get valbatch0pred.jpg with the correct categories from my dataset. However, when using the 'val' command separately, the result is 0, and the categories are not from my dataset. How to tackle this problem?

[glenn-jocher]

@Xiao-Qi-Guai I'm sorry to hear you are running into issues. If you encounter a reproducible bug during RTDETR training or validation please raise a bug report directly at https://github.com/ultralytics/ultralytics/issues with code to reproduce. Thank you!

[kooqi]

Hi , thanks for your nice work !
I am using RT-DETR Pre-trained weights rtdetr-l.pt , I cannot resume training from an interrupted session, the error message is as follows:

New https://pypi.org/project/ultralytics/8.0.223 available 😃 Update with 'pip install -U ultralytics'
Ultralytics YOLOv8.0.175 🚀 Python-3.9.17 torch-2.0.1+cu117 CUDA:0 (Tesla P40, 24446MiB)
                                                            CUDA:1 (Tesla P40, 24446MiB)
engine/trainer: task=detect, mode=train, model=/home/lb/work/plants-visio/runs/detect/train7/weights/last.pt, data=/data/working/dataset.yaml, epochs=100, patience=50, batch=32, imgsz=640, save=True, save_period=-1, cache=False, device=(0, 1), workers=8, project=None, name=None, exist_ok=False, pretrained=True, optimizer=auto, verbose=True, seed=0, deterministic=True, single_cls=False, rect=False, cos_lr=False, close_mosaic=10, resume=False, amp=True, fraction=1.0, profile=False, freeze=None, overlap_mask=True, mask_ratio=4, dropout=0.0, val=True, split=val, save_json=False, save_hybrid=False, conf=None, iou=0.7, max_det=300, half=False, dnn=False, plots=True, source=None, show=False, save_txt=False, save_conf=False, save_crop=False, show_labels=True, show_conf=True, vid_stride=1, stream_buffer=False, line_width=None, visualize=False, augment=False, agnostic_nms=False, classes=None, retina_masks=False, boxes=True, format=torchscript, keras=False, optimize=False, int8=False, dynamic=False, simplify=False, opset=None, workspace=4, nms=False, lr0=0.01, lrf=0.01, momentum=0.937, weight_decay=0.0005, warmup_epochs=3.0, warmup_momentum=0.8, warmup_bias_lr=0.0, box=7.5, cls=0.5, dfl=1.5, pose=12.0, kobj=1.0, label_smoothing=0.0, nbs=64, hsv_h=0.015, hsv_s=0.7, hsv_v=0.4, degrees=0.0, translate=0.1, scale=0.5, shear=0.0, perspective=0.0, flipud=0.0, fliplr=0.5, mosaic=1.0, mixup=0.0, copy_paste=0.0, cfg=None, tracker=botsort.yaml, save_dir=runs/detect/train7
DDP: debug command /home/lb/anaconda3/envs/vision-test/bin/python -m torch.distributed.run --nproc_per_node 2 --master_port 54321 /home/lb/.config/Ultralytics/DDP/_temp_0b5jiffb139986701020512.py
TensorBoard: Start with 'tensorboard --logdir runs/detect/train7', view at http://localhost:6006/
WARNING ⚠️ no model scale passed. Assuming scale='l'.
Transferred 941/941 items from pretrained weights
AMP: running Automatic Mixed Precision (AMP) checks with YOLOv8n...
AMP: checks passed ✅
train: Scanning /data/working/train/labels.cache... 21711 images, 109 backgrounds, 0 corrupt: 100%|██████████| 21719/21719 [00:00<?, ?it/s]
train: WARNING ⚠️ /data/working/train/images/near30_near30_01_06_2021_v_0_63.png: 1 duplicate labels removed
train: WARNING ⚠️ /data/working/train/images/vwg-0010-0006_jpg.rf.2360eca5d05bb92021afd136c7634a7f.jpg: 1 duplicate labels removed
train: WARNING ⚠️ /data/working/train/images/vwg-0010-0006_jpg.rf.54d2168a88537cae71834e2caa19144a.jpg: 1 duplicate labels removed
train: WARNING ⚠️ /data/working/train/images/vwg-0010-0006_jpg.rf.a89ff2aa562efeb4c7efb1f5ea95cd93.jpg: 1 duplicate labels removed
val: Scanning /data/working/valid/labels.cache... 2225 images, 1 backgrounds, 0 corrupt: 100%|██████████| 2226/2226 [00:00<?, ?it/s]
Plotting labels to runs/detect/train7/labels.jpg... 
optimizer: 'optimizer=auto' found, ignoring 'lr0=0.01' and 'momentum=0.937' and determining best 'optimizer', 'lr0' and 'momentum' automatically... 
optimizer: SGD(lr=0.01, momentum=0.9) with parameter groups 143 weight(decay=0.0), 206 weight(decay=0.0005), 226 bias(decay=0.0)
/home/lb/anaconda3/envs/vision-test/lib/python3.9/site-packages/ultralytics/nn/modules/transformer.py:86: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  grid_w = torch.arange(int(w), dtype=torch.float32)
/home/lb/anaconda3/envs/vision-test/lib/python3.9/site-packages/ultralytics/nn/modules/transformer.py:87: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  grid_h = torch.arange(int(h), dtype=torch.float32)
/home/lb/anaconda3/envs/vision-test/lib/python3.9/site-packages/ultralytics/nn/modules/transformer.py:89: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert embed_dim % 4 == 0, \
WARNING ⚠️ TensorBoard graph visualization failure 'NoneType' object is not subscriptable
Image sizes 640 train, 640 val
Using 16 dataloader workers
Logging results to runs/detect/train7
Starting training for 100 epochs...
Traceback (most recent call last):
  File "/home/lb/.config/Ultralytics/DDP/_temp_0b5jiffb139986701020512.py", line 9, in <module>
    trainer.train()
  File "/home/lb/anaconda3/envs/vision-test/lib/python3.9/site-packages/ultralytics/engine/trainer.py", line 195, in train
    self._do_train(world_size)
  File "/home/lb/anaconda3/envs/vision-test/lib/python3.9/site-packages/ultralytics/engine/trainer.py", line 348, in _do_train
    self.loss, self.loss_items = self.model(batch)
  File "/home/lb/anaconda3/envs/vision-test/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/lb/anaconda3/envs/vision-test/lib/python3.9/site-packages/torch/nn/parallel/distributed.py", line 1156, in forward
    output = self._run_ddp_forward(*inputs, **kwargs)
  File "/home/lb/anaconda3/envs/vision-test/lib/python3.9/site-packages/torch/nn/parallel/distributed.py", line 1110, in _run_ddp_forward
    return module_to_run(*inputs[0], **kwargs[0])  # type: ignore[index]
  File "/home/lb/anaconda3/envs/vision-test/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/lb/anaconda3/envs/vision-test/lib/python3.9/site-packages/ultralytics/nn/tasks.py", line 44, in forward
    return self.loss(x, *args, **kwargs)
  File "/home/lb/anaconda3/envs/vision-test/lib/python3.9/site-packages/ultralytics/nn/tasks.py", line 212, in loss
    self.criterion = self.init_criterion()
  File "/home/lb/anaconda3/envs/vision-test/lib/python3.9/site-packages/ultralytics/nn/tasks.py", line 292, in init_criterion
    return v8DetectionLoss(self)
  File "/home/lb/anaconda3/envs/vision-test/lib/python3.9/site-packages/ultralytics/utils/loss.py", line 119, in __init__
    self.stride = m.stride  # model strides
  File "/home/lb/anaconda3/envs/vision-test/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1614, in __getattr__
    raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'RTDETRDecoder' object has no attribute 'stride'
Traceback (most recent call last):
  File "/home/lb/.config/Ultralytics/DDP/_temp_0b5jiffb139986701020512.py", line 9, in <module>
    trainer.train()
  File "/home/lb/anaconda3/envs/vision-test/lib/python3.9/site-packages/ultralytics/engine/trainer.py", line 195, in train
    self._do_train(world_size)
  File "/home/lb/anaconda3/envs/vision-test/lib/python3.9/site-packages/ultralytics/engine/trainer.py", line 348, in _do_train
    self.loss, self.loss_items = self.model(batch)
  File "/home/lb/anaconda3/envs/vision-test/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/lb/anaconda3/envs/vision-test/lib/python3.9/site-packages/torch/nn/parallel/distributed.py", line 1156, in forward
    output = self._run_ddp_forward(*inputs, **kwargs)
  File "/home/lb/anaconda3/envs/vision-test/lib/python3.9/site-packages/torch/nn/parallel/distributed.py", line 1110, in _run_ddp_forward
    return module_to_run(*inputs[0], **kwargs[0])  # type: ignore[index]
  File "/home/lb/anaconda3/envs/vision-test/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/lb/anaconda3/envs/vision-test/lib/python3.9/site-packages/ultralytics/nn/tasks.py", line 44, in forward
    return self.loss(x, *args, **kwargs)
  File "/home/lb/anaconda3/envs/vision-test/lib/python3.9/site-packages/ultralytics/nn/tasks.py", line 212, in loss
    self.criterion = self.init_criterion()
  File "/home/lb/anaconda3/envs/vision-test/lib/python3.9/site-packages/ultralytics/nn/tasks.py", line 292, in init_criterion
    return v8DetectionLoss(self)
  File "/home/lb/anaconda3/envs/vision-test/lib/python3.9/site-packages/ultralytics/utils/loss.py", line 119, in __init__
    self.stride = m.stride  # model strides
  File "/home/lb/anaconda3/envs/vision-test/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1614, in __getattr__
    raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'RTDETRDecoder' object has no attribute 'stride'
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 3083) of binary: /home/lb/anaconda3/envs/vision-test/bin/python
Traceback (most recent call last):
  File "/home/lb/anaconda3/envs/vision-test/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/lb/anaconda3/envs/vision-test/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/lb/anaconda3/envs/vision-test/lib/python3.9/site-packages/torch/distributed/run.py", line 798, in <module>
    main()
  File "/home/lb/anaconda3/envs/vision-test/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
    return f(*args, **kwargs)
  File "/home/lb/anaconda3/envs/vision-test/lib/python3.9/site-packages/torch/distributed/run.py", line 794, in main
    run(args)
  File "/home/lb/anaconda3/envs/vision-test/lib/python3.9/site-packages/torch/distributed/run.py", line 785, in run
    elastic_launch(
  File "/home/lb/anaconda3/envs/vision-test/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/lb/anaconda3/envs/vision-test/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
/home/lb/.config/Ultralytics/DDP/_temp_0b5jiffb139986701020512.py FAILED
------------------------------------------------------------
Failures:
[1]:
  time      : 2023-12-08_08:30:55
  host      : ubuntulb
  rank      : 1 (local_rank: 1)
  exitcode  : 1 (pid: 3084)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2023-12-08_08:30:55
  host      : ubuntulb
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 3083)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
Traceback (most recent call last):
  File "/home/lb/anaconda3/envs/vision-test/bin/yolo", line 8, in <module>
    sys.exit(entrypoint())
  File "/home/lb/anaconda3/envs/vision-test/lib/python3.9/site-packages/ultralytics/cfg/__init__.py", line 445, in entrypoint
    getattr(model, mode)(**overrides)  # default args from model
  File "/home/lb/anaconda3/envs/vision-test/lib/python3.9/site-packages/ultralytics/engine/model.py", line 341, in train
    self.trainer.train()
  File "/home/lb/anaconda3/envs/vision-test/lib/python3.9/site-packages/ultralytics/engine/trainer.py", line 190, in train
    raise e
  File "/home/lb/anaconda3/envs/vision-test/lib/python3.9/site-packages/ultralytics/engine/trainer.py", line 188, in train
    subprocess.run(cmd, check=True)
  File "/home/lb/anaconda3/envs/vision-test/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/home/lb/anaconda3/envs/vision-test/bin/python', '-m', 'torch.distributed.run', '--nproc_per_node', '2', '--master_port', '54321', '/home/lb/.config/Ultralytics/DDP/_temp_0b5jiffb139986701020512.py']' returned non-zero exit status 1.

[glenn-jocher]

@kooqi if you encounter a reproducible bug please raise a bug report directly at https://github.com/ultralytics/ultralytics/issues with code to reproduce. Thank you!

[FrankARicardo]

Hello.

I downloaded model x, and using the same pipeline as YOLO, the training progressed without errors, but at some point it gave me "nan" values in the metrics..... Any thoughts on this?
YOLOv8 finished without the nan values.

Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
69/100 5.63G nan nan nan 35 640: 100%|██████████| 60/60 [00:13<00:00, 4.36it/s]
Class Images Instances Box(P R mAP50 mAP50-95): 100%|██████████| 8/8 [00:00<00:00, 9.79it/s]

[FrankARicardo]

To be more clear, I am comparing RT-DETR with YOLOv8, on the same data set.

The best model of RT-DETR does worse than YOLOv8 on the same test data set. I am pretty sure that due to the above problem in the training stage.

[glenn-jocher]

@FrankARicardo I wouldn't be surprised if this is the case, YOLOv8 should outperform in most real world situations. I'd imagine this is the reason it's the most popular model.

[UtkuEla]

Hello I would like to use RT-DETR with a project of mine. It already pre trained for many classes from the COCO dataset. I just want the model to classify 10 of those classes and I want to train the model for 10 new classes from OpenImagesV7 dataset. So at the end, I want a model that classifies those 20 classes only. How can I do that most efficiently?

[FrankARicardo]

Hello,
If you know how to train a YOLO model, just download the RT-DETR model and follow the same steps, if don't, look the guiding videos... These are the changes I made:

from ultralytics import RTDETR

detr_x = RTDETR("rtdetr-x.pt")
detr_x.train(data="custom.yaml", epochs=100, imgsz=640, batch=4, workers=1, dropout=0.1)

[UtkuEla]

Hello Frank, thank you for the answer!

So I can put different classes from different datasets in the custom.yaml right?

[glenn-jocher]

@UtkuEla you can accomplish this by combining both datasets and then training on the combined dataset. If you train on a second dataset after training on a first, the initial training will be forgotten as all resources are devoted to learning the new data.

[Mariegri]

Hello there!
I'm trying to train RT-DETR on custom dataset, and the isuue is with the structure of the dataset:
I tried:
dataset/
|-- train/
| |-- images/
-- 0001.jpg
-- 0002.jpg
-- 0003.jpg
...
| |-- labels/
-- 0001.txt
-- 0002.txt
-- 0003.txt
...
|-- val/
| |-- images/
-- 0101.jpg
-- 0102.jpg
-- 0103.jpg
...
| |-- labels/
-- 0101.txt
-- 0102.txt
-- 0103.txt
...
|-- test/
| |-- images/
-- 1001.jpg
-- 1002.jpg
-- 1003.jpg
...
| |-- labels/
-- 1001.txt
-- 1002.txt
-- 1003.txt
...

with YAML:

path: .\dataset
train: .dataset\train
val: .\dataset\val
test: .\dataset\test
names: '0'

and YAML:

path: .\dataset
train: .dataset\train\images
val: .\dataset\val\images
test: .\dataset\test\images
names: '0'

and had the error:
No labels found in .\dataset\train\labels
No images found in .\dataset\train\labels

I tried:
dataset/
|--images/
| |-- train/
-- 0001.jpg
-- 0002.jpg
-- 0003.jpg
...
| |-- val/
-- 0101.jpg
-- 0102.jpg
-- 0103.jpg
...
| |-- test/
-- 1001.jpg
-- 1002.jpg
-- 1003.jpg
...
|--labels/
| |-- train/
-- 0001.txt
-- 0002.txt
-- 0003.txt
...
| |-- val/
-- 0101.txt
-- 0102.txt
-- 0103.txt
...
| |-- test/
-- 1001.txt
-- 1002.txt
-- 1003.txt
...

with YAML:

path: .\dataset
train: .\dataset\images\train
val: .\dataset\images\val
test: .\dataset\images\test
names: '0'

and got:
No labels found in .\dataset\labels\train
No images found in .\dataset\abels\train

I tried:
dataset/
|--images/
| |-- train/
-- 0001.jpg
-- 0002.jpg
-- 0003.jpg
...
-- 0001.txt
-- 0002.txt
-- 0003.txt
...
| |-- val/
-- 0101.jpg
-- 0102.jpg
-- 0103.jpg
...
-- 0101.txt
-- 0102.txt
-- 0103.txt
...
| |-- test/
-- 1001.jpg
-- 1002.jpg
-- 1003.jpg
...
-- 1001.txt
-- 1002.txt
-- 1003.txt
...

with YAML:

path: .\dataset
train: .\dataset\images\train
val: .\dataset\images\val
test: .\dataset\images\test
names: '0'

and got:
No labels found in .\dataset\labels\train.cache
Cache directory .\dataset\labels is not writeable, cache not saved.

Could you please specify the structure of the dataset (where should labelsbe saved and how to specify the path to them, if needed) and what else is wrong in my attempts?

Thanks a lot in advance!

[glenn-jocher]

@Mariegri the simplest way to start is to train directly from an example dataset, let it autodownload, and then inspect it's structure. You can use this code to get started training RTDETR models on COCO8:

from ultralytics import RTDETR

# Load a model YAML architecture
model = RTDETR('yolov8n-rtdetr.yaml')

# Load pretrained weights
model = model.load('yolov8n.pt')

# Train the model on any dataset
model.train(data='coco8.yaml')

[UtkuEla]

Hello when I run the inference with RT-DETR,

"""
model = RTDETR('rtdetr-l.pt')
results = model(source= "IMG_3028.jpg", show=True, conf= 0.4, save= True)
"""
it shows the class numbers instead of names of the objects. So instead of person, it shows 0. How can I fix that?

[glenn-jocher]

@UtkuEla yes this is true, I see the same result. @Laughing-q how can we fix this?

Downloading https://github.com/ultralytics/assets/releases/download/v0.0.0/rtdetr-l.pt to 'rtdetr-l.pt'...
100%|██████████| 63.4M/63.4M [00:01<00:00, 42.7MB/s]
WARNING ⚠️ 'source' is missing. Using 'source=/Users/glennjocher/PycharmProjects/assistant/venv/lib/python3.11/site-packages/ultralytics/assets'.
image 1/2 /Users/glennjocher/PycharmProjects/assistant/venv/lib/python3.11/site-packages/ultralytics/assets/bus.jpg: 640x640 4 0s, 1 5, 1 9, 1 10, 2 27s, 737.0ms
image 2/2 /Users/glennjocher/PycharmProjects/assistant/venv/lib/python3.11/site-packages/ultralytics/assets/zidane.jpg: 640x640 2 0s, 2 27s, 553.6ms
Speed: 2.4ms preprocess, 645.3ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 640)

[ghost]

Hi, does Ultralytics support training and deployment of smaller versions of rt-detr such as 's' or 'm'. If so do you have configs for that or any example? i suppose paddle config wont work with Ultralytics package out of the box.

[ExtReMLapin]

You can copy/paste the file and rename it and change the l values inside, but you can't train from pretrained

[glenn-jocher]

@dfundo yes you can train any size hybrid YOLOv8-RTDETR model using yolov8-rtdetr.yaml and then loading any pretrained weights you want:

from ultralytics import RTDETR

# Load a model YAML architecture
model = RTDETR('yolov8n-rtdetr.yaml')

# Load pretrained weights
model = model.load('yolov8n.pt')

# Train the model on any dataset
model.train(data='coco8.yaml')

[eumentis-madhurzanwar]

Hi, should we expect .pt to .ptl conversion of rt-detr models trained via ultralytics package to perform the same on edge devices like other yolov8 models do? I am getting very weird results for rt-detr model when i convert it to .ptl via torchscript route on mobile.

[pderrenger]

Hey there! 👋

For the .pt to .ptl conversion of RT-DETR models, it's possible that the conversion process might introduce some discrepancies, especially when deploying on edge devices. The behavior you're observing could be due to various factors, including differences in the way operations are handled in the mobile environment or the conversion process itself.

It's important to ensure that the model is properly optimized for mobile deployment and that the conversion process preserves the integrity of the model's architecture and weights. If you're experiencing issues, you might want to double-check the conversion steps and look for any potential sources of incompatibility.

If you continue to face challenges, feel free to share more details or reach out to the community for further assistance. We're here to help! 🚀

[eumentis-madhurzanwar]

Thank you for the answer. The thing is I have been doing the conversion from .pt to .ptl for yolov8 models to deploy them on edge devices and the results are very much comparable to the web counterpart. With the same process, when I do it for RT-DETR models trained via ultralytics package and deploy it on edge devices, the results are not at all comparable with the web counterpart. If the process of converting the models had a flaw, the yolov8 models should also not have been performed up to the mark.

[eumentis-madhurzanwar]

Can this be the reason -> For yolov8 models, when I convert .pt to torchscript and from torchscript to .ptl the functions are getting converted which are in turn compatible with the edge devices. In Rt-detr there might be some functions that are not getting converted properly for edge devices.

[pderrenger]

Hey there! 👋

You're on the right track thinking about the function compatibility during the .pt to .ptl conversion process. It's indeed possible that certain operations used in RT-DETR might not be fully supported or optimized for mobile deployment, which could lead to unexpected behavior when running the model on edge devices.

For RT-DETR models, it's crucial to ensure that all operations are mobile-friendly and that the conversion process is tailored to maintain the model's integrity. If you're seeing significant discrepancies, it might be worth exploring the model's architecture to identify any operations that could be causing issues and looking for potential workarounds or updates that could improve compatibility.

Keep experimenting, and don't hesitate to reach out if you need more assistance! 🚀

[ien952129]

我使用RT-DETR進行訓練，發覺有大量nan，

[glenn-jocher]

@ien952129 hello,

Thank you for reaching out and providing the screenshot. It looks like you're encountering NaN values during training with RT-DETR. Let's work through this together to resolve the issue.

1. Reproducible Code Example: Could you please provide a minimum reproducible code example? This will help us understand the context and reproduce the issue on our end. You can refer to our guide on creating a minimum reproducible example.

2. Package Versions: Ensure you are using the latest versions of torch and ultralytics. You can update them using the following commands:

   pip install --upgrade torch ultralytics

3. Training Tips:

   • Lower Learning Rate: Sometimes, a high learning rate can cause instability. Try reducing the learning rate.
   • Pretrained Weights: Use pretrained RT-DETR weights instead of training from scratch. This can help stabilize the training process.
   • Disable AMP: Automatic Mixed Precision (AMP) can sometimes cause issues. Try setting amp=False in your training configuration.

Here is an example of how you can adjust these settings in your training script:

from ultralytics import RTDETR

# Load a COCO-pretrained RT-DETR-l model
model = RTDETR("rtdetr-l.pt")

# Train the model with adjusted settings
results = model.train(data="coco8.yaml", epochs=100, imgsz=640, lr0=0.001, amp=False)

Or via CLI:

yolo train model=rtdetr-l.pt data=coco8.yaml epochs=100 imgsz=640 lr0=0.001 amp=False

Please try these suggestions and let us know if the issue persists. We're here to help!

[bluenight12]

how to get some pre-trained model list in ultralytics? I think I saw yolo-rtdetr-l.pt somewhere, but I can't find it again.

[pderrenger]

@bluenight12 hi there!

To get a list of pre-trained models available in Ultralytics, including the RT-DETR models, you can refer to the Ultralytics documentation. Specifically, for RT-DETR, we offer the following pre-trained models:

• RT-DETR-L: Achieves 53.0% AP on COCO val2017 and runs at 114 FPS on a T4 GPU.
• RT-DETR-X: Achieves 54.8% AP on COCO val2017 and runs at 74 FPS on a T4 GPU.

Here’s how you can use these models with the Ultralytics Python API:

Python Example

from ultralytics import RTDETR

# Load a COCO-pretrained RT-DETR-l model
model = RTDETR("rtdetr-l.pt")

# Display model information (optional)
model.info()

# Train the model on the COCO8 example dataset for 100 epochs
results = model.train(data="coco8.yaml", epochs=100, imgsz=640)

# Run inference with the RT-DETR-l model on the 'bus.jpg' image
results = model("path/to/bus.jpg")

CLI Example

# Train the RT-DETR-l model on the COCO8 example dataset for 100 epochs
yolo train model=rtdetr-l.pt data=coco8.yaml epochs=100 imgsz=640

# Run inference with the RT-DETR-l model on the 'bus.jpg' image
yolo predict model=rtdetr-l.pt source=path/to/bus.jpg

For a comprehensive list of all pre-trained models and their supported tasks and modes, please visit the RT-DETR documentation page.

If you have any further questions or need additional assistance, feel free to ask! 😊