Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. Weโ€™ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix TensorRT inference issue on NVIDIA Jetson #9482

Merged
merged 2 commits into from Apr 2, 2024

Conversation

lakshanthad
Copy link
Collaborator

@lakshanthad lakshanthad commented Apr 2, 2024

We need to downgrade numpy to v1.23.5 to fix a potential issue when inferencing TensorRT exported models on NVIDIA Jetson platform. I have applied the fix globally.

nvidia@nvidia-desktop:~$ yolo detect predict model=yolov8l.engine source="0"
Ultralytics YOLOv8.1.40 ๐Ÿš€ Python-3.8.10 torch-2.1.0a0+41361538.nv23.06 CUDA:0 (Orin, 15503MiB)
Loading yolov8l.engine for TensorRT inference...
[04/02/2024-00:13:14] [TRT] [I] Loaded engine size: 168 MiB
[04/02/2024-00:13:15] [TRT] [I] [MemUsageChange] Init cuDNN: CPU +616, GPU +584, now: CPU 1164, GPU 5393 (MiB)
[04/02/2024-00:13:15] [TRT] [I] [MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +167, now: CPU 0, GPU 167 (MiB)
[04/02/2024-00:13:15] [TRT] [I] [MemUsageChange] Init cuDNN: CPU +1, GPU +0, now: CPU 997, GPU 5236 (MiB)
[04/02/2024-00:13:15] [TRT] [I] [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +93, now: CPU 0, GPU 260 (MiB)
/usr/lib/python3.8/dist-packages/tensorrt/__init__.py:166: FutureWarning: In the future `np.bool` will be defined as the corresponding NumPy scalar.
  bool: np.bool,
Traceback (most recent call last):
  File "/home/nvidia/.local/bin/yolo", line 8, in <module>
    sys.exit(entrypoint())
  File "/home/nvidia/ultralytics/ultralytics/cfg/__init__.py", line 582, in entrypoint
    getattr(model, mode)(**overrides)  # default args from model
  File "/home/nvidia/ultralytics/ultralytics/engine/model.py", line 444, in predict
    self.predictor.setup_model(model=self.model, verbose=is_cli)
  File "/home/nvidia/ultralytics/ultralytics/engine/predictor.py", line 297, in setup_model
    self.model = AutoBackend(
  File "/home/nvidia/.local/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/nvidia/ultralytics/ultralytics/nn/autobackend.py", line 244, in __init__
    dtype = trt.nptype(model.get_binding_dtype(i))
  File "/usr/lib/python3.8/dist-packages/tensorrt/__init__.py", line 166, in nptype
    bool: np.bool,
  File "/home/nvidia/.local/lib/python3.8/site-packages/numpy/__init__.py", line 305, in __getattr__
    raise AttributeError(__former_attrs__[attr])
AttributeError: module 'numpy' has no attribute 'bool'.
`np.bool` was a deprecated alias for the builtin `bool`. To avoid this error in existing code, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here.
The aliases was originally deprecated in NumPy 1.20; for more details and guidance see the original release note at:
    https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations

I have read the CLA Document and I hereby sign the CLA

๐Ÿ› ๏ธ PR Summary

Made with โค๏ธ by Ultralytics Actions

๐ŸŒŸ Summary

Improvement of NVIDIA Jetson compatibility and streamlined installation for deep learning dependencies.

๐Ÿ“Š Key Changes

  • Updated Dockerfile for NVIDIA Jetson: Specific versions of numpy are now only enforced for Jetson devices to ensure compatibility with TensorRT.
  • Refined Installation Requirements: The installation process within the Dockerfile has been streamlined by grouping additional dependencies under the ".[export]" tag.
  • PyProject.toml Adjustments: Added a numpy version specification for aarch64 platforms, directly addressing a known compatibility issue with TensorRT on Jetson devices.

๐ŸŽฏ Purpose & Impact

  • ๐Ÿš€ Enhanced Compatibility: Ensures that deep learning models run smoothly on NVIDIA Jetson devices by fixing a known numpy issue.
  • ๐Ÿ›  Simplified Setup: Streamlines the setup process for developers, making it easier to get started with deploying models on various platforms.
  • ๐Ÿ’พ Future-Proofing: By specifying dependency versions, the PR aims to minimize future compatibility issues, enhancing the user experience for both developers and end-users deploying AI models on NVIDIA's Jetson platform.

Copy link

codecov bot commented Apr 2, 2024

Codecov Report

All modified and coverable lines are covered by tests โœ…

Project coverage is 76.62%. Comparing base (959acf6) to head (5e4b4c7).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #9482      +/-   ##
==========================================
- Coverage   79.70%   76.62%   -3.08%     
==========================================
  Files         120      120              
  Lines       15174    15174              
==========================================
- Hits        12094    11627     -467     
- Misses       3080     3547     +467     
Flag Coverage ฮ”
Benchmarks 36.30% <รธ> (-0.06%) โฌ‡๏ธ
GPU 38.23% <รธ> (รธ)
Tests 71.88% <รธ> (-3.48%) โฌ‡๏ธ

Flags with carried forward coverage won't be shown. Click here to find out more.

โ˜” View full report in Codecov by Sentry.
๐Ÿ“ข Have feedback on the report? Share it here.

@lakshanthad
Copy link
Collaborator Author

@glenn-jocher FYI

@glenn-jocher glenn-jocher merged commit 2f77b2e into ultralytics:main Apr 2, 2024
14 checks passed
@glenn-jocher
Copy link
Member

@lakshanthad looks good!

hmurari pushed a commit to hmurari/ultralytics that referenced this pull request Apr 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants