Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: Support BF16 data type in TensorRT backend #7310

Merged
merged 9 commits into from
Jun 7, 2024

Conversation

pskiran1
Copy link
Member

@pskiran1 pskiran1 commented May 31, 2024

What does the PR do?

  • Added new L0_trt_bf16_dtype job to test BF16 datatype for TensorRT backend.
  • Updated model generation script to generate BF16 datatype TensorRT models.

Note: BF16 datatype data type is supported for GPU with CC 8.0 and above only.

Checklist

  • PR title reflects the change and is of format <commit_type>: <Title>
  • Changes are described in the pull request.
  • Related issues are referenced.
  • Populated github labels field
  • Added test plan and verified test passes.
  • Verified that the PR passes existing CI.
  • Verified copyright is correct on all changed files.
  • Added succinct git squash message before merging ref.
  • All template sections are filled out.
  • Optional: Additional screenshots for behavior/output changes with before/after.

Commit Type:

Check the conventional commit type
box here and add the label to the github PR.

  • build
  • ci
  • docs
  • feat
  • fix
  • perf
  • refactor
  • revert
  • style
  • test

Related PRs:

triton-inference-server/tensorrt_backend#90

Where should the reviewer start?

Test plan:

  • CI Pipeline ID: 15487143

Caveats:

Background

https://docs.nvidia.com/deeplearning/tensorrt/support-matrix/index.html#hardware-precision-matrix

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

  • closes GitHub issue: #xxx

qa/common/gen_qa_models.py Outdated Show resolved Hide resolved
@pskiran1 pskiran1 requested a review from GuanLuo June 4, 2024 15:24
docs/user_guide/model_configuration.md Show resolved Hide resolved
qa/common/gen_qa_models.py Outdated Show resolved Hide resolved
qa/common/gen_qa_models.py Outdated Show resolved Hide resolved
qa/common/gen_qa_models.py Outdated Show resolved Hide resolved
qa/common/gen_qa_models.py Outdated Show resolved Hide resolved
qa/common/gen_qa_model_repository Outdated Show resolved Hide resolved
@pskiran1 pskiran1 requested a review from tanmayv25 June 6, 2024 09:09
@pskiran1 pskiran1 merged commit 906baa5 into main Jun 7, 2024
3 checks passed
@pskiran1 pskiran1 deleted the spolisetty_BF16_DLIS-6659 branch June 7, 2024 04:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

4 participants