Adding new export for Triton #2078

brightsparc · 2022-05-31T06:38:40Z

Added new util and CLI function to support exporting model for Triton inference server.

Adding new export for triton that creates torchscript model with Inference wrapper, and config.pbtxt
Fixes for load model for neuropod export.
Fixes neurpod to format output dict, added annotation for inputs.
Added test to export and load triton model and run inference.

…t, add output dict format

github-actions · 2022-05-31T07:25:16Z

Unit Test Results

      6 files ±0       6 suites ±0 2h 7m 24s ⏱️ - 1m 10s
2 799 tests +1 2 764 ✔️ +1   35 💤 ±0 0 ❌ ±0
8 397 runs +3 8 288 ✔️ +3 109 💤 ±0 0 ❌ ±0

Results for commit c156cb2. ± Comparison against base commit 8e4bf24.

♻️ This comment has been updated with latest results.

…dded annotation for neurpod

ludwig/utils/triton_utils.py

…rt-triton

Adding new export for triton. Fixes for load model for neuropod expor…

77461e5

…t, add output dict format

brightsparc force-pushed the enh-export-triton branch from 6fa27ee to a23baad Compare May 31, 2022 10:58

Adding test for triton. Fix to cast int to string for os.path.join. A…

6ed0e79

…dded annotation for neurpod

brightsparc force-pushed the enh-export-triton branch from a23baad to 6ed0e79 Compare May 31, 2022 11:04

Minor tweaks to config.pbtxt output

fa80951

justinxzhao approved these changes May 31, 2022

View reviewed changes

ludwig/utils/triton_utils.py Outdated Show resolved Hide resolved

brightsparc and others added 5 commits June 1, 2022 09:36

Remove logger that is not being used

2b77928

Restrict torchmetrics<0.9 and whylogs<1.0 until compatibility fixed

c24e532

Merge commit 'c24e532c7f0e50e149875a8ab5088fbe21c794e1' into enh-expo…

5e9f68d

…rt-triton

Update export_triton to return model path, and added docstrings

1b14b21

Update api to return both model path and config path

c156cb2

brightsparc merged commit 698a0e0 into master Jun 1, 2022

brightsparc deleted the enh-export-triton branch June 1, 2022 04:08