diff --git a/0.10.0/404.html b/0.10.0/404.html new file mode 100644 index 00000000..5a4a2b9e --- /dev/null +++ b/0.10.0/404.html @@ -0,0 +1,1764 @@ + + + +
+ + + + + + + + + + + + + + +nav.Module
accepts batching
flag which overrides a config setting and precision
which allows setting appropriate configuration for TensorRTnav.load_optimized()
nav.bundle.save
and nav.bundle.load
to save and load optimized models from cachenav.package.optimize
nav.profile()
nav.profile()
fix: Forward method not passed to other inplace modules.
+Version of external components used during testing:
+nav.profile()
(enabled by default)MODEL_NAVIGATOR_DEFAULT_CACHE_DIR
env variablenav.Module
can now receive a function name to be used instead of call in modules/submodules, allows customizing modules with non-standard callsnav.profile()
change: nav.optimize()
does not override module configuration
Version of external components used during testing:
+change: updated ONNX package to 1.16
+Version of external components used during testing:
+fix: Lower peak memory usage when loading Torch inplace optimized model
+Version of external components used during testing:
+change: Add input and output specs for Triton model repositories generated from packages
+Version of external components used during testing:
+fix: Passing input data to OnnxCUDA runner
+Version of external components used during testing:
+default_model_filename
in Triton model configurationfix: Limited Polygraphy version to work correctly with onnxruntime-gpu package
+Version of external components used during testing:
+change: removed Windows from support list
+Version of external components used during testing:
+change: always execute export for all supported formats
+Known issues and limitations:
+Version of external components used during testing:
+change: Raise exception instead of exit with code when required command has failed
+Version of external components used during testing:
+fix: use pytree metadata to flatten torch complex outputs
+Version of external components used during testing:
+torch.onnx.export
+ or polygraphy convert
change: Store separate reproduction scripts for runners used during correctness and profiling
+Version of external components used during testing:
+fix: Conditional imports of supported frameworks in export commands
+Version of external components used during testing:
+change: Improved rendering documentation
+Version of external components used during testing:
+fix: Add model from package to Triton model store with custom configs
+Version of external components used during testing:
+nav.pacakge.profile
API method to profile generated models on provided dataloaderbatch_sizes
and max_batch_size
limit the max profile in TensorRT conversionnav.package.optimize
on empty package - status generation onlytorch.inference_mode
for inference runner when PyTorch 2.x is availablemodel
in config when passing package generated during nav.{framework}.optimize
directly
+ to nav.package.optimize
commandOther minor fixes and improvements
+Version of external components used during testing:
+Other minor fixes and improvements
+Version of external components used during testing:
+nav.utilities
module with UnpackedDataloader wrapperfix: Invalid optimal shape calculation for odd values in max batch size
+Version of external components used during testing:
+fix: Save stdout to logfiles in debug mode
+Version of external components used during testing:
+fix: filter outputs using output_metadata in ONNX runners
+Version of external components used during testing:
+fix: Log full traceback for ModelNavigatorUserInputError
+Version of external components used during testing:
+fix: bugfixes for functional tests
+Version of external components used during testing:
+compile
supportchange: Stored find max batch size result in separate filed in status
+Version of external components used during testing:
+fix: when exporting single input model to saved model, unwrap one element list with inputs
+Version of external components used during testing:
+fix: in Keras inference use model.predict(tensor) for single input models
+Version of external components used during testing:
+fix: collecting metadata from ONNX model in main thread during ONNX to TRT conversion
+Version of external components used during testing:
+fix: when specified use dynamic axes from custom OnnxConfig
+Version of external components used during testing:
+optimize
method that replace export
and perform max batch size search and improved profiling during processoptimize
for better parametrization of export/conversion commandsBug fixes and other improvements
+Version of external components used during testing:
+Updated NVIDIA containers defaults to 22.11
+Version of external components used during testing:
+Updated NVIDIA containers defaults to 22.10
+Version of external components used during testing:
+Model Navigator Export API:
+Version of external components used during testing:
+Model Navigator Export API:
+use_cuda_graph=True
by defaultnan
or inf
valuesVersion of external components used during testing:
+TF32
precision for TensorRT from CLI options - will be removed in future versionsModel Navigator Export API:
+load
for packageonnx_runtimes
export function parameters to runtimes
TF32
precision for TensorRT from available options - will be removed in future versions_input_names
or _output_names
specifiedinput_names
Version of external components used during testing:
+Model Navigator Export API:
+Version of external components used during testing:
+uint64
or int64
input and/or outputs definedModel Navigator Export API:
+Version of external components used during testing:
+optimize
avoid running unsupported conversions in CLI--config-search-early-exit-enable
for Model Analyzer early exit support
+ in manual profiling mode--model-config-name
to the select
command.
+ It allows to pick a particular model configuration for deployment from the set of all configurations
+ generated by Triton Model Analyzer, even if it's not the best performing one.--tensorrt-strict-types
option has been removed due to deprecation of the functionality
+ in upstream libraries.Model Navigator Export API:
+Version of external components used during testing:
+run
command has been deprecated and may be removed in a future releaseoptimize
command replace run
and produces an output *.triton.nav
packageselect
selects the best-performing configuration from *.triton.nav
package and create a
+ Triton Inference Server model repositoryRemove wkhtmltopdf package dependency
+Version of external components used during testing:
+Profiling related:
+--config-search-max-preferred-batch-size
flag as is no longer supported in Triton Model AnalyzerVersion of external components used during testing:
+Fixes and improvements
+--model-format
argument when input and output files have no extensionVersion of external components used during testing:
+Known issues and limitations
+Configuration Model on Triton related
+Version of external components used during testing:
+Known issues and limitations
+dtypes
are not passedfixes and improvements
+Version of external components used during testing:
+Known issues and limitations
+model_repository
path to be not relative to <navigator_workspace>
dir--gpus
argument--tensorrt-sparse-weights
flag for sparse weight optimization for TensorRT--tensorrt-strict-types
flag forcing it to choose tactics based on the layer precision for TensorRT--tensorrt-explicit-precision
flag enabling explicit precision modeengine_count_per_device
triton_backend_parameters
commandtensorrt_max_workspace_size
config_search
prefix to all profiling parameters (BREAKING CHANGE)config_search_max_preferred_batch_size
parameterconfig_search_backend_parameters
parameterfixes and improvements
+Versions of used external components:
+Known issues and limitations
+Updated NVIDIA containers defaults to 21.08
+Versions of used external components:
+Known issues and limitations
+count_windows
mode as default option for Perf AnalyzerBugfixes
+Versions of used external components:
+Known issues and limitations
+comprehensive refactor of command-line API in order to provide more gradual + pipeline steps execution
+Versions of used external components:
+Known issues and limitations
+Release of main components:
+Versions of used external components:
+Known issues
+Contributions are welcome, and they are much appreciated! Every little +helps, and we will always give credit.
+Report bugs at https://github.com/triton-inference-server/model_navigator/issues.
+If you are reporting a bug, include:
+Look through the GitHub issues for bugs. Anything tagged with "bug" and "help +wanted" is open to whoever wants to implement it.
+Look through the GitHub issues for features. Anything tagged with "enhancement" and "help wanted" is open to whoever would like to implement it.
+The Triton Model Navigator could always use more documentation, whether as part of +the official Triton Model Navigator docs, in docstrings, or even on the web in blog posts, +articles, and such.
+The best way to send feedback is to file an issue at https://github.com/triton-inference-server/model_navigator/issues.
+If you are proposing a feature:
+We require that all contributors "sign-off" on their commits. This certifies that +the contribution is your original work, or you have the rights to submit it under +the same license or a compatible license.
+Any contribution which contains commits that are not Signed-Off will not be accepted.
+To sign off on a commit, you simply use the --signoff
(or -s
) option when committing your changes:
+
This will append the following to your commit message:
+ +By doing this, you certify the below:
+Developer Certificate of Origin
+Version 1.1
+
+Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
+1 Letterman Drive
+Suite D4700
+San Francisco, CA, 94129
+
+Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed.
+
+
+Developer's Certificate of Origin 1.1
+
+By making a contribution to this project, I certify that:
+
+(a) The contribution was created in whole or in part by me and I have the right to submit it under the open source license indicated in the file; or
+
+(b) The contribution is based upon previous work that, to the best of my knowledge, is covered under an appropriate open source license and I have the right under that license to submit that work with modifications, whether created in whole or in part by me, under the same open source license (unless I am permitted to submit under a different license), as indicated in the file; or
+
+(c) The contribution was provided directly to me by some other person who certified (a), (b) or (c) and I have not modified it.
+
+(d) I understand and agree that this project and the contribution are public and that a record of the contribution (including all personal information I submit with it, including my sign-off) is maintained indefinitely and may be redistributed consistent with this project or the open source license(s) involved.
+
Ready to contribute? Here's how to set up the Triton Model Navigator
for local development.
Triton Model Navigator
repo on GitHub.Now you can make your changes locally.
+$ make lint # will run i.a. flake8 and pytype linters
+$ make test # will run a test with on your current virtualenv
+$ make test-fw # will run a framework test inside framework container
+
$ git add .
+$ git commit -s -m "Your detailed description of your changes."
+$ git push origin name-of-your-bugfix-or-feature
+
Before you submit a pull request, check that it meets these guidelines:
+To run a subset of tests:
+ +As a reminder for the maintainers on how to deploy - +make sure all your changes are committed (including an entry in CHANGELOG.md) into the master branch. +Then run:
+ +Add/update docstrings as defined in the Google Style Guide.
+Triton requires that all contributors (or their corporate entity) send
+a signed copy of the Contributor License
+Agreement
+to triton-cla@nvidia.com.
+NOTE: Contributors with no company affiliation can fill N/A
in the
+Corporation Name
and Corporation Address
fields.
Apache License
+ Version 2.0, January 2004
+ http://www.apache.org/licenses/
+
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+Definitions.
+"License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document.
+"Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License.
+"Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity.
+"You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License.
+"Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files.
+"Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types.
+"Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below).
+"Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof.
+"Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution."
+"Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work.
+Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form.
+Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed.
+Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions:
+(a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and
+(b) You must cause any modified files to carry prominent notices + stating that You changed the files; and
+(c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and
+(d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License.
+You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License.
+Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions.
+Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file.
+Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License.
+Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages.
+Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability.
+