add ort session options #8433

leqiao-1 · 2023-05-15T06:39:57Z

Related Issues/PRs

#8413

What changes are proposed in this pull request?

Add onnxruntime session options when saving and loading onnx model with pyfunc

How is this patch tested?

Existing unit/integration tests
New unit/integration tests
Manual tests (describe details, including test results, below)

Does this PR change the documentation?

No. You can skip the rest of this section.
Yes. Make sure the changed pages / sections render correctly in the documentation preview.

Release Notes

Is this a user-facing change?

No. You can skip the rest of this section.
Yes. Give a description of this change to be included in the release notes for MLflow users.

User can provide onnxruntime session options when saving and loading onnx model with pyfunc.

What component(s), interfaces, languages, and integrations does this PR affect?

Components

Interface

area/uiux: Front-end, user experience, plotting, JavaScript, JavaScript dev server
area/docker: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
area/sqlalchemy: Use of SQLAlchemy in the Tracking Service or Model Registry
area/windows: Windows support

Language

language/r: R APIs and clients
language/java: Java APIs and clients
language/new: Proposals for new client languages

Integrations

integrations/azure: Azure and Azure ML integrations
integrations/sagemaker: SageMaker integrations
integrations/databricks: Databricks integrations

How should the PR be classified in the release notes? Choose one:

rn/breaking-change - The PR will be mentioned in the "Breaking Changes" section
rn/none - No description will be included. The PR will be mentioned only by the PR number in the "Small Bugfixes and Documentation Updates" section
rn/feature - A new user-facing feature worth mentioning in the release notes
rn/bug-fix - A user-facing bug fix worth mentioning in the release notes
rn/documentation - A user-facing documentation change worth mentioning in the release notes

mlflow-automation · 2023-05-15T06:40:13Z

Documentation preview for 82d4667 will be available here when this CircleCI job completes successfully.

More info

Ignore this comment if this PR does not change the documentation.
It takes a few minutes for the preview to be available.
The preview is updated when a new commit is pushed to this PR.
This comment was created by https://github.com/mlflow/mlflow/actions/runs/5020341199.

github-actions · 2023-05-15T06:40:14Z

@leqiao-1 Thank you for the contribution! Could you fix the following issue(s)?

⚠ DCO check

The DCO check failed. Please sign off your commit(s) by following the instructions here. See https://github.com/mlflow/mlflow/blob/master/CONTRIBUTING.md#sign-your-work for more details.

WeichenXu123 · 2023-05-15T09:40:19Z

mlflow/onnx.py

@@ -176,6 +178,7 @@ def save_model(
        onnx_version=onnx.__version__,
        data=model_data_subpath,
        providers=onnx_execution_providers,
+        options=onnx_session_options,


I think we need to add some validation here, validate the option keys and value types.

why not use onnx_session_options=onnx_session_options ?

WeichenXu123 · 2023-05-15T09:42:58Z

mlflow/onnx.py

+                sess_options.inter_op_num_threads = inter_op_num_threads
+            if intra_op_num_threads:
+                sess_options.intra_op_num_threads = intra_op_num_threads
+            if execution_mode:


Why not force execution_mode value to be one of "SEQUENTIAL" and "PARALLEL" ? and we should document it.

WeichenXu123 · 2023-05-15T09:44:09Z

mlflow/onnx.py

+                sess_options.graph_optimization_level = onnxruntime.GraphOptimizationLevel(
+                    graph_optimization_level
+                )
+            if extra_session_config:


Can it be an arbitrary dict ?

WeichenXu123

overall good, but I have some comments.

WeichenXu123 · 2023-05-16T00:36:41Z

mlflow/onnx.py

@@ -255,9 +255,9 @@ def __init__(self, path, providers=None):
            if intra_op_num_threads:
                sess_options.intra_op_num_threads = intra_op_num_threads
            if execution_mode:
-                if execution_mode == 0:
+                if execution_mode.upper() == "SEQUENTIAL":


We should document that when logging model, we should use "SEQUENTIAL" or "PARALLEL" as session option value.

mlflow/onnx.py

WeichenXu123 · 2023-05-16T10:29:55Z

mlflow/onnx.py

+    :param onnx_session_options: Dictionary of options to be passed to onnxruntime.InferenceSession.
+                                 For example:
+                                 ``{
+                                 'graph_optimization_level': 99,
+                                 'intra_op_num_threads': 1,
+                                 'inter_op_num_threads': 1,
+                                 'execution_mode': 'sequential'
+                                 }``
+                                 'execution_mode' can be set ot 'sequential' or 'parallel'.


Pls also add the doc in "log_model" method

It has been added

WeichenXu123 · 2023-05-16T10:31:20Z

mlflow/onnx.py

@@ -176,6 +188,7 @@ def save_model(
        onnx_version=onnx.__version__,
        data=model_data_subpath,
        providers=onnx_execution_providers,
+        onnx_session_options=onnx_session_options,


we can add simple verification:

check keys are valid

for execution_mode, check value are valid

for other option keys, check types are correct, e.g., inter_op_num_threads should be integer that is > 0

Signed-off-by: leqiao <leqiao@microsoft.com>

WeichenXu123 · 2023-05-18T12:03:55Z

mlflow/onnx.py

+                if execution_mode.upper() == "SEQUENTIAL":
+                    sess_options.execution_mode = onnxruntime.ExecutionMode.ORT_SEQUENTIAL
+                elif execution_mode.upper() == "PARALLEL":
+                    sess_options.execution_mode = onnxruntime.ExecutionMode.ORT_PARALLEL


nit: for other values, raises error.

OK we don't need this. We already do verification when saving model

WeichenXu123

LGTM

WeichenXu123 · 2023-05-19T01:21:10Z

@leqiao-1 Could you fix the lint errors https://github.com/mlflow/mlflow/actions/runs/5011618497/jobs/8987189752?pr=8433

Once you fix them, I will merge the PR.

Signed-off-by: leqiao <leqiao@microsoft.com>

leqiao-1 · 2023-05-19T02:56:04Z

Format fixed. Thanks

harupy · 2023-05-19T05:30:57Z

mlflow/utils/model_utils.py

+                raise ValueError(
+                    f"Key {key} in onnx_session_options is not a valid \
+                        ONNX Runtime session options key"
+                )


This is how the actual error message look like:

key = 1 raise ValueError( f"Key {key} in onnx_session_options is not a valid \ ONNX Runtime session options key" )

Traceback (most recent call last): File "a.py", line 2, in <module> raise ValueError( ValueError: Key 1 in onnx_session_options is not a valid ONNX Runtime session options key

Good catch!

harupy · 2023-05-19T05:31:40Z

mlflow/onnx.py

+            inter_op_num_threads = options.get("inter_op_num_threads")
+            intra_op_num_threads = options.get("intra_op_num_threads")
+            execution_mode = options.get("execution_mode")
+            graph_optimization_level = options.get("graph_optimization_level")
+            extra_session_config = options.get("extra_session_config")
+            if inter_op_num_threads:
+                sess_options.inter_op_num_threads = inter_op_num_threads
+            if intra_op_num_threads:
+                sess_options.intra_op_num_threads = intra_op_num_threads


This can be simplified by using assignment expressions:

if inter_op_num_threads := options.get("inter_op_num_threads"): # do something with inter_op_num_threads

leqiao-1 · 2023-05-19T06:44:58Z

@harupy I created another PR to fix the comments. #8472

github-actions bot added area/models MLmodel format, model serialization/deserialization, flavors rn/feature Mention under Features in Changelogs. labels May 15, 2023

leqiao-1 force-pushed the leqiao/ort_session_options branch 2 times, most recently from 99e9bfa to 8a75bc6 Compare May 15, 2023 09:31

WeichenXu123 reviewed May 15, 2023

View reviewed changes

WeichenXu123 reviewed May 16, 2023

View reviewed changes

mlflow/onnx.py Outdated Show resolved Hide resolved

WeichenXu123 reviewed May 16, 2023

View reviewed changes

mlflow/onnx.py Outdated Show resolved Hide resolved

WeichenXu123 reviewed May 16, 2023

View reviewed changes

leqiao-1 force-pushed the leqiao/ort_session_options branch 2 times, most recently from a7ad12c to 39c3a21 Compare May 18, 2023 07:26

leqiao-1 added 8 commits May 18, 2023 15:27

add ort session options

16307f0

Signed-off-by: leqiao <leqiao@microsoft.com>

format update

bc6106a

Signed-off-by: leqiao <leqiao@microsoft.com>

code update

1c0c91a

Signed-off-by: leqiao <leqiao@microsoft.com>

documentation update

7e1a84e

Signed-off-by: leqiao <leqiao@microsoft.com>

doc fix

fb195a1

Signed-off-by: leqiao <leqiao@microsoft.com>

test update

32e9c77

Signed-off-by: leqiao <leqiao@microsoft.com>

typo fix

18938a2

Signed-off-by: leqiao <leqiao@microsoft.com>

add onnx_session_options validation

275ddca

Signed-off-by: leqiao <leqiao@microsoft.com>

leqiao-1 force-pushed the leqiao/ort_session_options branch from 39c3a21 to 275ddca Compare May 18, 2023 07:27

WeichenXu123 reviewed May 18, 2023

View reviewed changes

WeichenXu123 approved these changes May 18, 2023

View reviewed changes

WeichenXu123 enabled auto-merge (squash) May 18, 2023 12:06

auto-merge was automatically disabled May 19, 2023 02:54
Head branch was pushed to by a user without write access

format fix

82d4667

Signed-off-by: leqiao <leqiao@microsoft.com>

leqiao-1 force-pushed the leqiao/ort_session_options branch from 049735c to 82d4667 Compare May 19, 2023 02:55

WeichenXu123 enabled auto-merge (squash) May 19, 2023 04:22

WeichenXu123 merged commit 4c91827 into mlflow:master May 19, 2023
33 checks passed

harupy reviewed May 19, 2023

View reviewed changes

leqiao-1 mentioned this pull request May 19, 2023

fix error message #8472

Merged

33 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add ort session options #8433

add ort session options #8433

leqiao-1 commented May 15, 2023

mlflow-automation commented May 15, 2023 •

edited

github-actions bot commented May 15, 2023

WeichenXu123 May 15, 2023 •

edited

WeichenXu123 May 15, 2023 •

edited

WeichenXu123 May 15, 2023

WeichenXu123 May 15, 2023

WeichenXu123 left a comment

WeichenXu123 May 16, 2023

WeichenXu123 May 16, 2023

leqiao-1 May 18, 2023

WeichenXu123 May 16, 2023

WeichenXu123 May 18, 2023

WeichenXu123 May 18, 2023

WeichenXu123 left a comment

WeichenXu123 commented May 19, 2023

leqiao-1 commented May 19, 2023

harupy May 19, 2023

WeichenXu123 May 19, 2023

harupy May 19, 2023

leqiao-1 commented May 19, 2023

add ort session options #8433

add ort session options #8433

Conversation

leqiao-1 commented May 15, 2023

Related Issues/PRs

What changes are proposed in this pull request?

How is this patch tested?

Does this PR change the documentation?

Release Notes

Is this a user-facing change?

What component(s), interfaces, languages, and integrations does this PR affect?

How should the PR be classified in the release notes? Choose one:

mlflow-automation commented May 15, 2023 • edited

github-actions bot commented May 15, 2023

⚠ DCO check

WeichenXu123 May 15, 2023 • edited

Choose a reason for hiding this comment

WeichenXu123 May 15, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

WeichenXu123 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

WeichenXu123 left a comment

Choose a reason for hiding this comment

WeichenXu123 commented May 19, 2023

leqiao-1 commented May 19, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

leqiao-1 commented May 19, 2023

mlflow-automation commented May 15, 2023 •

edited

WeichenXu123 May 15, 2023 •

edited

WeichenXu123 May 15, 2023 •

edited