feat: add utility for signature validation of logged model to dataset

[serena-ruan]

Signed-off-by: Xinyue Ruan serena.rxy@gmail.com

Related Issues/PRs

Resolve #6092

What changes are proposed in this pull request?

Add utility method for get_model_info and validate_schema.

How is this patch tested?

• I have written tests (not required for typo or doc fix) and confirmed the proposed feature/bug-fix/change works.

Does this PR change the documentation?

• No. You can skip the rest of this section.
• Yes. Make sure the changed pages / sections render correctly by following the steps below.

1. Click the Details link on the Preview docs check.
2. Find the changed pages / sections and make sure they render correctly.

Release Notes

Is this a user-facing change?

• No. You can skip the rest of this section.
• Yes. Give a description of this change to be included in the release notes for MLflow users.

Add utility functions that enable users to 1. easily get model info from the model uri directly; 2. validate dataset against the target schema.

What component(s), interfaces, languages, and integrations does this PR affect?

Components

• area/artifacts: Artifact stores and artifact logging
• area/build: Build and test infrastructure for MLflow
• area/docs: MLflow documentation pages
• area/examples: Example code
• area/model-registry: Model Registry service, APIs, and the fluent client calls for Model Registry
• area/models: MLmodel format, model serialization/deserialization, flavors
• area/pipelines: Pipelines, Pipeline APIs, Pipeline configs, Pipeline Templates
• area/projects: MLproject format, project running backends
• area/scoring: MLflow Model server, model deployment tools, Spark UDFs
• area/server-infra: MLflow Tracking server backend
• area/tracking: Tracking Service, tracking client APIs, autologging

Interface

• area/uiux: Front-end, user experience, plotting, JavaScript, JavaScript dev server
• area/docker: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
• area/sqlalchemy: Use of SQLAlchemy in the Tracking Service or Model Registry
• area/windows: Windows support

Language

• language/r: R APIs and clients
• language/java: Java APIs and clients
• language/new: Proposals for new client languages

Integrations

• integrations/azure: Azure and Azure ML integrations
• integrations/sagemaker: SageMaker integrations
• integrations/databricks: Databricks integrations

How should the PR be classified in the release notes? Choose one:

• rn/breaking-change - The PR will be mentioned in the "Breaking Changes" section
• rn/none - No description will be included. The PR will be mentioned only by the PR number in the "Small Bugfixes and Documentation Updates" section
• rn/feature - A new user-facing feature worth mentioning in the release notes
• rn/bug-fix - A user-facing bug fix worth mentioning in the release notes
• rn/documentation - A user-facing documentation change worth mentioning in the release notes

[serena-ruan]

Raising a question here: this was previous set as the string format of "scipy.sparse.csc_matrix" and "scipy.sparse.csr_matrix", is there any specific reason for them?

[dbczumar]

Instead of having two methods, it would be great to include the model signature in the ModelInfo class so that users can call get_model_info().signature. I realize that ModelInfo has a signature_dict attribute; is it possible to add a signature attribute of type ModelSignature (rather than dict) and mark signature_dict as deprecated via the @deprecated decorator?

[serena-ruan]

Actually @depracted decorator can only be applied on classes or methods, so I added another similar decorator for this reference (let me know if you have other preferences since I'm not super expert here).

[dbczumar]

Do we need this check? Looks like pyfunc.predict() doesn't check isinstance() before calling _enforce_schema:

mlflow/mlflow/pyfunc/__init__.py

Lines 650 to 651 in 9d278b1

	if input_schema is not None:
	data = _enforce_schema(data, input_schema)

.

If we do need the check, can we check isinstance(data, PyFuncInput)? Though I don't think the check is necessary.

[serena-ruan]

Yes it doesn't check for predict, perhaps I can just delete this check and call _enforce_schema directly.

[dbczumar]

It doesn't look like pd.Series is considered valid by _enforce_schema (it will throw)

[serena-ruan]

Oh I did miss some parts, will add support as well.

[dbczumar]

Following from https://github.com/mlflow/mlflow/pull/6494/files#r950650995, should we change this to:

Suggested change

	def validate_schema(data: DataInputType, expected_schema: Schema) -> DataInputType:
	def validate_schema(data: PyFuncInput, expected_schema: Schema) -> DataInputType:

Since _enforce_schema doesn't support pandas series and validation of output schemas doesn't appear to be critically important?

[serena-ruan]

I actually added support of pandas series, as it can be directly turned into a pandas DataFrame and does the following validation. And in fact we could accept pandas.Series as a dataset input.

[WeichenXu123]

I think it is nice to make PyFuncInput support pd.Series type. @dbczumar Thoughts ?

But, @serena-ruan , your definition DataInputType = Union[PyFuncInput, PyFuncOutput] looks weird. We can directly reuse PyFuncInput here, if we add pd.Series into PyFuncInput

[serena-ruan]

Ahh yes, will drop DataInputType.

[dbczumar]

@serena-ruan Apologies for the delay in the review. This is looking great! Just left a few comments - let me know if you have any questions.

[serena-ruan]

@serena-ruan Apologies for the delay in the review. This is looking great! Just left a few comments - let me know if you have any questions.

Thanks for those comments! Didn't get a chance to take a look over the weekend, will address them later today!

[serena-ruan]

@dbczumar There's a constraint that if ModelInfo depends on ModelSignature, then we need to rely on pandas/numpy so we can't import them in mlflow-skinny. Besides, Model contains a method get_model_info so it also relies on ModelInfo, thus relying on pandas/numpy if we add ModelSignature into ModelInfo field. So I suggest we keep using signature_dict in ModelInfo and add another classmethod for ModelInfo to convert signature_dict back to ModelSignature using from_dict method. WDYT?

[dbczumar]

@dbczumar There's a constraint that if ModelInfo depends on ModelSignature, then we need to rely on pandas/numpy so we can't import them in mlflow-skinny. Besides, Model contains a method get_model_info so it also relies on ModelInfo, thus relying on pandas/numpy if we add ModelSignature into ModelInfo field. So I suggest we keep using signature_dict in ModelInfo and add another classmethod for ModelInfo to convert signature_dict back to ModelSignature using from_dict method. WDYT?

@serena-ruan Can we import ModelSignature in the implementation of the signature propertymethod, instead of importing in the top-level module? That should resolve the issue.

[serena-ruan]

@dbczumar There's a constraint that if ModelInfo depends on ModelSignature, then we need to rely on pandas/numpy so we can't import them in mlflow-skinny. Besides, Model contains a method get_model_info so it also relies on ModelInfo, thus relying on pandas/numpy if we add ModelSignature into ModelInfo field. So I suggest we keep using signature_dict in ModelInfo and add another classmethod for ModelInfo to convert signature_dict back to ModelSignature using from_dict method. WDYT?

@serena-ruan Can we import ModelSignature in the implementation of the signature propertymethod, instead of importing in the top-level module? That should resolve the issue.

This is a NamedTuple instead of a normal class containing attributes 🤔 But I can have a try

[dbczumar]

@dbczumar There's a constraint that if ModelInfo depends on ModelSignature, then we need to rely on pandas/numpy so we can't import them in mlflow-skinny. Besides, Model contains a method get_model_info so it also relies on ModelInfo, thus relying on pandas/numpy if we add ModelSignature into ModelInfo field. So I suggest we keep using signature_dict in ModelInfo and add another classmethod for ModelInfo to convert signature_dict back to ModelSignature using from_dict method. WDYT?

@serena-ruan Can we import ModelSignature in the implementation of the signature propertymethod, instead of importing in the top-level module? That should resolve the issue.

This is a NamedTuple instead of a normal class containing attributes 🤔 But I can have a try

Thanks! Hopefully that still works :). If not, we can convert from NamedTuple to class

[dbczumar]

@serena-ruan This is awesome! Almost ready for merge! Just a couple more small comments

[harupy]

Suggested change

	artifact_path: str = None,
	flavors: Dict[str, Any] = None,
	model_uri: str = None,
	model_uuid: str = None,
	run_id: str = None,
	saved_input_example_info: Optional[Dict[str, Any]] = None,
	signature_dict: Optional[Dict[str, Any]] = None,
	signature=None, # Optional[ModelSignature]
	utc_time_created: str = None,
	mlflow_version: str = None,
	artifact_path: str,
	flavors: Dict[str, Any],
	model_uri: str,
	model_uuid: str,
	run_id: str,
	saved_input_example_info: Optional[Dict[str, Any]],
	signature_dict: Optional[Dict[str, Any]],
	signature, # Optional[ModelSignature]
	utc_time_created: str,
	mlflow_version: str,

Do we need the default values? In the original ModelInfo, all the constructor arguments are positional.

[serena-ruan]

In general I guess not, but for signature_dict as it will be deprecated I suggest we keep default value None for it :)

[serena-ruan]

@dbczumar @harupy Can we merge this in if there's no extra comments? :D

[dbczumar]

@serena-ruan Apologies for the delay! Excited to merge this once three small comments are addressed!