Run test_sqlalchemy_store.py against PostgreSQL, MySQL, and MSSQL

[harupy]

Signed-off-by: harupy 17039389+harupy@users.noreply.github.com

What changes are proposed in this pull request?

Run test_sqlalchemy_store.py against PostgreSQL, MySQL, and MSSQL to ensure MLflow tracking operations work in different databases and prevent issues / regressions.

How is this patch tested?

Fixed tests

Does this PR change the documentation?

• No. You can skip the rest of this section.
• Yes. Make sure the changed pages / sections render correctly by following the steps below.

1. Check the status of the ci/circleci: build_doc check. If it's successful, proceed to the
   next step, otherwise fix it.
2. Click Details on the right to open the job page of CircleCI.
3. Click the Artifacts tab.
4. Click docs/build/html/index.html.
5. Find the changed pages / sections and make sure they render correctly.

Release Notes

Is this a user-facing change?

• No. You can skip the rest of this section.
• Yes. Give a description of this change to be included in the release notes for MLflow users.

(Details in 1-2 sentences. You can just refer to another PR with a description if this PR is part of a larger change.)

What component(s), interfaces, languages, and integrations does this PR affect?

Components

• area/artifacts: Artifact stores and artifact logging
• area/build: Build and test infrastructure for MLflow
• area/docs: MLflow documentation pages
• area/examples: Example code
• area/model-registry: Model Registry service, APIs, and the fluent client calls for Model Registry
• area/models: MLmodel format, model serialization/deserialization, flavors
• area/projects: MLproject format, project running backends
• area/scoring: MLflow Model server, model deployment tools, Spark UDFs
• area/server-infra: MLflow Tracking server backend
• area/tracking: Tracking Service, tracking client APIs, autologging

Interface

• area/uiux: Front-end, user experience, plotting, JavaScript, JavaScript dev server
• area/docker: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
• area/sqlalchemy: Use of SQLAlchemy in the Tracking Service or Model Registry
• area/windows: Windows support

Language

• language/r: R APIs and clients
• language/java: Java APIs and clients
• language/new: Proposals for new client languages

Integrations

• integrations/azure: Azure and Azure ML integrations
• integrations/sagemaker: SageMaker integrations
• integrations/databricks: Databricks integrations

How should the PR be classified in the release notes? Choose one:

• rn/breaking-change - The PR will be mentioned in the "Breaking Changes" section
• rn/none - No description will be included. The PR will be mentioned only by the PR number in the "Small Bugfixes and Documentation Updates" section
• rn/feature - A new user-facing feature worth mentioning in the release notes
• rn/bug-fix - A user-facing bug fix worth mentioning in the release notes
• rn/documentation - A user-facing documentation change worth mentioning in the release notes

[harupy]

Temporary change to check which tests fail.

[harupy]

Let's say we have code like this:

with mlflow.start_run():
    log_metric("m", 1)

with mlflow.start_run():
    log_metric("m", np.nan)

with mlflow.start_run():
    pass

mlflow.search_runs(["0"], order_by=["metrics.m"])

This code creates a table like below, then sorts rows using the metrics.m and clause_1 columns.

run_id	metrics.m	is_nan	clause_0
1	1	False	0
2	0	True	1
3	NULL	False	1

SELECT ... ORDER BY clause_1, metrics.m

In PostgreSQL, the result looks like this because NULLs are sorted AFTER non-null values:

run_id	metrics.m	is_nan	clause_1
1	1	False	0
2	0	True	1
3	NULL	False	1

In other databases (SQLite, MySQL, SQL server), the result looks like this because NULLs are sorted BEFORE non-null values:

run_id	metrics.m	is_nan	clause_1
1	1	False	0
3	NULL	False	1
2	0	True	1

[harupy]

In the new code, we sort a table like this:

run_id	metrics.m	is_nan	clause_1
1	1	False	0
2	0	True	1
3	NULL	False	2

which produces the same result in all databases

[harupy]

LIKE operator is case-insensitive in MySQL and SQL server.

[harupy]

This block enables case_sensitive_like for SQLite:

mlflow/mlflow/store/db/utils.py

Lines 76 to 78 in d9489d0

	if db_type == SQLITE:
	session.execute("PRAGMA foreign_keys = ON;")
	session.execute("PRAGMA case_sensitive_like = true;")

[BenWilson2]

Do we want to include Kana sensitive as well?
"SQL_Latin1_General_CP1_CS_AS_KS" or does sorting based on characters make the ordering confusing?

[harupy]

I think we do :)

[harupy]

Updated!

[harupy]

SQL_Latin1_General_CP1_CS_AS_KS didn't work:

https://github.com/mlflow/mlflow/runs/5728726867?check_suite_focus=true#step:11:10259

E           mlflow.exceptions.MlflowException: (pyodbc.ProgrammingError) ('42000', "[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Invalid collation 'SQL_Latin1_General_CP1_CS_AS_KS'. (448) (SQLExecDirectW)")

[harupy]

Is _KS only available for Japanese collations such as Japanese_Bushu_Kakusu_100_CS_AS_KS_WS_UTF8?

[harupy]

I don't think kana insensitivity is a big issue though.

[harupy]

Offline discussion: we should also support kana sensitivity if there are no downsides.

[BenWilson2]

I'm wondering if the _UTF8 addition as suffix is required. It's pretty hard to tell from their docs... https://docs.microsoft.com/en-us/sql/relational-databases/collations/collation-and-unicode-support?view=sql-server-ver15#Collation_Defn
I wonder if people using MSSQL as a tracking backend end up writing a T-SQL script when they setup MLflow to enable faster filtering and apply specific column collations to speed it up?

I agree, not that big of a deal to enable this (it would just be nice to enable functionality for as many users as possible, but if the config is broken, then it's broken...)

[harupy]

Offline discussion: insert a non-binary LIKE to improve the performance.

LIKE :pattern AND LIKE BINARY :pattern

[harupy]

Code to check kana sensitivity in SQL server:

import mlflow
import uuid

name = uuid.uuid4().hex
experiment_id = mlflow.create_experiment(name)
mlflow.set_experiment(name)

for val in ["りんご", "リンゴ"]:
    with mlflow.start_run():
        mlflow.log_param("p", val)

print(mlflow.search_runs([experiment_id], filter_string="parameters.p LIKE 'リンゴ'"))

[harupy]

Suggested change

	like_op = column.collate("Japanese_Bushu_Kakusu_100_CS_AS_KS_WS").like
	like_op = column.collate("Japanese_Bushu_Kakusu_100_CS_AS_KS_WS_UTF8").like

I tried Japanese_Bushu_Kakusu_100_CS_AS_KS_WS on local but got an invalid collation error.

[BenWilson2]

curiosity question: does this behavior create a UI bug with an MSSQL backend if someone searches an invalid condition?

[harupy]

@BenWilson2
It doesn't because MLflow UI doesn't allow users to set the number of runs to fetch.

[BenWilson2]

This is great stuff, Haru! Improving usability for users utilizing different backend stores and helping to notify us if any changes will create issues for them is wonderful.
LGTM!