MLFlow 3.13 Raises Exception When Using Filesystem Backend

[ericspod]

Describe the bug
Version 3.13 released and changed warnings about the filesystem backend into an exception raise:

2026-06-01T09:41:37.5462995Z ======================================================================
2026-06-01T09:41:37.5463667Z ERROR: test_shape_1__home_runner_work_MONAI_MONAI_tests_testing_data_inference_yaml (tests.integration.test_integration_bundle_run.TestBundleRun)
2026-06-01T09:41:37.5464309Z ----------------------------------------------------------------------
2026-06-01T09:41:37.5464642Z Traceback (most recent call last):
2026-06-01T09:41:37.5465074Z   File "/home/runner/work/MONAI/MONAI/tests/test_utils.py", line 877, in command_line_tests
2026-06-01T09:41:37.5465652Z     normal_out = subprocess.run(cmd, env=test_env, check=True, capture_output=True)
2026-06-01T09:41:37.5466275Z   File "/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/subprocess.py", line 526, in run
2026-06-01T09:41:37.5466758Z     raise CalledProcessError(retcode, process.args,
2026-06-01T09:41:37.5468813Z subprocess.CalledProcessError: Command '['coverage', 'run', '-m', 'monai.bundle', 'run', '--postprocessing#transforms#2#output_postfix', 'seg', '--network', '%/tmp/tmpdrak6f7u/override1.JSON#move_net', '--dataset#_target_', '%/tmp/tmpdrak6f7u/jsons/override2.JSON', '--no_epoch', 'False', '--output_dir', '/tmp/tmpdrak6f7u', '--device', "$torch.device('cpu')", '--meta_file', '/tmp/tmpdrak6f7u/meta.yaml', '--config_file', '/home/runner/work/MONAI/MONAI/tests/testing_data/inference.yaml', '--args_file', '/tmp/tmpdrak6f7u/def_args.json', '--tracking', '/tmp/tmpdrak6f7u/mlflow.json']' returned non-zero exit status 1.
2026-06-01T09:41:37.5470662Z 
2026-06-01T09:41:37.5470839Z The above exception was the direct cause of the following exception:
2026-06-01T09:41:37.5471115Z 
2026-06-01T09:41:37.5471215Z Traceback (most recent call last):
2026-06-01T09:41:37.5471804Z   File "/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/parameterized/parameterized.py", line 620, in standalone_func
2026-06-01T09:41:37.5472399Z     return func(*(a + p.args), **p.kwargs, **kw)
2026-06-01T09:41:37.5472931Z   File "/home/runner/work/MONAI/MONAI/tests/integration/test_integration_bundle_run.py", line 208, in test_shape
2026-06-01T09:41:37.5473573Z     command_line_tests(la + ["--args_file", def_args_file] + ["--tracking", settings_file])
2026-06-01T09:41:37.5474135Z   File "/home/runner/work/MONAI/MONAI/tests/test_utils.py", line 883, in command_line_tests
2026-06-01T09:41:37.5474727Z     raise RuntimeError(f"subprocess call error {e.returncode}: {errors}, {output}") from e
2026-06-01T09:41:37.5475253Z RuntimeError: subprocess call error 1: b'Traceback (most recent call last):
2026-06-01T09:41:37.5475786Z   File "/home/runner/work/MONAI/MONAI/monai/utils/module.py", line 245, in instantiate
2026-06-01T09:41:37.5476250Z     return component(**kwargs)
2026-06-01T09:41:37.5476668Z   File "/home/runner/work/MONAI/MONAI/monai/handlers/mlflow_handler.py", line 159, in __init__
2026-06-01T09:41:37.5477248Z     self.client = mlflow.MlflowClient(tracking_uri=tracking_uri if tracking_uri else None)
2026-06-01T09:41:37.5477927Z   File "/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/mlflow/tracking/client.py", line 248, in __init__
2026-06-01T09:41:37.5478530Z     self._tracking_client = TrackingServiceClient(final_tracking_uri)
2026-06-01T09:41:37.5479203Z   File "/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/mlflow/tracking/_tracking_service/client.py", line 99, in __init__
2026-06-01T09:41:37.5479772Z     self.store
2026-06-01T09:41:37.5480297Z   File "/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/mlflow/tracking/_tracking_service/client.py", line 103, in store
2026-06-01T09:41:37.5480959Z     return utils._get_store(self.tracking_uri)
2026-06-01T09:41:37.5481566Z   File "/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/mlflow/tracking/_tracking_service/utils.py", line 270, in _get_store
2026-06-01T09:41:37.5482220Z     return _tracking_store_registry.get_store(store_uri, artifact_uri)
2026-06-01T09:41:37.5482892Z   File "/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/mlflow/tracking/_tracking_service/registry.py", line 45, in get_store
2026-06-01T09:41:37.5483658Z     return self._get_store_with_resolved_uri(resolved_store_uri, artifact_uri)
2026-06-01T09:41:37.5484425Z   File "/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/mlflow/tracking/_tracking_service/registry.py", line 56, in _get_store_with_resolved_uri
2026-06-01T09:41:37.5485156Z     return builder(store_uri=resolved_store_uri, artifact_uri=artifact_uri)
2026-06-01T09:41:37.5485920Z   File "/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/mlflow/tracking/_tracking_service/utils.py", line 184, in _get_file_store
2026-06-01T09:41:37.5486700Z     return FileStore(store_uri, store_uri)
2026-06-01T09:41:37.5487272Z   File "/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/mlflow/store/tracking/file_store.py", line 225, in __init__
2026-06-01T09:41:37.5487848Z     raise MlflowException(
2026-06-01T09:41:37.5489711Z mlflow.exceptions.MlflowException: The filesystem tracking backend (e.g., \'./mlruns\') is in maintenance mode and will not receive further updates. Please migrate to a database backend (e.g., \'sqlite:///mlflow.db\') to access the latest MLflow features. The `mlflow migrate-filestore` tool migrates your existing data losslessly. See https://mlflow.org/docs/latest/self-hosting/migrate-from-file-store for migration guidance. If the filesystem backend is required for your workflow, set `MLFLOW_ALLOW_FILE_STORE=true` to opt out of this exception.
2026-06-01T09:41:37.5491447Z 
2026-06-01T09:41:37.5491635Z The above exception was the direct cause of the following exception:
2026-06-01T09:41:37.5491892Z 
2026-06-01T09:41:37.5492000Z Traceback (most recent call last):
2026-06-01T09:41:37.5501761Z   File "/home/runner/work/MONAI/MONAI/monai/bundle/__main__.py", line 31, in <module>
2026-06-01T09:41:37.5502277Z     fire.Fire()
2026-06-01T09:41:37.5502783Z   File "/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/fire/core.py", line 135, in Fire
2026-06-01T09:41:37.5503415Z     component_trace = _Fire(component, args, parsed_flag_args, context, name)
2026-06-01T09:41:37.5504023Z   File "/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/fire/core.py", line 468, in _Fire
2026-06-01T09:41:37.5504551Z     component, remaining_args = _CallAndUpdateTrace(
2026-06-01T09:41:37.5505136Z   File "/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
2026-06-01T09:41:37.5505683Z     component = fn(*varargs, **kwargs)
2026-06-01T09:41:37.5506189Z   File "/home/runner/work/MONAI/MONAI/monai/bundle/scripts.py", line 1012, in run
2026-06-01T09:41:37.5506582Z     workflow.run()
2026-06-01T09:41:37.5506938Z   File "/home/runner/work/MONAI/MONAI/monai/bundle/workflows.py", line 488, in run
2026-06-01T09:41:37.5507354Z     return self._run_expr(id=self.run_id)
2026-06-01T09:41:37.5507780Z   File "/home/runner/work/MONAI/MONAI/monai/bundle/workflows.py", line 532, in _run_expr
2026-06-01T09:41:37.5508275Z     ret.append(self.parser.get_parsed_content(sub_id, **kwargs))
2026-06-01T09:41:37.5508826Z   File "/home/runner/work/MONAI/MONAI/monai/bundle/config_parser.py", line 498, in get_parsed_content
2026-06-01T09:41:37.5509358Z     return self.ref_resolver.get_resolved_content(id=id, **kwargs)
2026-06-01T09:41:37.5509926Z   File "/home/runner/work/MONAI/MONAI/monai/bundle/reference_resolver.py", line 193, in get_resolved_content
2026-06-01T09:41:37.5510436Z     return self._resolve_one_item(id=id, **kwargs)
2026-06-01T09:41:37.5510948Z   File "/home/runner/work/MONAI/MONAI/monai/bundle/reference_resolver.py", line 163, in _resolve_one_item
2026-06-01T09:41:37.5511674Z     self._resolve_one_item(id=d, waiting_list=waiting_list, **kwargs)
2026-06-01T09:41:37.5512228Z   File "/home/runner/work/MONAI/MONAI/monai/bundle/reference_resolver.py", line 163, in _resolve_one_item
2026-06-01T09:41:37.5512765Z     self._resolve_one_item(id=d, waiting_list=waiting_list, **kwargs)
2026-06-01T09:41:37.5513310Z   File "/home/runner/work/MONAI/MONAI/monai/bundle/reference_resolver.py", line 171, in _resolve_one_item
2026-06-01T09:41:37.5513967Z     self.resolved_content[id] = item.instantiate() if kwargs.get("instantiate", True) else item
2026-06-01T09:41:37.5514574Z   File "/home/runner/work/MONAI/MONAI/monai/bundle/config_item.py", line 292, in instantiate
2026-06-01T09:41:37.5515029Z     return instantiate(modname, mode, **args)
2026-06-01T09:41:37.5515499Z   File "/home/runner/work/MONAI/MONAI/monai/utils/module.py", line 255, in instantiate
2026-06-01T09:41:37.5515917Z     raise RuntimeError(
2026-06-01T09:41:37.5516550Z RuntimeError: Failed to instantiate component \'monai.handlers.mlflow_handler.MLFlowHandler\' with keywords: tracking_uri,iteration_log
2026-06-01T09:41:37.5517159Z  set \'_mode_=debug\' to enter the debugging mode.

This can be produced when instantiating MLFlowHandler. This class needs to be updated to address this but still allows the filesystem backend if MLFLOW_ALLOW_FILE_STORE=true is set.

[Lawson-Darrow]

On mlflow 3.13 the file store now raises instead of warning. FileStore.__init__ raises unless MLFLOW_ALLOW_FILE_STORE=true, and that fires for any file backend, including MONAI's default ./mlruns and an explicit file:// URI, so MLFlowHandler breaks whenever no database tracking URI is set.

There seem to be three ways to handle it and I am not sure which you would prefer:

1. Keep the file-store default by opting back in: the handler sets MLFLOW_ALLOW_FILE_STORE=true when the resolved backend is a local file path, preserving current behavior on 3.13+ with no user-facing change.
2. Move the default to a sqlite backend, which is the direction mlflow is steering toward. More future proof but it changes the on-disk layout and default for existing users.
3. Pin mlflow<3.13 in the test requirements as a stopgap, which unblocks CI but does not fix it for users on 3.13+.

I have a small patch ready for option 1 if that is the direction you want. Happy to go with 2 instead if you would rather migrate the default. Which would you prefer?