Python version mismatch after loading a Conda env

[GDegrove]

Hello,

I'm trying to run a MLFlow saved pyfunc model. This model is fairly simple, it's a linear regression with a custom identity transformer for the pipeline. The idea is to demonstrate how to load custom code at runtime.

The upload/download of the artifact works perfectly.

Now comes the weird part.

My model run with python 3.11.2, this version is saved by MLFlow as conda environment. I packed the conda env using conda-pack and it's being uploaded as an artifact.

This artifact is downloaded with the model, and I see that the conda env is being loaded by MLServer.

In the logs of the mlserver docker container, I see:

--> Unpacking environment at /mnt/models/environment.tar.gz...
--> Sourcing new environment at ./envs/environment...
--> Calling conda-unpack...
--> Disabling user-installed packages...

Then, when I start mlserver using, then I see this in the logs:

2023/10/27 14:57:47 WARNING mlflow.pyfunc: Detected one or more mismatches between the model's dependencies and the current Python environment:
 - mlflow (current: 2.4.1, required: mlflow==2.5.0)
 - numpy (current: 1.24.3, required: numpy==1.24.4)
 - scikit-learn (current: 1.3.0, required: scikit-learn==1.2.2)
 - scipy (current: 1.10.1, required: scipy==1.9.3)
To fix the mismatches, call `mlflow.pyfunc.get_model_dependencies(model_uri)` to fetch the model's environment and install dependencies using the resulting environment file.
2023/10/27 14:57:47 WARNING mlflow.pyfunc: The version of Python that the model was saved in, `Python 3.11.2`, differs from the version of Python that is currently running, `Python 3.8.16`, and may be incompatible

This seems odd as the conda env should be correctly setup.

Here is the MLModel attached to this particular model:

artifact_path: model
flavors:
  python_function:
    cloudpickle_version: 2.2.1
    code: code
    env:
      conda: conda.yaml
      virtualenv: python_env.yaml
    loader_module: mlflow.pyfunc.model
    python_model: python_model.pkl
    python_version: 3.11.2
mlflow_version: 2.5.0
model_uuid: 8f5c6ad749ad48f6ada69b82b2daa1c0
run_id: d6cef80c75c24b55ab90982a9cfe41b5
signature:
  inputs: '[{"type": "double", "name": "acc_rate"}, {"type": "long", "name": "avg_daily_trips"},
    {"type": "double", "name": "conv_rate"}, {"type": "long", "name": "driver_id"}]'
  outputs: '[{"type": "tensor", "tensor-spec": {"dtype": "float64", "shape": [-1,
    1]}}]'
utc_time_created: '2023-08-18 12:31:23.791466'

I therefore wonder what am I missing in the model or in mlserver to make this working. I expected that with the conda.yaml and the correct env loaded, I would have the correct python env (python version and dependencies) to easily server a model.

Thanks already for the help (and the work done with this awesome project)

[agrski]

Hi @GDegrove,

I suspect the fundamental issue is that the model is built for Python 3.11 but the MLServer version is running Python 3.8. It should be possible to resolve this by using a newer version of Python for MLServer itself, or alternatively by using dependency versions compatible with Python 3.8.

I believe this happens because when MLServer spawns worker processes, these use the same Python version as the main process, which is whatever that version of MLServer was built with. The Conda environment is added to the system path, but does not change the Python version of the already running process using that system path.