mlserver-mlflow ROCK returns response in different format than upstream during integration tests

[orfeas-k]

Context

After omitting the activation of environment in the ROCK's command due to #70, we noticed that during integration tests of mlserver-mlflow ROCK, the output of the upstream SeldonDeployment and the ROCK we use was different.

This can also be observed in changes we 've done in the past. When integrating the ROCK for 1.7, we changed the expected output. And we essentailly reverted this change in the PR that updated Seldon from 1.15 (using ROCKs) to 1.17 (using upstream images). This difference has also been documented here.

The result of the metric is essentially the same but the differences are in other parameters of the response. Here's the tests' output:

AssertionError: assert [('id', 'None'), ('model_name', 'classifier'), ('model_version', 'v1'), ('outputs', [{'name': 'output-1', 'shape': [1, 1], 'datatype': 'FP64', 'parameters': {'content_type': 'np'}, 'data': [6.016145744177844]}]), ('parameters', {'content_type': 'np'})] == [('id', 'None'), ('model_name', 'classifier'), ('model_version', 'v1'), ('outputs', [{'name': 'predict', 'shape': [1], 'datatype': 'FP64', 'parameters': None, 'data': [6.016145744177844]}]), ('parameters', None)]
  At index 3 diff: ('outputs', [{'name': 'output-1', 'shape': [1, 1], 'datatype': 'FP64', 'parameters': {'content_type': 'np'}, 'data': [6.016145744177844]}]) != ('outputs', [{'name': 'predict', 'shape': [1], 'datatype': 'FP64', 'parameters': None, 'data': [6.016145744177844]}])
  Full diff:
    [
     ('id', 'None'),
     ('model_name', 'classifier'),
     ('model_version', 'v1'),
     ('outputs',
      [{'data': [6.016145744177844],
        'datatype': 'FP64',
-       'name': 'predict',
?                 ^^^^^
+       'name': 'output-1',
?                +++ ^ ++
-       'parameters': None,
+       'parameters': {'content_type': 'np'},
-       'shape': [1]}]),
+       'shape': [1, 1]}]),
?                 +++
-    ('parameters', None),
+    ('parameters', {'content_type': 'np'}),
    ]

This could be due to different packages installed. Keep in mind that we 're using a different Python version to run the deployment's .py file (logs from the SeldonDeployment pod)

2023-12-13T10:12:18.469Z [mlserver-mlflow] 2023/12/13 10:12:18 WARNING mlflow.pyfunc: The version of Python that the model was saved in, `Python 3.7.10`, differs from the version of Python that is currently running, `Python 3.8.16`, and may be incompatible

Additional Context

For a reference, here are the output of pip freeze from upstream image and our ROCK, where we can see most of the packages having been bumped.
upstream-pip-freeze.txt
rock-no-env-pip-freeze.txt

Here's the full logs of the classifier container of the seldonDeployment's pod

╰─$ kl mlflow-default-0-classifier-cd8b874b4-48gtr                                                            1 ↵
Defaulted container "classifier" out of: classifier, seldon-container-engine, classifier-model-initializer (init)
2023-12-13T10:12:14.621Z [pebble] Started daemon.
2023-12-13T10:12:14.630Z [pebble] POST /v1/services 8.759098ms 202
2023-12-13T10:12:14.631Z [pebble] Started default services with change 1.
2023-12-13T10:12:14.635Z [pebble] Service "mlserver-mlflow" starting: bash -c 'export PATH=/opt/conda/bin/:/opt/mlserver/.local/bin:${PATH}:/usr/bin && export PYTHONPATH=/opt/mlserver/.local/lib/python3.8/site-packages/:${PYTHONPATH} && eval $(/opt/conda/bin/conda shell.bash hook 2> /dev/null) && mlserver start ${MLSERVER_MODELS_DIR}'
2023-12-13T10:12:16.696Z [mlserver-mlflow] 2023-12-13 10:12:16,696 [mlserver.parallel] DEBUG - Starting response processing loop...
2023-12-13T10:12:16.697Z [mlserver-mlflow] /opt/mlserver/.local/lib/python3.8/site-packages/starlette_exporter/middleware.py:97: FutureWarning: group_paths and filter_unhandled_paths will change defaults from False to True in the next release. See https://github.com/stephenhillier/starlette_exporter/issues/79 for more info
2023-12-13T10:12:16.697Z [mlserver-mlflow]   warnings.warn(
2023-12-13T10:12:16.697Z [mlserver-mlflow] 2023-12-13 10:12:16,697 [mlserver.rest] INFO - HTTP server running on http://0.0.0.0:9000
2023-12-13T10:12:16.720Z [mlserver-mlflow] INFO:     Started server process [16]
2023-12-13T10:12:16.720Z [mlserver-mlflow] INFO:     Waiting for application startup.
2023-12-13T10:12:16.733Z [mlserver-mlflow] 2023-12-13 10:12:16,733 [mlserver.metrics] INFO - Metrics server running on http://0.0.0.0:6000
2023-12-13T10:12:16.733Z [mlserver-mlflow] 2023-12-13 10:12:16,733 [mlserver.metrics] INFO - Prometheus scraping endpoint can be accessed on http://0.0.0.0:6000/prometheus
2023-12-13T10:12:16.733Z [mlserver-mlflow] INFO:     Started server process [16]
2023-12-13T10:12:16.733Z [mlserver-mlflow] INFO:     Waiting for application startup.
2023-12-13T10:12:17.786Z [mlserver-mlflow] INFO:     Application startup complete.
2023-12-13T10:12:17.787Z [mlserver-mlflow] 2023-12-13 10:12:17,787 [mlserver.grpc] INFO - gRPC server running on http://0.0.0.0:9500
2023-12-13T10:12:17.787Z [mlserver-mlflow] INFO:     Application startup complete.
2023-12-13T10:12:17.787Z [mlserver-mlflow] INFO:     Uvicorn running on http://0.0.0.0:9000 (Press CTRL+C to quit)
2023-12-13T10:12:17.787Z [mlserver-mlflow] INFO:     Uvicorn running on http://0.0.0.0:6000 (Press CTRL+C to quit)
2023-12-13T10:12:18.469Z [mlserver-mlflow] 2023/12/13 10:12:18 WARNING mlflow.pyfunc: The version of Python that the model was saved in, `Python 3.7.10`, differs from the version of Python that is currently running, `Python 3.8.16`, and may be incompatible
2023-12-13T10:12:18.744Z [mlserver-mlflow] 2023-12-13 10:12:18,744 [mlserver] INFO - Loaded model 'classifier' succesfully.
2023-12-13T10:12:18.746Z [mlserver-mlflow] 2023-12-13 10:12:18,746 [mlserver] INFO - Loaded model 'classifier' succesfully.
2023-12-13T10:12:28.772Z [mlserver-mlflow] INFO:     192.168.2.3:57036 - "GET /v2/health/ready HTTP/1.1" 200 OK
2023-12-13T10:12:28.773Z [mlserver-mlflow] INFO:     192.168.2.3:57048 - "GET /v2/health/ready HTTP/1.1" 200 OK
2023-12-13T10:12:29.163Z [mlserver-mlflow] INFO:     192.168.2.3:42970 - "POST /v2/models/classifier/infer HTTP/1.1" 200 OK