[BUG] #1975

ankxyz · 2019-10-22T13:00:35Z

System information

Have I written custom code (as opposed to using a stock example script provided in MLflow): custom model training script
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 16.04 in Docker container
MLflow installed from (source or binary): source (mlflow repository, master branch)
MLflow version (run mlflow --version): 1.3.1.dev0
Python version: 3.7.3
npm version, if running the dev UI: 3.10.10
Exact command to reproduce: docker-compose up

Describe the problem

On open some run in mlflow ui:

2019/10/22 10:20:08 ERROR mlflow.server: Exception on /ajax-api/2.0/preview/mlflow/model-versions/search [GET]
 Traceback (most recent call last):
   File "/usr/local/lib/python3.7/site-packages/Flask-1.1.1-py3.7.egg/flask/app.py", line 2446, in wsgi_app
     response = self.full_dispatch_request()
   File "/usr/local/lib/python3.7/site-packages/Flask-1.1.1-py3.7.egg/flask/app.py", line 1951, in full_dispatch_request
     rv = self.handle_user_exception(e)
   File "/usr/local/lib/python3.7/site-packages/Flask-1.1.1-py3.7.egg/flask/app.py", line 1820, in handle_user_exception
     reraise(exc_type, exc_value, tb)
   File "/usr/local/lib/python3.7/site-packages/Flask-1.1.1-py3.7.egg/flask/_compat.py", line 39, in reraise
     raise value
   File "/usr/local/lib/python3.7/site-packages/Flask-1.1.1-py3.7.egg/flask/app.py", line 1949, in full_dispatch_request
     rv = self.dispatch_request()
   File "/usr/local/lib/python3.7/site-packages/Flask-1.1.1-py3.7.egg/flask/app.py", line 1935, in dispatch_request
     return self.view_functions[rule.endpoint](**req.view_args)
   File "/usr/local/lib/python3.7/site-packages/mlflow-1.3.1.dev0-py3.7.egg/mlflow/server/handlers.py", line 137, in wrapper
     return func(*args, **kwargs)
   File "/usr/local/lib/python3.7/site-packages/mlflow-1.3.1.dev0-py3.7.egg/mlflow/server/handlers.py", line 581, in _search_model_versions
     model_versions_detailed = _get_model_registry_store().search_model_versions(
 AttributeError: 'NoneType' object has no attribute 'search_model_versions'

On click Register Model:

Code to reproduce issue

Project sources mlflow_model_registry.zip

Archive content:

I. Dockerfile

FROM python
WORKDIR /home

# RUN pip install sklearn git+https://github.com/mparkhe/mlflow.git@ssue-19
RUN apt-get update && \
    curl -sL https://deb.nodesource.com/setup_6.x | bash - && \
    apt-get update && \
    apt-get install -y nodejs npm

RUN pip install sklearn

RUN git clone https://github.com/mlflow/mlflow && \
    cd mlflow && python setup.py install && \
    cd mlflow/server/js && npm install

ENV MLFLOW_TRACKING_URI=/home/mlruns

CMD mlflow server --host 0.0.0.0 --port 5000 & (cd mlflow/mlflow/server/js && npm start)

II. docker-compose.yml

version: '3'

services:
  mlflow_model_registry:
    build:
      context: .

    image: mlflow_model_registry:latest

    container_name: mlflow_model_registry_1

    ports:
      - 5000:5000
      - 3000:3000

    volumes:
      - ./main.py:/home/main.py
      - ./mlruns:/home/mlruns

III. main.py

import joblib
import mlflow
from mlflow.sklearn import log_model
from sklearn import svm
from sklearn import datasets


clf = svm.SVC(gamma='scale')
iris = datasets.load_iris()
X, y = iris.data, iris.target
clf.fit(X, y)

joblib.dump(clf, 'iris.model')

log_model('iris.model', 'model')

Run all:

build image and run container

docker-compose build
docker-compose up

enter docker container in new terminal window

docker exec -ti mlflow_model_registry_1 /bin/bash

run training script

python main.py

open dev UI

The text was updated successfully, but these errors were encountered:

harupy · 2019-10-22T13:05:18Z

@MeisterUrian I ran into the exact same issue. This is because Model Registry does not support file based stores yet. (https://github.com/mlflow/mlflow/blob/master/mlflow/server/handlers.py#L60)

I used my local PostgreSQL as a tracking server and it worked.

ankxyz · 2019-10-22T13:11:00Z

@harupy Thank you very much! I will try it some later.

UPD: it works! I have tried it with SQLite and it solved my problem.

Thanks again!

harupy · 2019-10-22T13:16:23Z

@MeisterUrian I followed this doc.

https://www.mlflow.org/docs/latest/tracking.html#logging-to-a-tracking-server

kadakia · 2019-11-05T06:36:43Z

@harupy Should model registry work if the tracking server is on an ec2 instance (with artifacts being logged to s3)? Because I'm running into the same issue, it seems.

harupy · 2019-11-05T06:48:32Z

@kadakia I haven't tested that case yet. How does your tracking server URI look like?

kadakia · 2019-11-05T06:57:46Z

@harupy Thanks for getting back to me so quickly. Can you clarify what you mean? Thanks.

kadakia · 2019-11-05T07:12:14Z

@harupy And when I try mlflow.sklearn.log_model(model, "model", registered_model_name="test-model"), I am seeing mlflow.exceptions.MlflowException: API request to https://<REMOTE_TRACKING_URI>/api/2.0/preview/mlflow/registered-models/create failed to return code 200 after 3 tries in my terminal

harupy · 2019-11-05T07:22:03Z

mlflow.exceptions.MlflowException: API request to https://<REMOTE_TRACKING_URI>/api/2.0/preview/mlflow/registered-models/create failed to return code 200 after 3 tries

Is this the whole error message?

kadakia · 2019-11-05T17:53:20Z

@harupy This is the script that I'm running. I just adapted one of the MLflow sample scripts.

from __future__ import print_function

import numpy as np
from sklearn.linear_model import LogisticRegression

import mlflow
import mlflow.sklearn
mlflow.set_tracking_uri("<REMOTE_TRACKING_URI>")
mlflow.set_experiment("registry-testing")

if __name__ == "__main__":
    X = np.array([-2, -1, 0, 1, 2, 1]).reshape(-1, 1)
    y = np.array([0, 0, 1, 1, 1, 0])
    lr = LogisticRegression()
    lr.fit(X, y)
    score = lr.score(X, y)
    print("Score: %s" % score)
    mlflow.log_metric("score", score)
    mlflow.sklearn.log_model(lr, "model", registered_model_name='test-model')
    print("Model saved in run %s" % mlflow.active_run().info.run_uuid)

And this is the whole error message:

Traceback (most recent call last):
  File "train.py", line 19, in <module>
    mlflow.sklearn.log_model(lr, "model", registered_model_name='test-model')
  File "/Users/rkadakia/miniconda3/envs/mlflow-4c48459e7b9deeb62ee26b15ac20a443c423e5f2/lib/python3.6/site-packages/mlflow/sklearn.py", line 200, in log_model
    registered_model_name=registered_model_name)
  File "/Users/rkadakia/miniconda3/envs/mlflow-4c48459e7b9deeb62ee26b15ac20a443c423e5f2/lib/python3.6/site-packages/mlflow/models/__init__.py", line 86, in log
    registered_model_name)
  File "/Users/rkadakia/miniconda3/envs/mlflow-4c48459e7b9deeb62ee26b15ac20a443c423e5f2/lib/python3.6/site-packages/mlflow/tracking/_model_registry/fluent.py", line 33, in register_model
    raise e
  File "/Users/rkadakia/miniconda3/envs/mlflow-4c48459e7b9deeb62ee26b15ac20a443c423e5f2/lib/python3.6/site-packages/mlflow/tracking/_model_registry/fluent.py", line 26, in register_model
    create_model_response = client.create_registered_model(name)
  File "/Users/rkadakia/miniconda3/envs/mlflow-4c48459e7b9deeb62ee26b15ac20a443c423e5f2/lib/python3.6/site-packages/mlflow/tracking/client.py", line 345, in create_registered_model
    return self._get_registry_client().create_registered_model(name)
  File "/Users/rkadakia/miniconda3/envs/mlflow-4c48459e7b9deeb62ee26b15ac20a443c423e5f2/lib/python3.6/site-packages/mlflow/tracking/_model_registry/client.py", line 37, in create_registered_model
    return self.store.create_registered_model(name)
  File "/Users/rkadakia/miniconda3/envs/mlflow-4c48459e7b9deeb62ee26b15ac20a443c423e5f2/lib/python3.6/site-packages/mlflow/store/model_registry/rest_store.py", line 48, in create_registered_model
    response_proto = self._call_endpoint(CreateRegisteredModel, req_body)
  File "/Users/rkadakia/miniconda3/envs/mlflow-4c48459e7b9deeb62ee26b15ac20a443c423e5f2/lib/python3.6/site-packages/mlflow/store/model_registry/rest_store.py", line 34, in _call_endpoint
    return call_endpoint(self.get_host_creds(), endpoint, method, json_body, response_proto)
  File "/Users/rkadakia/miniconda3/envs/mlflow-4c48459e7b9deeb62ee26b15ac20a443c423e5f2/lib/python3.6/site-packages/mlflow/utils/rest_utils.py", line 135, in call_endpoint
    host_creds=host_creds, endpoint=endpoint, method=method, json=json_body)
  File "/Users/rkadakia/miniconda3/envs/mlflow-4c48459e7b9deeb62ee26b15ac20a443c423e5f2/lib/python3.6/site-packages/mlflow/utils/rest_utils.py", line 80, in http_request
    (url, retries))
mlflow.exceptions.MlflowException: API request to <REMOTE_TRACKING_URI>/api/2.0/preview/mlflow/registered-models/create failed to return code 200 after 3 tries

Thanks for your help.

harupy · 2019-11-05T23:35:00Z

mlflow.set_tracking_uri("<REMOTE_TRACKING_URI>")

Did you replace <REMOTE_TRACKING_URI> with a valid URI when you ran the code?

kadakia · 2019-11-05T23:35:53Z

@harupy Yes

kadakia · 2019-11-06T01:46:53Z

@harupy Let me know if I can provide more information about the issue

harupy · 2019-11-06T07:37:32Z

@kadakia Got it

Crazybean-lwb · 2019-11-07T11:30:37Z

@MeisterUrian I ran into the exact same issue. This is because Model Registry does not support file based stores yet. (https://github.com/mlflow/mlflow/blob/master/mlflow/server/handlers.py#L60)

I used my local PostgreSQL as a tracking server and it worked.

I met the same problem, how can PostgreSQL save model flies, thanks!

Guoyoyo · 2019-11-13T09:12:53Z

I also tried postgresql to replace file-based and it worked. Just revise this line: --backend-store-uri postgresql://<user>:<password>@<ip>/<db>

SteNicholas · 2019-12-03T04:05:39Z

@MeisterUrian I ran into the exact same issue. This is because Model Registry does not support file based stores yet. (https://github.com/mlflow/mlflow/blob/master/mlflow/server/handlers.py#L60)

I used my local PostgreSQL as a tracking server and it worked.

@harupy When does Model Registry support file based stores? And MLflow Model Registry doesn't mention about this. It is necessary to tell the truth for new users that Model Registry doesn't support file based stores.

harupy · 2019-12-03T06:02:20Z

@SteNicholas

When does Model Registry support file based stores?

Sorry, I'm not sure...

SteNicholas · 2019-12-03T06:05:10Z

@SteNicholas

When does Model Registry support file based stores?

Sorry, I'm not sure...

Okay, thanks.

ankxyz added the bug Something isn't working label Oct 22, 2019

ankxyz closed this as completed Oct 22, 2019

harupy mentioned this issue Nov 3, 2019

[BUG] Model Registry problem with sklearn log method mlflow version 1.4.0 #2037

Closed

harupy mentioned this issue Nov 5, 2019

[BUG] can't register model in registry via code or UI #2047

Closed

harupy mentioned this issue Nov 12, 2019

[BUG]Model Registry is not working from the UI. #2079

Closed

ggliem mentioned this issue Jan 21, 2020

exception triggered with mlflow ui: AttributeError: 'NoneType' object has no attribute 'search_model_versions' #2322

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] #1975

[BUG] #1975

ankxyz commented Oct 22, 2019 •

edited

harupy commented Oct 22, 2019 •

edited

ankxyz commented Oct 22, 2019 •

edited

harupy commented Oct 22, 2019

kadakia commented Nov 5, 2019 •

edited

harupy commented Nov 5, 2019

kadakia commented Nov 5, 2019

kadakia commented Nov 5, 2019

harupy commented Nov 5, 2019

kadakia commented Nov 5, 2019 •

edited

harupy commented Nov 5, 2019

kadakia commented Nov 5, 2019

kadakia commented Nov 6, 2019

harupy commented Nov 6, 2019

Crazybean-lwb commented Nov 7, 2019

Guoyoyo commented Nov 13, 2019 •

edited

SteNicholas commented Dec 3, 2019 •

edited

harupy commented Dec 3, 2019 •

edited

SteNicholas commented Dec 3, 2019

[BUG] #1975

[BUG] #1975

Comments

ankxyz commented Oct 22, 2019 • edited

System information

Describe the problem

Code to reproduce issue

harupy commented Oct 22, 2019 • edited

ankxyz commented Oct 22, 2019 • edited

harupy commented Oct 22, 2019

kadakia commented Nov 5, 2019 • edited

harupy commented Nov 5, 2019

kadakia commented Nov 5, 2019

kadakia commented Nov 5, 2019

harupy commented Nov 5, 2019

kadakia commented Nov 5, 2019 • edited

harupy commented Nov 5, 2019

kadakia commented Nov 5, 2019

kadakia commented Nov 6, 2019

harupy commented Nov 6, 2019

Crazybean-lwb commented Nov 7, 2019

Guoyoyo commented Nov 13, 2019 • edited

SteNicholas commented Dec 3, 2019 • edited

harupy commented Dec 3, 2019 • edited

SteNicholas commented Dec 3, 2019

ankxyz commented Oct 22, 2019 •

edited

harupy commented Oct 22, 2019 •

edited

ankxyz commented Oct 22, 2019 •

edited

kadakia commented Nov 5, 2019 •

edited

kadakia commented Nov 5, 2019 •

edited

Guoyoyo commented Nov 13, 2019 •

edited

SteNicholas commented Dec 3, 2019 •

edited

harupy commented Dec 3, 2019 •

edited