Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MLFlow server-- ModuleNotFoundError: No module named 'prediction' #2874

Closed
jonathanfostercannon opened this issue Jan 23, 2021 · 2 comments
Closed

MLFlow server-- ModuleNotFoundError: No module named 'prediction' #2874

jonathanfostercannon opened this issue Jan 23, 2021 · 2 comments
Labels
bug triage Needs to be triaged and prioritised accordingly

Comments

@jonathanfostercannon
Copy link

jonathanfostercannon commented Jan 23, 2021 •
edited
Loading

Hi, I am trying to run an mlflow server on Seldon core but I’m getting a PYTHONPATH error, the application isn’t able to resolve the modules in the original python project. It is able to download the artifact and create the conda env.

I’ve tried to figure out the filesystem inside the container but I haven’t been able to resolve it on my own.

I’ve tried to point the Python path on the resource definition like this, with several variations, but haven’t been able to get it to work:

                env:
                  - name: PYTHONPATH
                    value: '/mnt/models/'

Here’s the full stacktrace:

Activating Conda environment 'mlflow'
starting microservice
2021-01-23 19:00:46,394 - seldon_core.microservice:main:201 - INFO:  Starting microservice.py:main
2021-01-23 19:00:46,395 - seldon_core.microservice:main:202 - INFO:  Seldon Core version: 1.5.0
2021-01-23 19:00:46,396 - seldon_core.microservice:main:314 - INFO:  Parse JAEGER_EXTRA_TAGS []
2021-01-23 19:00:46,397 - seldon_core.microservice:load_annotations:153 - INFO:  Found annotation cni.projectcalico.org/podIP:100.98.224.204/32 
2021-01-23 19:00:46,397 - seldon_core.microservice:load_annotations:153 - INFO:  Found annotation cni.projectcalico.org/podIPs:100.98.224.204/32 
2021-01-23 19:00:46,397 - seldon_core.microservice:load_annotations:153 - INFO:  Found annotation kubernetes.io/config.seen:2021-01-23T18:58:25.565000940Z 
2021-01-23 19:00:46,397 - seldon_core.microservice:load_annotations:153 - INFO:  Found annotation kubernetes.io/config.source:api 
2021-01-23 19:00:46,397 - seldon_core.microservice:load_annotations:153 - INFO:  Found annotation prometheus.io/path:/prometheus 
2021-01-23 19:00:46,397 - seldon_core.microservice:load_annotations:153 - INFO:  Found annotation prometheus.io/scrape:true 
2021-01-23 19:00:46,397 - seldon_core.microservice:load_annotations:153 - INFO:  Found annotation seldon.io/grpc-max-message-size:10000000 
2021-01-23 19:00:46,397 - seldon_core.microservice:load_annotations:153 - INFO:  Found annotation seldon.io/grpc-read-timeout:15000000 
2021-01-23 19:00:46,397 - seldon_core.microservice:load_annotations:153 - INFO:  Found annotation seldon.io/rest-read-timeout:15000000 
2021-01-23 19:00:46,397 - seldon_core.microservice:load_annotations:153 - INFO:  Found annotation seldon.io/rest-timeout:15000000 
2021-01-23 19:00:46,397 - seldon_core.microservice:main:317 - INFO:  Annotations: {'cni.projectcalico.org/podIP': '100.98.224.204/32', 'cni.projectcalico.org/podIPs': '100.98.224.204/32', 'kubernetes.io/config.seen': '2021-01-23T18:58:25.565000940Z', 'kubernetes.io/config.source': 'api', 'prometheus.io/path': '/prometheus', 'prometheus.io/scrape': 'true', 'seldon.io/grpc-max-message-size': '10000000', 'seldon.io/grpc-read-timeout': '15000000', 'seldon.io/rest-read-timeout': '15000000', 'seldon.io/rest-timeout': '15000000'}
2021-01-23 19:00:46,397 - seldon_core.microservice:main:321 - INFO:  Importing MLFlowServer
2021-01-23 19:00:46,775 - root:__init__:21 - INFO:  Creating MLFLow server with URI /mnt/models
2021-01-23 19:00:46,775 - root:__init__:22 - INFO:  xtype: ndarray
2021-01-23 19:00:46,787 - seldon_core.microservice:main:403 - INFO:  REST gunicorn microservice running on port 9000
2021-01-23 19:00:46,788 - seldon_core.microservice:main:457 - INFO:  REST metrics microservice running on port 6000
2021-01-23 19:00:46,788 - seldon_core.microservice:main:467 - INFO:  Starting servers
2021-01-23 19:00:46,797 - seldon_core.wrapper:_set_flask_app_configs:208 - INFO:  App Config:  <Config {'ENV': 'production', 'DEBUG': False, 'TESTING': False, 'PROPAGATE_EXCEPTIONS': None, 'PRESERVE_CONTEXT_ON_EXCEPTION': None, 'SECRET_KEY': None, 'PERMANENT_SESSION_LIFETIME': datetime.timedelta(days=31), 'USE_X_SENDFILE': False, 'SERVER_NAME': None, 'APPLICATION_ROOT': '/', 'SESSION_COOKIE_NAME': 'session', 'SESSION_COOKIE_DOMAIN': None, 'SESSION_COOKIE_PATH': None, 'SESSION_COOKIE_HTTPONLY': True, 'SESSION_COOKIE_SECURE': False, 'SESSION_COOKIE_SAMESITE': None, 'SESSION_REFRESH_EACH_REQUEST': True, 'MAX_CONTENT_LENGTH': None, 'SEND_FILE_MAX_AGE_DEFAULT': datetime.timedelta(seconds=43200), 'TRAP_BAD_REQUEST_ERRORS': None, 'TRAP_HTTP_EXCEPTIONS': False, 'EXPLAIN_TEMPLATE_LOADING': False, 'PREFERRED_URL_SCHEME': 'http', 'JSON_AS_ASCII': True, 'JSON_SORT_KEYS': True, 'JSONIFY_PRETTYPRINT_REGULAR': False, 'JSONIFY_MIMETYPE': 'application/json', 'TEMPLATES_AUTO_RELOAD': None, 'MAX_COOKIE_SIZE': 4093}>
2021-01-23 19:00:46,801 - seldon_core.wrapper:get_grpc_server:271 - INFO:  Setting grpc max message and receive length to 10000000
2021-01-23 19:00:46,807 - root:load:28 - INFO:  Downloading model from /mnt/models
2021-01-23 19:00:46,807 - root:download:31 - INFO:  Copying contents of /mnt/models to local
2021-01-23 19:00:46,818 - seldon_core.wrapper:_set_flask_app_configs:208 - INFO:  App Config:  <Config {'ENV': 'production', 'DEBUG': False, 'TESTING': False, 'PROPAGATE_EXCEPTIONS': None, 'PRESERVE_CONTEXT_ON_EXCEPTION': None, 'SECRET_KEY': None, 'PERMANENT_SESSION_LIFETIME': datetime.timedelta(days=31), 'USE_X_SENDFILE': False, 'SERVER_NAME': None, 'APPLICATION_ROOT': '/', 'SESSION_COOKIE_NAME': 'session', 'SESSION_COOKIE_DOMAIN': None, 'SESSION_COOKIE_PATH': None, 'SESSION_COOKIE_HTTPONLY': True, 'SESSION_COOKIE_SECURE': False, 'SESSION_COOKIE_SAMESITE': None, 'SESSION_REFRESH_EACH_REQUEST': True, 'MAX_CONTENT_LENGTH': None, 'SEND_FILE_MAX_AGE_DEFAULT': datetime.timedelta(seconds=43200), 'TRAP_BAD_REQUEST_ERRORS': None, 'TRAP_HTTP_EXCEPTIONS': False, 'EXPLAIN_TEMPLATE_LOADING': False, 'PREFERRED_URL_SCHEME': 'http', 'JSON_AS_ASCII': True, 'JSON_SORT_KEYS': True, 'JSONIFY_PRETTYPRINT_REGULAR': False, 'JSONIFY_MIMETYPE': 'application/json', 'TEMPLATES_AUTO_RELOAD': None, 'MAX_COOKIE_SIZE': 4093}>
[2021-01-23 19:00:46 +0000] [7] [INFO] Starting gunicorn 20.0.4
[2021-01-23 19:00:46 +0000] [7] [INFO] Listening at: http://0.0.0.0:9000 (7)
[2021-01-23 19:00:46 +0000] [7] [INFO] Using worker: threads
[2021-01-23 19:00:46 +0000] [1425] [INFO] Starting gunicorn 20.0.4
[2021-01-23 19:00:46 +0000] [1425] [INFO] Listening at: http://0.0.0.0:6000 (1425)
[2021-01-23 19:00:46 +0000] [1425] [INFO] Using worker: sync
[2021-01-23 19:00:46 +0000] [1432] [INFO] Booting worker with pid: 1432
[2021-01-23 19:00:46 +0000] [1431] [INFO] Booting worker with pid: 1431
2021-01-23 19:00:46,843 - seldon_core.app:load:82 - INFO:  Tracing branch is active
2021-01-23 19:00:46,848 - seldon_core.utils:setup_tracing:724 - INFO:  Initializing tracing
2021-01-23 19:00:46,881 - seldon_core.utils:setup_tracing:731 - INFO:  Using default tracing config
2021-01-23 19:00:46,881 - jaeger_tracing:_create_local_agent_channel:446 - INFO:  Initializing Jaeger Tracer with UDP reporter
2021-01-23 19:00:46,885 - jaeger_tracing:new_tracer:384 - INFO:  Using sampler ConstSampler(True)
2021-01-23 19:00:46,886 - jaeger_tracing:_initialize_global_tracer:436 - INFO:  opentracing.tracer initialized to <jaeger_client.tracer.Tracer object at 0x7f9c3eadb7d0>[app_name=MLFlowServer]
2021-01-23 19:00:46,886 - seldon_core.app:load:87 - INFO:  Set JAEGER_EXTRA_TAGS []
2021-01-23 19:00:46,887 - root:load:28 - INFO:  Downloading model from /mnt/models
2021-01-23 19:00:46,887 - root:download:31 - INFO:  Copying contents of /mnt/models to local
Process Process-2:
Traceback (most recent call last):
  File "/opt/conda/envs/mlflow/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/opt/conda/envs/mlflow/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/conda/envs/mlflow/lib/python3.7/site-packages/seldon_core/microservice.py", line 425, in grpc_prediction_server
    user_object.load()
  File "/microservice/MLFlowServer.py", line 30, in load
    self._model = pyfunc.load_model(model_folder)
  File "/opt/conda/envs/mlflow/lib/python3.7/site-packages/mlflow/pyfunc/__init__.py", line 481, in load_model
    model_impl = importlib.import_module(conf[MAIN])._load_pyfunc(data_path)
  File "/opt/conda/envs/mlflow/lib/python3.7/site-packages/mlflow/pytorch/__init__.py", line 460, in _load_pyfunc
    return _PyTorchWrapper(_load_model(path, **kwargs))
  File "/opt/conda/envs/mlflow/lib/python3.7/site-packages/mlflow/pytorch/__init__.py", line 393, in _load_model
    return torch.load(model_path, **kwargs)
  File "/opt/conda/envs/mlflow/lib/python3.7/site-packages/torch/serialization.py", line 594, in load
    return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
  File "/opt/conda/envs/mlflow/lib/python3.7/site-packages/torch/serialization.py", line 853, in _load
    result = unpickler.load()
ModuleNotFoundError: No module named 'prediction'
[2021-01-23 19:00:47 +0000] [1431] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/opt/conda/envs/mlflow/lib/python3.7/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
    worker.init_process()
  File "/opt/conda/envs/mlflow/lib/python3.7/site-packages/gunicorn/workers/gthread.py", line 92, in init_process
    super().init_process()
  File "/opt/conda/envs/mlflow/lib/python3.7/site-packages/gunicorn/workers/base.py", line 119, in init_process
    self.load_wsgi()
  File "/opt/conda/envs/mlflow/lib/python3.7/site-packages/gunicorn/workers/base.py", line 144, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/opt/conda/envs/mlflow/lib/python3.7/site-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/opt/conda/envs/mlflow/lib/python3.7/site-packages/seldon_core/app.py", line 92, in load
    self.user_object.load()
  File "/microservice/MLFlowServer.py", line 30, in load
    self._model = pyfunc.load_model(model_folder)
  File "/opt/conda/envs/mlflow/lib/python3.7/site-packages/mlflow/pyfunc/__init__.py", line 481, in load_model
    model_impl = importlib.import_module(conf[MAIN])._load_pyfunc(data_path)
  File "/opt/conda/envs/mlflow/lib/python3.7/site-packages/mlflow/pytorch/__init__.py", line 460, in _load_pyfunc
    return _PyTorchWrapper(_load_model(path, **kwargs))
  File "/opt/conda/envs/mlflow/lib/python3.7/site-packages/mlflow/pytorch/__init__.py", line 393, in _load_model
    return torch.load(model_path, **kwargs)
  File "/opt/conda/envs/mlflow/lib/python3.7/site-packages/torch/serialization.py", line 594, in load
    return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
  File "/opt/conda/envs/mlflow/lib/python3.7/site-packages/torch/serialization.py", line 853, in _load
    result = unpickler.load()
ModuleNotFoundError: No module named 'prediction'
[2021-01-23 19:00:47 +0000] [1431] [INFO] Worker exiting (pid: 1431)
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/opt/conda/envs/mlflow/lib/python3.7/site-packages/mlflow/utils/file_utils.py", line 111, in mkdir
    raise e
  File "/opt/conda/envs/mlflow/lib/python3.7/site-packages/mlflow/utils/file_utils.py", line 108, in mkdir
    os.makedirs(target)
  File "/opt/conda/envs/mlflow/lib/python3.7/os.py", line 223, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/microservice/mlruns'
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/opt/conda/envs/mlflow/lib/python3.7/multiprocessing/util.py", line 357, in _exit_function
    p.join()
  File "/opt/conda/envs/mlflow/lib/python3.7/multiprocessing/process.py", line 138, in join
    assert self._parent_pid == os.getpid(), 'can only join a child process'
AssertionError: can only join a child process
[2021-01-23 19:00:47 +0000] [1425] [INFO] Handling signal: term
[2021-01-23 19:00:47 +0000] [1432] [INFO] Worker exiting (pid: 1432)
[2021-01-23 19:00:47 +0000] [1425] [INFO] Shutting down: Master
[2021-01-23 19:00:47 +0000] [7] [INFO] Shutting down: Master
[2021-01-23 19:00:47 +0000] [7] [INFO] Reason: Worker failed to boot.
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/opt/conda/envs/mlflow/lib/python3.7/site-packages/mlflow/utils/file_utils.py", line 111, in mkdir
    raise e
  File "/opt/conda/envs/mlflow/lib/python3.7/site-packages/mlflow/utils/file_utils.py", line 108, in mkdir
    os.makedirs(target)
  File "/opt/conda/envs/mlflow/lib/python3.7/os.py", line 223, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/microservice/mlruns'

Appreciate any assistance.

Thanks
@jonathanfostercannon jonathanfostercannon added bug triage Needs to be triaged and prioritised accordingly labels Jan 23, 2021
@nickhill97
Copy link

I'm facing the same error. Did you have a fix for this?

@deepanshutyagi18
Copy link

@jonathanfostercannon Since I'm experiencing the same difficulty, could you please give me the solution?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug triage Needs to be triaged and prioritised accordingly
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@nickhill97 @jonathanfostercannon @deepanshutyagi18