[BUG]: Cannot load service with unregistered service type

[mushfiqulIslam]

Contact Details [Optional]

No response

System Information

ZenML version: 0.23.0
Python version: 3.8.13
Platform information: {'os': 'linux', 'linux_distro': 'ubuntu', 'linux_distro_like': 'debian', 'linux_distro_version': '22.04'}
Environment: native
Integrations: ['evidently', 'mlflow', 'plotly', 'scipy', 'sklearn', 'tensorboard', 'tensorflow']

What happened?

I have created custom model deployer flavor and wrote all functionalities by following mlflow model deployer steps. When I run the pipeline it says.
TypeError: Cannot load service with unregistered service type: type='model-serving' flavor='demo' name='demo-deployment' description='Demo detection service'
I have debug to findout the reason of this error, noticed that ServiceRegistry class get_service_type()
method returns none value for demo service and the error triggered.
I searched about registering the service on the doc but did not find any information about it.

Reproduction steps

1. Prepared a demo model deployer by flollowing mlflow model deployer
2. Prepare an api service with flask same as mlflow
3. registered the flavor
4. now add it to the pipline and run
   ...

Relevant log output

Updating an existing Demo deployment service: DemoDeploymentService[547692ed-8f3c-4938-b1a0-42cf73821c86] (type: model-serving, flavor: demo)
Timed out waiting for service DemoDeploymentService[547692ed-8f3c-4938-b1a0-42cf73821c86] (type: model-serving, flavor: demo) to become active:
  Administrative state: active
  Operational state: inactive
  Last status message: 'service daemon is not running'
For more information on the service status, please see the following log file: /home/mushfiq/.config/zenml/local_stores/634ae6dd-fa52-4871-a827-afd1a4494676/547692ed-8f3c-4938-b1a0-42cf73821c86/service.log

╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /home/mushfiq/Projects/zenml-test/demo/demo_detection/run.py:21 in <module>   │
│                                                                              │
│   18                                                                         │
│   19                                                                         │
│   20 if __name__ == "__main__":                                              │
│ ❱ 21 │   main()                                                              │
│   22                                                                         │
│                                                                              │
│ /home/mushfiq/Projects/zenml-test/demo/demo_detection/run.py:9 in main        │
│                                                                              │
│    6                                                                         │
│    7 def main():                                                             │
│    8 │   # initialize and run the training pipeline                          │
│ ❱  9 │   lstm_training_pipeline(                                             │
│   10 │   │   training_data_loader=training_data_loader(),                    │
│   11 │   │   feature_generator=feature_generator(),                          │
│   12 │   │   pre_processor=pre_processor(),                                  │
│                                                                              │
│ /home/mushfiq/Projects/zenml-test/venv/lib/python3.8/site-packages/zenml/pip │
│ elines/base_pipeline.py:526 in run                                           │
│                                                                              │
│   523 │   │   # behavior                                                     │
│   524 │   │   constants.SHOULD_PREVENT_PIPELINE_EXECUTION = True             │
│   525 │   │   try:                                                           │
│ ❱ 526 │   │   │   return_value = stack.deploy_pipeline(pipeline_deployment)  │
│   527 │   │   finally:                                                       │
│   528 │   │   │   constants.SHOULD_PREVENT_PIPELINE_EXECUTION = False        │
│   529                                                                        │
│                                                                              │
│ /home/mushfiq/Projects/zenml-test/venv/lib/python3.8/site-packages/zenml/sta │
│ ck/stack.py:677 in deploy_pipeline                                           │
│                                                                              │
│   674 │   │   Returns:                                                       │
│   675 │   │   │   The return value of the call to `orchestrator.run_pipeline │
│   676 │   │   """                                                            │
│ ❱ 677 │   │   return self.orchestrator.run(deployment=deployment, stack=self │
│   678 │                                                                      │
│   679 │   def _get_active_components_for_step(                               │
│   680 │   │   self, step_config: "StepConfiguration"                         │
│                                                                              │
│ /home/mushfiq/Projects/zenml-test/venv/lib/python3.8/site-packages/zenml/orc │
│ hestrators/base_orchestrator.py:286 in run                                   │
│                                                                              │
│   283 │   │   """                                                            │
│   284 │   │   self._prepare_run(deployment=deployment)                       │
│   285 │   │                                                                  │
│ ❱ 286 │   │   result = self.prepare_or_run_pipeline(                         │
│   287 │   │   │   deployment=deployment, stack=stack                         │
│   288 │   │   )                                                              │
│   289                                                                        │
│                                                                              │
│ /home/mushfiq/Projects/zenml-test/venv/lib/python3.8/site-packages/zenml/orc │
│ hestrators/local/local_orchestrator.py:76 in prepare_or_run_pipeline         │
│                                                                              │
│    73 │   │   │   │   │   step.config.name,                                  │
│    74 │   │   │   │   )                                                      │
│    75 │   │   │                                                              │
│ ❱  76 │   │   │   self.run_step(                                             │
│    77 │   │   │   │   step=step,                                             │
│    78 │   │   │   )                                                          │
│    79                                                                        │
│                                                                              │
│ /home/mushfiq/Projects/zenml-test/venv/lib/python3.8/site-packages/zenml/orc │
│ hestrators/base_orchestrator.py:377 in run_step                              │
│                                                                              │
│   374 │   │   │   stack.prepare_step_run(info=step_run_info)                 │
│   375 │   │   │   step_failed = False                                        │
│   376 │   │   │   try:                                                       │
│ ❱ 377 │   │   │   │   execution_info = self._execute_step(component_launcher │
│   378 │   │   │   except:  # noqa: E722                                      │
│   379 │   │   │   │   self._publish_failed_run(run_name_or_id=run_model.id)  │
│   380 │   │   │   │   step_failed = True                                     │
│                                                                              │
│ /home/mushfiq/Projects/zenml-test/venv/lib/python3.8/site-packages/zenml/orc │
│ hestrators/base_orchestrator.py:556 in _execute_step                         │
│                                                                              │
│   553 │   │   │   "_publish_failed_execution",                               │
│   554 │   │   │   types.MethodType(_new_publish_failed_execution, tfx_launch │
│   555 │   │   )                                                              │
│ ❱ 556 │   │   execution_info = tfx_launcher.launch()                         │
│   557 │   │   if execution_failed:                                           │
│   558 │   │   │   raise RuntimeError(                                        │
│   559 │   │   │   │   "Failed to execute step. This is probably because some │
│                                                                              │
│ /home/mushfiq/Projects/zenml-test/venv/lib/python3.8/site-packages/tfx/orche │
│ stration/portable/launcher.py:549 in launch                                  │
│                                                                              │
│   546 │   │     self._executor_operator.with_execution_watcher(              │
│   547 │   │   │     executor_watcher.address)                                │
│   548 │   │     executor_watcher.start()                                     │
│ ❱ 549 │   │   executor_output = self._run_executor(execution_info)           │
│   550 │     except Exception as e:  # pylint: disable=broad-except           │
│   551 │   │   execution_output = (                                           │
│   552 │   │   │   e.executor_output if isinstance(e, _ExecutionFailedError)  │
│                                                                              │
│ /home/mushfiq/Projects/zenml-test/venv/lib/python3.8/site-packages/tfx/orche │
│ stration/portable/launcher.py:424 in _run_executor                           │
│                                                                              │
│   421 │                                                                      │
│   422 │   outputs_utils.make_output_dirs(execution_info.output_dict)         │
│   423 │   try:                                                               │
│ ❱ 424 │     executor_output = self._executor_operator.run_executor(execution │
│   425 │     code = executor_output.execution_result.code                     │
│   426 │     if code != 0:                                                    │
│   427 │   │   result_message = executor_output.execution_result.result_messa │
│                                                                              │
│ /home/mushfiq/Projects/zenml-test/venv/lib/python3.8/site-packages/tfx/orche │
│ stration/portable/python_executor_operator.py:135 in run_executor            │
│                                                                              │
│   132 │   │   pipeline_info=execution_info.pipeline_info,                    │
│   133 │   │   pipeline_run_id=execution_info.pipeline_run_id)                │
│   134 │   executor = self._executor_cls(context=context)                     │
│ ❱ 135 │   return run_with_executor(execution_info, executor)                 │
│   136                                                                        │
│                                                                              │
│ /home/mushfiq/Projects/zenml-test/venv/lib/python3.8/site-packages/tfx/orche │
│ stration/portable/python_executor_operator.py:58 in run_with_executor        │
│                                                                              │
│    55 │   │   artifact.read()                                                │
│    56                                                                        │
│    57   output_dict = copy.deepcopy(execution_info.output_dict)              │
│ ❱  58   result = executor.Do(execution_info.input_dict, output_dict,         │
│    59 │   │   │   │   │      execution_info.exec_properties)                 │
│    60   if not result:                                                       │
│    61 │   # If result is not returned from the Do function, then try to      │
│                                                                              │
│ /home/mushfiq/Projects/zenml-test/venv/lib/python3.8/site-packages/zenml/ste │
│ ps/utils.py:461 in Do                                                        │
│                                                                              │
│   458 │   │   │   step_run_info=step_run_info,                               │
│   459 │   │   │   cache_enabled=self.configuration.enable_cache,             │
│   460 │   │   ):                                                             │
│ ❱ 461 │   │   │   return_values = step_function(**function_params)           │
│   462 │   │                                                                  │
│   463 │   │   output_annotations = parse_return_type_annotations(spec.annota │
│   464 │   │   if len(output_annotations) > 0:                                │
│                                                                              │
│ /home/mushfiq/Projects/zenml-test/blackwidow/steps/demo_deployer.py:114 in    │
│ demo_model_deployer_step                                                      │
│                                                                              │
│   111 │                                                                      │
│   112 │   new_service = cast(                                                │
│   113 │   │   DemoDeploymentService,                                          │
│ ❱ 114 │   │   active_model_deployer.deploy_model(                            │
│   115 │   │   │   replace=True,                                              │
│   116 │   │   │   config=predictor_cfg,                                      │
│   117 │   │   │   timeout=params.timeout,                                    │
│                                                                              │
│ /home/mushfiq/Projects/zenml-test/blackwidow/model_deployers/demo_model_deplo │
│ yer.py:147 in deploy_model                                                   │
│                                                                              │
│   144 │   │   │   config.root_runtime_path = self.local_path                 │
│   145 │   │   │   service.stop(timeout=timeout, force=True)                  │
│   146 │   │   │   service.update(config)                                     │
│ ❱ 147 │   │   │   service.start(timeout=timeout)                             │
│   148 │   │   else:                                                          │
│   149 │   │   │   # create a new DemoDeploymentService instance               │
│   150 │   │   │   service = self._create_new_service(timeout, config)        │
│                                                                              │
│ /home/mushfiq/Projects/zenml-test/venv/lib/python3.8/site-packages/zenml/ser │
│ vices/local/local_service.py:432 in start                                    │
│                                                                              │
│   429 │   │   │   │   the service status.                                    │
│   430 │   │   """                                                            │
│   431 │   │   if not self.config.blocking:                                   │
│ ❱ 432 │   │   │   super().start(timeout)                                     │
│   433 │   │   else:                                                          │
│   434 │   │   │   self.run()                                                 │
│   435                                                                        │
│                                                                              │
│ /home/mushfiq/Projects/zenml-test/venv/lib/python3.8/site-packages/zenml/ser │
│ vices/service.py:392 in start                                                │
│                                                                              │
│   389 │   │   │   self.provision()                                           │
│   390 │   │   │   if timeout > 0:                                            │
│   391 │   │   │   │   if not self.poll_service_status(timeout):              │
│ ❱ 392 │   │   │   │   │   raise RuntimeError(                                │
│   393 │   │   │   │   │   │   f"Failed to start service {self}\n"            │
│   394 │   │   │   │   │   │   + self.get_service_status_message()            │
│   395 │   │   │   │   │   )                                                  │
╰──────────────────────────────────────────────────────────────────────────────╯
RuntimeError: Failed to start service 
DemoDeploymentService[547692ed-8f3c-4938-b1a0-42cf73821c86] (type: model-serving,
flavor: demo)
  Administrative state: `active`
  Operational state: `inactive`
  Last status message: 'service daemon is not running'
For more information on the service status, please see the following log file: 
/home/mushfiq/.config/zenml/local_stores/634ae6dd-fa52-4871-a827-afd1a4494676/54
7692ed-8f3c-4938-b1a0-42cf73821c86/service.log


Process finished with exit code 1
And Traceback from log file:
Traceback (most recent call last):
  File "/home/mushfiq/Projects/zenml-test/venv/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/mushfiq/Projects/zenml-test/venv/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/mushfiq/Projects/zenml-test/venv/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/mushfiq/Projects/zenml-test/venv/lib/python3.8/site-packages/zenml/services/local/local_daemon_entrypoint.py", line 92, in run
    launch_service(config_file)
  File "/home/mushfiq/Projects/zenml-test/venv/lib/python3.8/site-packages/zenml/utils/daemon.py", line 99, in daemon
    run_as_daemon(
  File "/home/mushfiq/Projects/zenml-test/venv/lib/python3.8/site-packages/zenml/utils/daemon.py", line 270, in run_as_daemon
    daemon_function(*args, **kwargs)
  File "/home/mushfiq/Projects/zenml-test/venv/lib/python3.8/site-packages/zenml/services/local/local_daemon_entrypoint.py", line 84, in launch_service
    service = ServiceRegistry().load_service_from_json(config)
  File "/home/mushfiq/Projects/zenml-test/venv/lib/python3.8/site-packages/zenml/services/service_registry.py", line 212, in load_service_from_json
    return self.load_service_from_dict(service_dict)
  File "/home/mushfiq/Projects/zenml-test/venv/lib/python3.8/site-packages/zenml/services/service_registry.py", line 191, in load_service_from_dict
    raise TypeError(
TypeError: Cannot load service with unregistered service type: type='model-serving' flavor='demo' name='demo-deployment' description='Demo detection service'
Cleanup: terminating children processes...
Cleanup: removing PID file /home/mushfiq/.config/zenml/local_stores/634ae6dd-fa52-4871-a827-afd1a4494676/547692ed-8f3c-4938-b1a0-42cf73821c86/service.pid

Code of Conduct

• I agree to follow this project's Code of Conduct

[htahir1]

Thank you for the report! The ZenML team is away until 3rd January to respond to issues. We'll get to this report as soon as we're back! Sorry for the trouble!

[stefannica]

Hi @mushfiqulIslam , thank you for filing this issue! It looks like you're declaring a new custom type of daemon service that is not part of the ZenML codebase (i.e. part of an official ZenML integration). This is a current ZenML limitation that we're planning on removing in one of the upcoming releases.

All you need to do to add your service to the service registry is import the Python module where you defined your custom service class before the service is instantiated. For daemon services, this needs to be done in the ZenML daemon service entrypoint module (zenml.services.local.local_daemon_entrypoint).

As a workaround for you in the meantime, there are two possible solutions:

1. you could model your code as a ZenML integration as covered here: https://docs.zenml.io/misc/integrating
2. you could manually import your service in the local daemon service entrypoint file (/home/mushfiq/Projects/zenml-test/venv/lib/python3.8/site-packages/zenml/services/local/local_daemon_entrypoint.py in your case) by adding something like import zenml-test.blackwidow.service.my_service at the top

[mushfiqulIslam]

Thanks for the workaround.