MLServer not working with deployments options with Ambassador, seldon-core

[kurianbenoy-sentient]

I have been trying to deploy a micro service we developed to use with MLServer. We have been deploying it previously with seldon-core and we are using ambassador as well.

seldon_deployment.yaml file is given below:

apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
  name: enmltranslation 
spec:
  protocol: kfserving
  annotations:
    project_name: enmltranslation
    deployment_version: v0.1.0
    seldon.io/rest-timeout: '60000'
    seldon.io/rest-connection-timeout: '60000'
    seldon.io/grpc-read-timeout: '60000'
    seldon.io/ambassador-config: |
      ---
      apiVersion: ambassador/v1
      kind: Mapping
      name: enmltrans_mapping_v0.1.0
      prefix: /microservices/nlp/enmltrans/v0/getpredictions
      service: enmltranslation-pod.default:8000
      rewrite: /nlp/enml/predictions

  predictors:
    - name: default
      graph:
        name: manglish-model 
        type: MODEL
      componentSpecs:
        - spec:
            containers:
              - name: manglish-model  
                image: manglishdummy:v0.1.0 
                ports:
                  - containerPort: 8080
                    name: http
                    protocol: TCP
                  - containerPort: 8081
                    name: grpc
                    protocol: TCP

When accessing the URL via ambassador we are 503 HTTP error indicating service is unavailable.

Update: (April 1, 2022)

I was able to bring up a normal kubernetees deployment by following a deployment.yaml similar to the one provided in tutorial. Yet ambassador support for MLServer seems not working at the moment.

Update (April 20, 2022)

With help of @adriangonz solution, by passing no-executors in seldon-core we are now able to customize the URL with ambassador. Yet is it possible without by passing no-executors so that we can avail the seldon-core's graph functionality?

[adriangonz]

Hey @kurianbenoy-sentient ,

If you remove the seldon.io/ambassador-config entry from the annotations of your SeldonDeployment manifest, then Seldon Core should be able to figure out the right config to create. I'm wondering whether the config that you've added there may be causing any conflict. Would you be able to try that out @kurianbenoy-sentient ?

Besides the above, @kurianbenoy-sentient could you provide more details of:

• What endpoint are you using to hit your model?
• Is the pod running correctly?
• Has Seldon Core been installed correctly with Ambassador support?
• Is your Ambasador set up leveraging the new Ambassador Edge stack, which is not supported yet in Seldon Core?

PS: @kurianbenoy-sentient given that this feels more like an issue at the Seldon Core / Ambassador level (rather than MLServer's), you may be able to get further help raising it on the https://github.com/SeldonIO/seldon-core repo (or Seldon slack) instead.

[kurianbenoy-sentient]

If you remove the seldon.io/ambassador-config entry from the annotations of your SeldonDeployment manifest, then Seldon Core should be able to figure out the right config to create. I'm wondering whether the config that you've added there may be causing any conflict. Would you be able to try that out @kurianbenoy-sentient ?

We tried after removing seldon.io/ambassador-config from annotations of seldonDeployment, using the modified seldon-deployment file as follows in given link.

Besides the above, @kurianbenoy-sentient could you provide more details of:

* What endpoint are you using to hit your model?

We are trying to use the model by using custom_methods, instead of the default predict method in MLServer/seldon-core. This is the code for the method:

"""Translation model - Inference class - seldon MLServer"""
from typing import Any

from transformers import AutoTokenizer
from transformers import AutoModelForSeq2SeqLM
from transformers import pipeline
from fastapi import Request

from mlserver import MLModel
from mlserver import types
from mlserver.handlers import custom_handler
from mlserver.types import InferenceRequest, InferenceResponse
from mlserver.errors import MLServerError

# from mlserver.types import InferenceRequest, InferenceResponse


class Enmltrans(MLModel):
    """English - Malayalam translation models in MLSERVER"""

    async def load(self, model_name="Helsinki-NLP/opus-mt-en-ml"):
        self.tokenizer = AutoTokenizer.from_pretrained(model_name)
        self.model = AutoModelForSeq2SeqLM.from_pretrained(model_name)
        self.translator = pipeline(
            "translation", model=self.model, tokenizer=self.tokenizer
        )

        self.ready = True
        return self.ready

    @custom_handler(rest_path="/nlp/enml/predictions", rest_method="POST")
    async def predict_raw(self, request: Request) -> Any:
        """The predict method in custom format like predict_raw"""
        inputjson = await request.json()
        output = self.translator(inputjson["text"])
        return output[0]

* Is the pod running correctly?

Yes, the pod is running correctly with no errors.

* Has Seldon Core been installed correctly with Ambassador support?

Yes. Normal seldon-core installation with similar ambassador configuration works correctly when I tried out. This issue occurs only in case of seldon-core.

* Is your Ambasador set up leveraging the [new Ambassador Edge stack](https://www.getambassador.io/products/edge-stack/), which is [not supported yet in Seldon Core](https://github.com/SeldonIO/seldon-core/issues/3872)?

Now we are using Ambassador with a version supported by seldon-core, and normal seldon-core stuff work with ambassador. We are not using Ambassador Edge stack.

Thanks for your detailed reply @adriangonz

[adriangonz]

@kurianbenoy-sentient could you share the full endpoint you're using to hit your model (including the custom path)?

This should be something like /seldon/<namespace>/<model-name>/nlp/enml/predictions. Also, it's worth noting that custom endpoints are probably not forwarded by the executor in Seldon Core, so you may also need to bypass the executor (this can be done through an annotation on the SeldonDeployment manifest).

[kurianbenoy-sentient]

@adriangonz we tried with the endpoint: http://host/seldon/default/enmltranslation-v0-1-0/nlp/enml/predictions.

Inorder to by pass the executor what needs to be done? In normal seldon-core due to this issue, we are actually using annotations with rewrite:

    seldon.io/ambassador-config: |
      ---
      apiVersion: ambassador/v1
      kind: Mapping
      name: enmltrans_mapping_v0.1.0
      prefix: /microservices/nlp/enmltrans/v0/getpredictions
      service: enmltranslation-pod.default:8000
      rewrite: /nlp/enml/predictions

[adriangonz]

Hey @kurianbenoy-sentient, you can find more info on bypassing the executor in the Seldon Core docs:

https://docs.seldon.io/projects/seldon-core/en/latest/graph/svcorch.html#bypass-service-orchestrator-version-0-5-0

I'd try that first, removing the Ambassador annotation. Mainly to avoid getting side effects.

[kurianbenoy-sentient]

Today after bypassing the executor in seldon core, by following the link you shared. We were able to access our custom API endpoint via ambassador with the url /seldon/<namespace>/<model-name>/nlp/enml/predictions.

@adriangonz We want to have the abilitiy to rewrite the endpoint from /seldon/<namespace>/<model-name>/nlp/enml/predictions to a custom endpoint in ambassador like /microservices/<url>. Is it possible?

[kurianbenoy-sentient]

We were able to customize the URL in ambassador, by using no-executors with the below configuration. Yet is it possible without using no-executors so that we can avail the seldon-core's graph functionality? Or with no-executors is graph functionality available ?

cc: @adriangonz

apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
  name: enmltranslation
  labels:
    model_name: enmltranslation
spec:
  name: enmltranslation
  annotations:
    seldon.io/ambassador-config: |
      ---
      apiVersion: ambassador/v1
      kind: Mapping
      name: enmltrans_mapping_v0.1.0
      prefix: /microservices/nlp/enmltrans/v0/getpredictions
      service: enmltranslation-pod.default:8080
      rewrite: /nlp/enml/predictions
  predictors:
  -  annotations:
       seldon.io/no-engine: "true" 
(rest same as previously)

[adriangonz]

Hey @kurianbenoy-sentient ,

On the question about the URL rewrite, I'm not an expert on Ambassador, but perhaps the Ambassador community may be better at answering that one?

On the second one, if you disable the executor you will lose the graph functionality on that particular model. The problem with custom endpoints is that the model's requests and responses become a black box, so there's no way for the Seldon Core executor to know how to chain them, and propagate them through the graph.

Given that the pending remaining issues seem to be related to other projects (i.e. Ambassador and Seldon Core), I'll be closing this ticket. Feel free to open up a new one if you find any other MLServer issue or if you've any extra questions.