Has anyone tried to deploy ludwig serve to a Google Vertex model endpoint?

[danielduckworth]

Hi folks, I've been following a guide on deploying custom models for serving on Vertex. The guide is Training, Tuning and Deploying a PyTorch Text Classification Model on Vertex AI.

In particular, the section at the end, "Invoking the Endpoint with deployed Model using Vertex SDK to make predictions".

So far I've managed to:

1. train a text classification model with ludwig locally
2. run ludwig serve locally
3. create a python client to request predictions from ludwig serve locally
4. create a ludwig docker container that includes a trained model (this is based on the ludwig containers on docker hub) that runs ludwig serve with the trained model
5. run the docker container locally and use the python client to request predictions
6. submit the docker container to the Google Vertex container registry
7. deploy the docker container as a custom model in Google Vertex
8. create a Google Vertex endpoint and attach the deployed model

From here I can make requests to the endpoint, but I always get the following response:

{"error":"entry must contain all input features"}

This happens locally if the request is not formatted correctly. For example, the expected format for my model is:

{"textfeature": "words to be classified"}

Vertex expects a bit of additional JSON and provides the following example for plain text:

{"instances": ["the quick brown fox", "the lazy dog"]}

Source: Get online predictions from custom-trained models

The instances key is required and is used to structure multiple inputs in a request.

I have tried including the feature name as follows:

{
  "instances": [
    {
      "textfeature": "Words to be classified"
    }
  ]
}

However I still get error response with "entry must contain all input features". I'm not sure how to progress from here.

I've created an issue in the Google Cloud Platform github also: Custom model deployed with a docker container but requests are not working as expected

Any ideas?

[brightsparc]

Hi, the default ludwig serve endpoint expects inputs to be provided as form fields. This is not a supported interface for Vertex AI prediction requests which requires Content-Type: application/json.

We are working on updating the serve capability in ludwig to be compatible with the KServe v2 Prediction Portocol which is a support format with Vertex AI and will remove the need to build a custom container to host ludwig models with Vertex AI.

In the meantime you could explore using TorchServe to deploy a PyTorch model to Vertex AI. See the following code to save the traced torchscript model which will include pre-processing for numeric and category fields:

inputs = {
    'cat_feature': ['foo', 'bar']
    'num_feature': torch.tensor([42, 7])
    'bin_feature1': torch.tensor([True, False])
    'bin_feature2': ['No', 'Yes']
}

# create torchscript model
scripted_model = model.to_torchscript()

# test the model 
output = scripted_model(inputs)

# save the traced torchscript model
scripted_model.save("/home/model-server/pytorch_model.pt")

You can then use the torch-model-archiver to package your model along with the handler required for Vertex AI.

torch-model-archiver -f \
  --model-name=$APP_NAME \
  --version=1.0 \
  --serialized-file=/home/model-server/pytorch_model.pt \
  --handler=/home/model-server/custom_handler.py \
  --export-path=/home/model-server/model-store

[danielduckworth]

Thanks! I'll give it a go.