• if you get validator error

pip install pydantic==1.10.9

• on ray cluster

serve run model_nf4_mg:chat_app_nf4_mg

• on ray cluster

serve run config.yaml

• testing

curl -X POST http://localhost:8000/ \
  -H "Content-Type: application/json" \
  -d '{"text": "How do I bake a pie?"}'