url is hardcoded to http://fastapi:8000 but how to handle not using docker compose?

[robmarkcole]

The current approach of hardcoding http://fastapi:8000 requires the use of docker compose, but how do we also handle the case that users are running the containers seperately? A situation where this occurs might be if you want to run the pytorch model on a dedicate GPU machine, and have this resource shared by multiple services (e.g.a web UI, some analytics tools). One option I used here is to pass the machine IP in via an environment variable like:

    environment: 
      - DEEPSTACK_IP=deepstack

Anyway I raise this in the hope you have an even better suggestion. It would be great to identify the optimum solutions and share this back with the community :-)

[davidefiocco]

Hi @robmarkcole sorry for the late reply, this comes late as I didn't feel confident about "state of the art" patterns to deal with ML serving beyond the simple example in the repo.

The current solution could indeed be modified to run the containers on different hosts. The backend container would sit on host A, and then the frontend container running on host B would need to be allowed to make requests to the backend. When deploying the frontend, one would just need to add the address of the host running the backend service (as you do) with an env variable referenced by the code.

I haven't tried that yet, but I assume that a better way to serve the (PyTorch) model is to use https://pytorch.org/serve/rest_api.html (which comes with OpenAPI documentation) and have the frontend sit on another machine allowing to make requests to the API exposed by TorchServe. Unless I am mistaken, this solution wouldn't need to use FastAPI though.