You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the solution you'd like
Use python RPC library to create a RPC server which automatically loads all methods implemented of the model implementation (that it wraps around).
Describe alternatives you've considered
We could also consider wrapping the model implementation in an RPC client. This however would cause large overhead on the prediction backend side as well as considerable traffic overhead. Since we would need to constantly ask the prediction backend if jobs are available.
Additional context
This will be part of the next milestone. Python3 RPC Server
Tasks
Load all functions of the underlying model implementation
Use TLS
Only accept a connection from the prediction backend (e.g., use the public key of the prediction backend)
Accept client requests and execute function calls
The text was updated successfully, but these errors were encountered:
We chose to not use RPC for the sake of cleaner code and less overhead.
Instead we use FastAPI and two separate Routes, which are added depending on the settings.node_type variable.
It might be possible to implement the middleware again for routing, iff an alternative approach to consuming the body (encode/starlette#495) is found. However the two routes setup allows for easier understanding of the code which might also be preferable.
Closing as this was done in 84b37a8 and ultimatly finished in #24
Describe the solution you'd like
Use python RPC library to create a RPC server which automatically loads all methods implemented of the model implementation (that it wraps around).
Describe alternatives you've considered
We could also consider wrapping the model implementation in an RPC client. This however would cause large overhead on the prediction backend side as well as considerable traffic overhead. Since we would need to constantly ask the prediction backend if jobs are available.
Additional context
This will be part of the next milestone.
Python3 RPC Server
Tasks
The text was updated successfully, but these errors were encountered: