At export time, models should be able to specify version(s) of a framework that the model is compatible with.
When we load a model, Neuropod will attempt to find a backend that satisfies the specified range.
This combined with OPE will enable environments to provide many supported versions of each framework and have models choose which one to use.
This is useful in several cases:
- If a model uses custom ops that are only compatible with one version of the framework (e.g. custom ops built against TF 1.15 headers vs TF 1.13.1 headers)
- Being able to ensure that runtime and test environments are as similar as possible
- Avoiding bugs in a specific version of the underlying framework
Target version ranges are specified as semver ranges (see https://semver.org/, https://docs.npmjs.com/misc/semver#ranges, https://docs.npmjs.com/misc/semver#advanced-range-syntax) which provides a very flexible way of defining the required framework.
TODO:
Once this is production ready, using dockerized OPE worker processes will also let us specify required CUDA versions (if any) and provide even more isolation.
At export time, models should be able to specify version(s) of a framework that the model is compatible with.
When we load a model, Neuropod will attempt to find a backend that satisfies the specified range.
This combined with OPE will enable environments to provide many supported versions of each framework and have models choose which one to use.
This is useful in several cases:
Target version ranges are specified as semver ranges (see https://semver.org/, https://docs.npmjs.com/misc/semver#ranges, https://docs.npmjs.com/misc/semver#advanced-range-syntax) which provides a very flexible way of defining the required framework.
TODO:
sofiles depending on the version of the framework they're built against (e.g.libneuropod_torchscript_1_4_0_cpu_backend.so)Once this is production ready, using dockerized OPE worker processes will also let us specify required CUDA versions (if any) and provide even more isolation.