-
Notifications
You must be signed in to change notification settings - Fork 828
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #138 from cliveseldon/java_wrappers
Java wrappers
- Loading branch information
Showing
58 changed files
with
2,851 additions
and
1 deletion.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,216 @@ | ||
# Packaging a Java model for Seldon Core using s2i | ||
|
||
|
||
In this guide, we illustrate the steps needed to wrap your own Java model in a docker image ready for deployment with Seldon Core using [source-to-image app s2i](https://github.com/openshift/source-to-image). | ||
|
||
If you are not familar with s2i you can read [general instructions on using s2i](./s2i.md) and then follow the steps below. | ||
|
||
|
||
# Step 1 - Install s2i | ||
|
||
[Download and install s2i](https://github.com/openshift/source-to-image#installation) | ||
|
||
* Prequisites for using s2i are: | ||
* Docker | ||
* Git (if building from a remote git repo) | ||
|
||
To check everything is working you can run | ||
|
||
```bash | ||
s2i usage seldonio/seldon-core-s2i-java-build | ||
``` | ||
|
||
# Step 2 - Create your source code | ||
|
||
To use our s2i builder image to package your Java model you will need: | ||
|
||
* A Maven project that depends on ```io.seldon.wrapper``` library | ||
* A Spring Boot configuration class | ||
* A class that implements ```io.seldon.wrapper.SeldonPredictionService``` for the type of component you are creating | ||
* .s2i/environment - model definitions used by the s2i builder to correctly wrap your model | ||
|
||
We will go into detail for each of these steps: | ||
|
||
## Maven Project | ||
Create a Spring Boot Maven project and include the dependency: | ||
|
||
```XML | ||
<dependency> | ||
<groupId>io.seldon.wrapper</groupId> | ||
<artifactId>seldon-core-wrapper</artifactId> | ||
<version>0.1.0</version> | ||
</dependency> | ||
``` | ||
|
||
A full example can be found at ```wrappers/s2i/java/test/model-template-app/pom.xml```. | ||
|
||
## Spring Boot Intialization | ||
|
||
Create a main App class: | ||
* Add @EnableAsync annotation (to allow the embedded gRPC server to start at Spring Boot startup) | ||
* include the ```io.seldon.wrapper``` in the scan base packages list along with your App's package, in the example below the Apps's package is ```io.seldon.example```. | ||
* Import the config class at ```io.seldon.wrapper.config.AppConfig.class``` | ||
|
||
For example: | ||
|
||
```java | ||
@EnableAsync | ||
@SpringBootApplication(scanBasePackages = {"io.seldon.wrapper","io.seldon.example"}) | ||
@Import({ io.seldon.wrapper.config.AppConfig.class }) | ||
public class App { | ||
public static void main(String[] args) throws Exception { | ||
SpringApplication.run(App.class, args); | ||
} | ||
} | ||
``` | ||
|
||
## Prediction Class | ||
To handle requests to your model or other component you need to implement one or more of the methods in ```io.seldon.wrapper.SeldonPredictionService```, in particular: | ||
|
||
```java | ||
default public SeldonMessage predict(SeldonMessage request); | ||
default public SeldonMessage route(SeldonMessage request); | ||
default public SeldonMessage sendFeedback(Feedback request); | ||
default public SeldonMessage transformInput(SeldonMessage request); | ||
default public SeldonMessage transformOutput(SeldonMessage request); | ||
default public SeldonMessage aggregate(SeldonMessageList request); | ||
``` | ||
|
||
Your implementing class should be created as a Spring Component so it will be managed by Spring. There is a full H2O example in ```examples/models/h2o_mojo/src/main/java/io/seldon/example/h2o/model```, whose implmentation is show below: | ||
|
||
```java | ||
@Component | ||
public class H2OModelHandler implements SeldonPredictionService { | ||
private static Logger logger = LoggerFactory.getLogger(H2OModelHandler.class.getName()); | ||
EasyPredictModelWrapper model; | ||
|
||
public H2OModelHandler() throws IOException { | ||
MojoReaderBackend reader = | ||
MojoReaderBackendFactory.createReaderBackend( | ||
getClass().getClassLoader().getResourceAsStream( | ||
"model.zip"), | ||
MojoReaderBackendFactory.CachingStrategy.MEMORY); | ||
MojoModel modelMojo = ModelMojoReader.readFrom(reader); | ||
model = new EasyPredictModelWrapper(modelMojo); | ||
logger.info("Loaded model"); | ||
} | ||
|
||
@Override | ||
public SeldonMessage predict(SeldonMessage payload) { | ||
List<RowData> rows = H2OUtils.convertSeldonMessage(payload.getData()); | ||
List<AbstractPrediction> predictions = new ArrayList<>(); | ||
for(RowData row : rows) | ||
{ | ||
try | ||
{ | ||
BinomialModelPrediction p = model.predictBinomial(row); | ||
predictions.add(p); | ||
} catch (PredictException e) { | ||
logger.info("Error in prediction ",e); | ||
} | ||
} | ||
DefaultData res = H2OUtils.convertH2OPrediction(predictions, payload.getData()); | ||
|
||
return SeldonMessage.newBuilder().setData(res).build(); | ||
} | ||
|
||
} | ||
|
||
``` | ||
|
||
The above code: | ||
|
||
* loads a model from the local resources folder on startup | ||
* Converts the proto buffer message into H2O RowData using provided utility classes. | ||
* Runs a BionomialModel prediction and converts the result back into a ```SeldonMessage``` for return | ||
|
||
### H2O Helper Classes | ||
|
||
We provide H2O utility class ```io.seldon.wrapper.utils.H2OUtils``` in seldon-core-wrapper to convert to and from the seldon-core proto buffer message types. | ||
|
||
### DL4J Helper Classes | ||
|
||
We provide a DL4J utility class ```io.seldon.wrapper.utils.DL4JUtils``` in seldon-core-wrapper to convert to and from the seldon-core proto buffer message types. | ||
|
||
## .s2i/environment | ||
|
||
Define the core parameters needed by our R builder image to wrap your model. An example is: | ||
|
||
```bash | ||
API_TYPE=REST | ||
SERVICE_TYPE=MODEL | ||
``` | ||
|
||
These values can also be provided or overriden on the command line when building the image. | ||
|
||
# Step 3 - Build your image | ||
Use ```s2i build``` to create your Docker image from source code. You will need Docker installed on the machine and optionally git if your source code is in a public git repo. | ||
|
||
Using s2i you can build directly from a git repo or from a local source folder. See the [s2i docs](https://github.com/openshift/source-to-image/blob/master/docs/cli.md#s2i-build) for further details. The general format is: | ||
|
||
```bash | ||
s2i build <git-repo> seldonio/seldon-core-s2i-java-build <my-image-name> --runtime-image seldonio/seldon-core-s2i-java-runtime | ||
s2i build <src-folder> seldonio/seldon-core-s2i-java-build <my-image-name> --runtime-image seldonio/seldon-core-s2i-java-runtime | ||
``` | ||
|
||
An example invocation using the test template model inside seldon-core: | ||
|
||
```bash | ||
s2i build https://github.com/seldonio/seldon-core.git --context-dir=wrappers/s2i/python/test/model-template-app seldonio/seldon-core-s2i-java-build h2o-test:0.1 --runtime-image seldonio/seldon-core-s2i-java-runtime | ||
``` | ||
|
||
The above s2i build invocation: | ||
|
||
* uses the GitHub repo: https://github.com/seldonio/seldon-core.git and the directory ```wrappers/s2i/R/test/model-template-app``` inside that repo. | ||
* uses the builder image ```seldonio/seldon-core-s2i-java-build``` | ||
* uses the runtime image ```seldonio/seldon-core-s2i-java-runtime``` | ||
* creates a docker image ```seldon-core-template-model``` | ||
|
||
|
||
For building from a local source folder, an example where we clone the seldon-core repo: | ||
|
||
```bash | ||
git clone https://github.com/seldonio/seldon-core.git | ||
cd seldon-core | ||
s2i build wrappers/s2i/R/test/model-template-app seldonio/seldon-core-s2i-java-build h2o-test:0.1 --runtime-image seldonio/seldon-core-s2i-java-runtime | ||
``` | ||
|
||
For more help see: | ||
|
||
``` | ||
s2i usage seldonio/seldon-core-s2i-java-build | ||
s2i build --help | ||
``` | ||
|
||
# Reference | ||
|
||
## Environment Variables | ||
The required environment variables understood by the builder image are explained below. You can provide them in the ```.s2i/enviroment``` file or on the ```s2i build``` command line. | ||
|
||
|
||
### API_TYPE | ||
|
||
API type to create. Can be REST or GRPC. | ||
|
||
### SERVICE_TYPE | ||
|
||
The service type being created. Available options are: | ||
|
||
* MODEL | ||
* ROUTER | ||
* TRANSFORMER | ||
* COMBINER | ||
|
||
|
||
## Creating different service types | ||
|
||
### MODEL | ||
|
||
* [A minimal skeleton for model source code](https://github.com/cliveseldon/seldon-core/tree/s2i/wrappers/s2i/java/test/model-template-app) | ||
* [Example H2O MOJO](https://github.com/SeldonIO/seldon-core/tree/master/examples/models/h2o-mojo/README.md) | ||
|
||
|
||
|
||
|
||
|
||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
/target/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,2 @@ | ||
API_TYPE=REST | ||
SERVICE_TYPE=MODEL |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,13 @@ | ||
|
||
|
||
build_jar: | ||
mvn clean package | ||
|
||
clean: | ||
mvn clean | ||
rm -rf experiment | ||
rm -f src/main/resources/model.zip | ||
train: | ||
mkdir -p experiment | ||
python train.py | ||
mv experiment/*.zip src/main/resources/model.zip |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,83 @@ | ||
# H2O MOJO Model | ||
An example H2O model | ||
|
||
## Dependencies | ||
|
||
* [H2O python library](http://docs.h2o.ai/h2o/latest-stable/h2o-docs/downloading.html#install-in-python) | ||
|
||
## Train locally | ||
|
||
```bash | ||
make train | ||
``` | ||
|
||
## Wrap using [s2i](https://github.com/openshift/source-to-image#installation). | ||
|
||
```bash | ||
s2i build . seldonio/seldon-core-s2i-java-build h2o-test:0.1 --runtime-image seldonio/seldon-core-s2i-java-runtime | ||
``` | ||
|
||
## Local Docker Smoke Test | ||
|
||
Run under docker. | ||
|
||
```bash | ||
docker run --rm -p 5000:5000 h2o-test:0.1 | ||
``` | ||
|
||
Ensure test grpc modules compiled. | ||
|
||
```bash | ||
pushd ../../../wrappers/testing ; make build_protos ; popd | ||
``` | ||
|
||
Send a data request using the wrapper tester. | ||
|
||
```bash | ||
python ../../../wrappers/testing/tester.py contract.json 0.0.0.0 5000 -p | ||
``` | ||
|
||
## Minikube test | ||
|
||
```bash | ||
minikube start --memory 4096 | ||
``` | ||
|
||
[Install seldon core](/readme.md#install) | ||
|
||
Connect to Minikube Docker daemon | ||
|
||
```bash | ||
eval $(minikube docker-env) | ||
``` | ||
|
||
Build image using minikube docker daemon. | ||
|
||
```bash | ||
s2i build . seldonio/seldon-core-s2i-java-build h2o-test:0.1 --runtime-image seldonio/seldon-core-s2i-java-runtime | ||
``` | ||
|
||
Launch deployment | ||
|
||
```bash | ||
kubectl create -f h2o_deployment.json | ||
``` | ||
|
||
Port forward API server | ||
|
||
```bash | ||
kubectl port-forward $(kubectl get pods -n seldon -l app=seldon-apiserver-container-app -o jsonpath='{.items[0].metadata.name}') -n seldon 8080:8080 | ||
``` | ||
|
||
Ensure tester gRPC modules compiled. | ||
|
||
```bash | ||
pushd ../../../util/api_tester ; make build_protos ; popd | ||
``` | ||
|
||
Send test request | ||
```bash | ||
python ../../../util/api_tester/api-tester.py contract.json 0.0.0.0 8080 --oauth-key oauth-key --oauth-secret oauth-secret -p | ||
``` | ||
|
||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,42 @@ | ||
{ | ||
"features":[ | ||
{ | ||
"name":"AGE", | ||
"dtype":"FLOAT", | ||
"ftype":"continuous", | ||
"range":[20,80] | ||
}, | ||
{ | ||
"name":"RACE", | ||
"ftype":"categorical", | ||
"values":[0,1,2] | ||
}, | ||
{ | ||
"name":"DCAPS", | ||
"ftype":"categorical", | ||
"values":[0,1,2] | ||
}, | ||
{ | ||
"name":"VOL", | ||
"ftype":"categorical", | ||
"values":[0,1,2] | ||
}, | ||
{ | ||
"name":"GLEASON", | ||
"dtype":"FLOAT", | ||
"ftype":"continuous", | ||
"range":[0,10] | ||
} | ||
], | ||
"targets":[ | ||
{ | ||
"name":"class", | ||
"dtype":"FLOAT", | ||
"ftype":"continuous", | ||
"range":[0,1], | ||
"repeat":2 | ||
} | ||
] | ||
} | ||
|
||
|
Oops, something went wrong.