UnknownHostException: seldon-deployment

[hanneshapke]

Running through the Keras MNIST deployment example, I experienced this error in the REST response below:

{u'code': 103,
 u'info': u'java.net.UnknownHostException: seldon-deployment: Try again',
 u'reason': u'Microservice error',
 u'status': u'FAILURE'}

Digging through the logs from the api-server I came across this error message:

2018-07-25 20:09:46.000  INFO 5 --- [nio-8080-exec-6] i.s.a.service.InternalPredictionService  : Requesting http://seldon-deployment:8000/api/v0.1/predictions
2018-07-25 20:09:50.010  INFO 5 --- [pool-3-thread-1] io.seldon.apife.k8s.DeploymentWatcher    : The time is now 20:09:50
2018-07-25 20:09:50.010  INFO 5 --- [pool-3-thread-1] io.seldon.apife.k8s.DeploymentWatcher    : Watching with rs 12371
2018-07-25 20:09:51.013 ERROR 5 --- [nio-8080-exec-6] i.s.a.service.InternalPredictionService  : Couldn't retrieve prediction from external prediction server - 

java.net.UnknownHostException: seldon-deployment: Try again
    at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method) ~[na:1.8.0_171]
        ...

Furthermore, the cluster manager logs listed the following error

22:01:15.709 [pool-1-thread-1] DEBUG i.s.c.k8s.SeldonDeploymentWatcher - Watching with rs null in namespace default
22:01:15.828 [pool-1-thread-1] DEBUG i.s.c.k8s.SeldonDeploymentWatcher - ADDED
 : {"apiVersion":"machinelearning.seldon.io/v1alpha2","kind":"SeldonDeployment","metadata":{"clusterName":"","creationTimestamp":"2018-07-25T22:01:07Z","labels":{"app":"seldon"},"name":"seldon-deployment-example","namespace":"default","resourceVersion":"1111","selfLink":"/apis/machinelearning.seldon.io/v1alpha2/namespaces/default/seldondeployments/seldon-deployment-example","uid":"3b580ceb-9056-11e8-a12d-080027068900"},"spec":{"annotations":{"deployment_version":"0.0","project_name":"Digits classification"},"name":"keras-mnist-deployment","oauth_key":"oauth-key","oauth_secret":"oauth-secret","predictors":[{"annotations":{"predictor_version":"0.0"},"componentSpecs":[{"spec":{"containers":[{"image":"keras-mnist:0.1","imagePullPolicy":"IfNotPresent","name":"keras-mnist-classifier","resources":{"requests":{"memory":"1Mi"}}}],"terminationGracePeriodSeconds":20.0}}],"graph":{"children":[],"endpoint":{"type":"REST"},"name":"keras-mnist-classifier","subtype":"MICROSERVICE","type":"MODEL"},"name":"keras-mnist-predictor","replicas":1.0}]}}

22:01:16.423 [pool-1-thread-1] WARN  i.s.c.k8s.SeldonDeploymentWatcher - Failed to parse SeldonDelployment {"apiVersion":"machinelearning.seldon.io/v1alpha2","kind":"SeldonDeployment","metadata":{"clusterName":"","creationTimestamp":"2018-07-25T22:01:07Z","labels":{"app":"seldon"},"name":"seldon-deployment-example","namespace":"default","resourceVersion":"1111","selfLink":"/apis/machinelearning.seldon.io/v1alpha2/namespaces/default/seldondeployments/seldon-deployment-example","uid":"3b580ceb-9056-11e8-a12d-080027068900"},"spec":{"annotations":{"deployment_version":"0.0","project_name":"Digits classification"},"name":"keras-mnist-deployment","oauth_key":"oauth-key","oauth_secret":"oauth-secret","predictors":[{"annotations":{"predictor_version":"0.0"},"componentSpecs":[{"spec":{"containers":[{"image":"keras-mnist:0.1","imagePullPolicy":"IfNotPresent","name":"keras-mnist-classifier","resources":{"requests":{"memory":"1Mi"}}}],"terminationGracePeriodSeconds":20.0}}],"graph":{"children":[],"endpoint":{"type":"REST"},"name":"keras-mnist-classifier","subtype":"MICROSERVICE","type":"MODEL"},"name":"keras-mnist-predictor","replicas":1.0}]}}
com.google.protobuf.InvalidProtocolBufferException: Cannot find field: subtype in message seldon.protos.PredictiveUnit
    at io.seldon.clustermanager.pb.JsonFormat$ParserImpl.mergeMessage(JsonFormat.java:1292)
    at io.seldon.clustermanager.pb.JsonFormat$ParserImpl.merge(JsonFormat.java:1252)
    at io.seldon.clustermanager.pb.JsonFormat$ParserImpl.parseFieldValue(JsonFormat.java:1797)
    at io.seldon.clustermanager.pb.JsonFormat$ParserImpl.mergeField(JsonFormat.java:1462)
    at 

@cliveseldon pointed out that the subtype was deprecated and therefore the deployment script for this example should be updated.
Upon updating the deployment script and removing the subtype, the deployment to my minikube setup worked.

Thanks to @cliveseldon for the prompt and detailed help!
Will submit a PR in a few minutes.

[ukclivecox]

Thanks for the PR #195