APPLICATION FAILED TO START - Example python notebook (fx-market-predictor)

[dappygoo]

While trying to get the example to setup:
https://github.com/SeldonIO/seldon-core/blob/master/notebooks/kubectl_demo_minikube_rbac.ipynb

I ran into some issues (excuse the formatting, surrounding it with code formatting didn't seem to work)
Ran through the steps and got minikube setup. On the last couple of steps when I see this below:

hwaaa notebooks $ kubectl logs -n seldon test-deployment-fx-market-predictor-svc-orch-84b5dffcc-r768n

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::        (v1.4.3.RELEASE)

2018-08-28 21:49:54.103  INFO 5 --- [           main] io.seldon.engine.App                     : Starting App v0.2.3-SNAPSHOT on test-deployment-fx-market-predictor-svc-orch-84b5dffcc-r768n with PID 5 (/app.jar started by ? in /)
2018-08-28 21:49:54.126  INFO 5 --- [           main] io.seldon.engine.App                     : No active profile set, falling back to default profiles: default
2018-08-28 21:49:54.917  INFO 5 --- [           main] ationConfigEmbeddedWebApplicationContext : Refreshing org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@2eafffde: startup date [Tue Aug 28 21:49:54 GMT 2018]; root of context hierarchy
2018-08-28 21:50:01.024  INFO 5 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.transaction.annotation.ProxyTransactionManagementConfiguration' of type [class org.springframework.transaction.annotation.ProxyTransactionManagementConfiguration$$EnhancerBySpringCGLIB$$ac4ead4e] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2018-08-28 21:50:02.366  INFO 5 --- [           main] i.s.engine.config.CustomizationBean      : Customizing EmbeddedServlet
2018-08-28 21:50:02.367  INFO 5 --- [           main] i.s.engine.config.CustomizationBean      : FOUND env var [ENGINE_SERVER_PORT], will use for engine server port
2018-08-28 21:50:02.372  INFO 5 --- [           main] i.s.engine.config.CustomizationBean      : setting serverPort[8000]
2018-08-28 21:50:02.942  INFO 5 --- [           main] s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat initialized with port(s): 8000 (http) 8082 (http)
2018-08-28 21:50:03.010  INFO 5 --- [           main] o.apache.catalina.core.StandardService   : Starting service Tomcat
2018-08-28 21:50:03.016  INFO 5 --- [           main] org.apache.catalina.core.StandardEngine  : Starting Servlet Engine: Apache Tomcat/8.5.6
2018-08-28 21:50:03.392  INFO 5 --- [ost-startStop-1] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
2018-08-28 21:50:03.393  INFO 5 --- [ost-startStop-1] o.s.web.context.ContextLoader            : Root WebApplicationContext: initialization completed in 8712 ms
2018-08-28 21:50:04.743  INFO 5 --- [ost-startStop-1] o.s.b.w.servlet.ServletRegistrationBean  : Mapping servlet: 'dispatcherServlet' to [/]
2018-08-28 21:50:04.787  INFO 5 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean   : Mapping filter: 'metricsFilter' to: [/*]
2018-08-28 21:50:04.788  INFO 5 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean   : Mapping filter: 'characterEncodingFilter' to: [/*]
2018-08-28 21:50:04.789  INFO 5 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean   : Mapping filter: 'hiddenHttpMethodFilter' to: [/*]
2018-08-28 21:50:04.789  INFO 5 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean   : Mapping filter: 'httpPutFormContentFilter' to: [/*]
2018-08-28 21:50:04.789  INFO 5 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean   : Mapping filter: 'requestContextFilter' to: [/*]
2018-08-28 21:50:04.789  INFO 5 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean   : Mapping filter: 'webRequestLoggingFilter' to: [/*]
2018-08-28 21:50:04.793  INFO 5 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean   : Mapping filter: 'applicationContextIdFilter' to: [/*]
2018-08-28 21:50:06.921  INFO 5 --- [           main] i.s.engine.predictors.EnginePredictor    : init
2018-08-28 21:50:06.923  INFO 5 --- [           main] i.s.engine.predictors.EnginePredictor    : FOUND env var [ENGINE_PREDICTOR], will use for engine predictor
2018-08-28 21:50:09.273  INFO 5 --- [           main] i.s.engine.predictors.EnginePredictor    : Installed engine predictor: {"name":"fx-market-predictor","graph":{"name":"classifier","children":[],"type":"MODEL","implementation":"UNKNOWN_IMPLEMENTATION","methods":[],"endpoint":{"service_host":"test-deployment-fx-market-predictor-classifier","service_port":9000,"type":"REST"},"parameters":[]},"componentSpecs":[{"metadata":{"name":"","generateName":"","namespace":"","selfLink":"","uid":"","resourceVersion":"","generation":0,"deletionGracePeriodSeconds":0,"labels":{"seldon-app-classifier":"test-deployment-fx-market-predictor-classifier"},"annotations":{},"ownerReferences":[],"finalizers":[],"clusterName":""},"spec":{"volumes":[],"containers":[{"name":"classifier","image":"seldonio/mock_classifier:1.0","command":[],"args":[],"workingDir":"","ports":[{"name":"http","hostPort":0,"containerPort":9000,"protocol":"","hostIP":""}],"env":[{"name":"PREDICTIVE_UNIT_SERVICE_PORT","value":"9000"},{"name":"PREDICTIVE_UNIT_PARAMETERS","value":"[]"},{"name":"PREDICTIVE_UNIT_ID","value":"classifier"},{"name":"PREDICTOR_ID","value":"fx-market-predictor"},{"name":"SELDON_DEPLOYMENT_ID","value":"seldon-deployment-example"}],"resources":{"limits":{},"requests":{"memory":{"string":"1Mi"}}},"volumeMounts":[{"name":"podinfo","readOnly":true,"mountPath":"/etc/podinfo","subPath":""}],"livenessProbe":{"handler":{"tcpSocket":{"port":{"type":1,"intVal":0,"strVal":"http"},"host":""}},"initialDelaySeconds":10,"timeoutSeconds":0,"periodSeconds":5,"successThreshold":0,"failureThreshold":0},"readinessProbe":{"handler":{"tcpSocket":{"port":{"type":1,"intVal":0,"strVal":"http"},"host":""}},"initialDelaySeconds":10,"timeoutSeconds":0,"periodSeconds":5,"successThreshold":0,"failureThreshold":0},"lifecycle":{"preStop":{"exec":{"command":["/bin/sh","-c","/bin/sleep 5"]}}},"terminationMessagePath":"","imagePullPolicy":"IfNotPresent","stdin":false,"stdinOnce":false,"tty":false,"envFrom":[],"terminationMessagePolicy":""}],"restartPolicy":"","terminationGracePeriodSeconds":20,"activeDeadlineSeconds":0,"dnsPolicy":"","nodeSelector":{},"serviceAccountName":"","serviceAccount":"","nodeName":"","hostNetwork":false,"hostPID":false,"hostIPC":false,"imagePullSecrets":[],"hostname":"","subdomain":"","schedulerName":"","initContainers":[],"automountServiceAccountToken":false,"tolerations":[],"hostAliases":[],"priorityClassName":"","priority":0}}],"replicas":1,"annotations":{"predictor_version":"v1"},"labels":{}}
2018-08-28 21:50:09.361  WARN 5 --- [           main] ationConfigEmbeddedWebApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'internalPredictionService' defined in URL [jar:file:/app.jar!/BOOT-INF/classes!/io/seldon/engine/service/InternalPredictionService.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'seldonGrpcServer' defined in URL [jar:file:/app.jar!/BOOT-INF/classes!/io/seldon/engine/grpc/SeldonGrpcServer.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'predictionService': Unsatisfied dependency expressed through field 'predictorBean'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'predictorBean': Unsatisfied dependency expressed through field 'predictiveUnitBean'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'predictiveUnitBean': Unsatisfied dependency expressed through field 'internalPredictionService'; nested exception is org.springframework.beans.factory.BeanCurrentlyInCreationException: Error creating bean with name 'internalPredictionService': Requested bean is currently in creation: Is there an unresolvable circular reference?
2018-08-28 21:50:09.362  INFO 5 --- [           main] i.s.engine.predictors.EnginePredictor    : cleanup
2018-08-28 21:50:09.373  INFO 5 --- [           main] o.apache.catalina.core.StandardService   : Stopping service Tomcat
2018-08-28 21:50:09.440  INFO 5 --- [           main] utoConfigurationReportLoggingInitializer : 

Error starting ApplicationContext. To display the auto-configuration report re-run your application with 'debug' enabled.
2018-08-28 21:50:09.476 ERROR 5 --- [           main] o.s.b.d.LoggingFailureAnalysisReporter   : 

***************************
APPLICATION FAILED TO START
***************************

Description:

The dependencies of some of the beans in the application context form a cycle:

┌─────┐
|  internalPredictionService defined in URL [jar:file:/app.jar!/BOOT-INF/classes!/io/seldon/engine/service/InternalPredictionService.class]
↑     ↓
|  seldonGrpcServer defined in URL [jar:file:/app.jar!/BOOT-INF/classes!/io/seldon/engine/grpc/SeldonGrpcServer.class]
↑     ↓
|  predictionService (field io.seldon.engine.predictors.PredictorBean io.seldon.engine.service.PredictionService.predictorBean)
↑     ↓
|  predictorBean (field io.seldon.engine.predictors.PredictiveUnitBean io.seldon.engine.predictors.PredictorBean.predictiveUnitBean)
↑     ↓
|  predictiveUnitBean (field io.seldon.engine.service.InternalPredictionService io.seldon.engine.predictors.PredictiveUnitBean.internalPredictionService)
└─────┘


hwaaa notebooks $ kubectl get po --all-namespaces
NAMESPACE     NAME                                                              READY     STATUS             RESTARTS   AGE
kube-system   etcd-minikube                                                     1/1       Running            0          15m
kube-system   kube-addon-manager-minikube                                       1/1       Running            1          1h
kube-system   kube-apiserver-minikube                                           1/1       Running            0          15m
kube-system   kube-controller-manager-minikube                                  1/1       Running            0          15m
kube-system   kube-dns-86f4d74b45-ggw58                                         3/3       Running            4          1h
kube-system   kube-proxy-g47qx                                                  1/1       Running            0          14m
kube-system   kube-scheduler-minikube                                           1/1       Running            1          1h
kube-system   kubernetes-dashboard-5498ccf677-kgxts                             1/1       Running            3          1h
kube-system   storage-provisioner                                               1/1       Running            3          1h
kube-system   tiller-deploy-895d57dd9-72jql                                     1/1       Running            1          1h
seldon        seldon-core-redis-5f6bc97f64-xjkpl                                1/1       Running            0          4m
seldon        seldon-core-seldon-apiserver-68587c59f-wwl62                      1/1       Running            0          4m
seldon        seldon-core-seldon-cluster-manager-7fb9798f98-d2d4r               1/1       Running            0          4m
seldon        test-deployment-fx-market-predictor-classifier-0-79c4dcb879ft2c   1/1       Running            0          3m
seldon        test-deployment-fx-market-predictor-svc-orch-84b5dffcc-r768n      0/1       CrashLoopBackOff   3          3m

I was wondering if this project just has a circular dependency and that is why the image isn't able to run as a container. Has anyone else ran into similar issues?

[ukclivecox]

This should be fixed in the latest snapshot build 0.2.3-SNAPSHOT. Can you test and report back?

[dappygoo]

This solved it, thanks!