Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cannot get working external api but internal api is ok #448

Closed
benoitbayol opened this issue Feb 13, 2019 · 9 comments
Closed

cannot get working external api but internal api is ok #448

benoitbayol opened this issue Feb 13, 2019 · 9 comments
Assignees
@ukclivecox
Labels
bug
Projects
Bugs 0.2.7
Milestone
0.2.x

Comments

@benoitbayol
Copy link
Contributor

Hey,

I am creating some images with:

s2i build . seldonio/seldon-core-s2i-python3:0.4 my.registry.com/my-image:version
docker push my.registry.com/my-image:version

If I test this image locally with docker run -p 5000:5000 my-image:version I have no problem to call /predict with internal api in order to have a response with a json payload with "strData" and outputting a json in "binData" (I don't understand why it does come as a strData)

Then I am creating a deployement in kubeflow with seldon and ksonnet with:

ks registry add kubeflow github.com/kubeflow/kubeflow/tree/v0.4.1/kubeflow
ks pkg install kubeflow/seldon
ks generate seldon seldon
ks apply default -c seldon
ks generate seldon-serve-simple-v1alpha2 my-model --image my.registry.com/my-image:version
ks apply default -c my-model

Everything is fine until I call the my-model/api/v0.1/predictions external api.

In the log I have:

2019-02-13 21:58:17.655  INFO 5 --- [nio-8000-exec-3] i.s.e.service.InternalPredictionService  : Requesting http://seldon-xxxxxxx:9000/predict
2019-02-13 21:58:17.935  INFO 5 --- [nio-8000-exec-3] i.s.e.service.InternalPredictionService  : {"binData":[[0.9999990463256836],[0.0002424759150017053]],"meta":{}}
2019-02-13 21:58:17.938 ERROR 5 --- [nio-8000-exec-3] i.s.e.service.InternalPredictionService  : Couldn't retrieve prediction from external prediction server -

For the binData I have also tried by outputting a json but in any case the result of my /predict cannot be transfered to the external with some com.google.protobuf.InvalidProtocolBufferException problems (when outputting json.dumps() I have a problem on length)

Do you have any idea ? I have looked all the examples and I cannot see the obvious reason why it is not working.

Thanks.
@ukclivecox
Copy link
Contributor

Is there a full exception stack trace from the engine container?
Is there any logs from your model container?

@ukclivecox
Copy link
Contributor

The reason it comes back as bindata looks like a bug.

seldon-core/python/seldon_core/model_microservice.py

Lines 99 to 100 in da0a2fb

else:
response = {"binData": predictions, "meta": {}}

We are rewriting the python wrapper so we can fix that as part of that work

@ukclivecox ukclivecox added the bug label Feb 13, 2019
@ukclivecox ukclivecox added this to Needs triage in Bugs via automation Feb 13, 2019
@ukclivecox ukclivecox added this to the 0.2.x milestone Feb 13, 2019
@ukclivecox ukclivecox added this to To do in 0.2.6 via automation Feb 13, 2019
@ukclivecox ukclivecox self-assigned this Feb 13, 2019
@benoitbayol
Copy link
Contributor Author

Is that what you are looking for ?

com.google.protobuf.InvalidProtocolBufferException: null
	at com.google.protobuf.util.JsonFormat$ParserImpl.merge(JsonFormat.java:1084) ~[protobuf-java-util-3.2.0rc2.jar!/:na]
	at com.google.protobuf.util.JsonFormat$Parser.merge(JsonFormat.java:277) ~[protobuf-java-util-3.2.0rc2.jar!/:na]
	at io.seldon.engine.service.InternalPredictionService.queryREST(InternalPredictionService.java:354) [classes!/:0.2.3]
	at io.seldon.engine.service.InternalPredictionService.transformInput(InternalPredictionService.java:216) [classes!/:0.2.3]
	at io.seldon.engine.predictors.PredictiveUnitBean.transformInput(PredictiveUnitBean.java:178) [classes!/:0.2.3]
	at io.seldon.engine.predictors.PredictiveUnitBean.getOutputAsync(PredictiveUnitBean.java:76) [classes!/:0.2.3]
	at io.seldon.engine.predictors.PredictiveUnitBean.getOutput(PredictiveUnitBean.java:60) [classes!/:0.2.3]
	at io.seldon.engine.predictors.PredictiveUnitBean$$FastClassBySpringCGLIB$$ddb0d27b.invoke(<generated>) [classes!/:0.2.3]
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) [spring-core-4.3.5.RELEASE.jar!/:4.3.5.RELEASE]
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:652) [spring-aop-4.3.5.RELEASE.jar!/:4.3.5.RELEASE]
	at io.seldon.engine.predictors.PredictiveUnitBean$$EnhancerBySpringCGLIB$$1d5f6264.getOutput(<generated>) [classes!/:0.2.3]
	at io.seldon.engine.predictors.PredictorBean.predict(PredictorBean.java:54) [classes!/:0.2.3]
	at io.seldon.engine.service.PredictionService.predict(PredictionService.java:84) [classes!/:0.2.3]
	at io.seldon.engine.api.rest.RestClientController.predictions(RestClientController.java:120) [classes!/:0.2.3]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_171]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_171]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_171]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_171]
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:220) [spring-web-4.3.5.RELEASE.jar!/:4.3.5.RELEASE]
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:134) [spring-web-4.3.5.RELEASE.jar!/:4.3.5.RELEASE]
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:116) [spring-webmvc-4.3.5.RELEASE.jar!/:4.3.5.RELEASE]
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) [spring-webmvc-4.3.5.RELEASE.jar!/:4.3.5.RELEASE]
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) [spring-webmvc-4.3.5.RELEASE.jar!/:4.3.5.RELEASE]
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) [spring-webmvc-4.3.5.RELEASE.jar!/:4.3.5.RELEASE]
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963) [spring-webmvc-4.3.5.RELEASE.jar!/:4.3.5.RELEASE]
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897) [spring-webmvc-4.3.5.RELEASE.jar!/:4.3.5.RELEASE]
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) [spring-webmvc-4.3.5.RELEASE.jar!/:4.3.5.RELEASE]
	at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872) [spring-webmvc-4.3.5.RELEASE.jar!/:4.3.5.RELEASE]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:648) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) [spring-webmvc-4.3.5.RELEASE.jar!/:4.3.5.RELEASE]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) [tomcat-embed-websocket-8.5.6.jar!/:8.5.6]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55) [spring-boot-1.4.3.RELEASE.jar!/:1.4.3.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.5.RELEASE.jar!/:4.3.5.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:105) [spring-boot-actuator-1.4.3.RELEASE.jar!/:1.4.3.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.5.RELEASE.jar!/:4.3.5.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) [spring-web-4.3.5.RELEASE.jar!/:4.3.5.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.5.RELEASE.jar!/:4.3.5.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:89) [spring-web-4.3.5.RELEASE.jar!/:4.3.5.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.5.RELEASE.jar!/:4.3.5.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77) [spring-web-4.3.5.RELEASE.jar!/:4.3.5.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.5.RELEASE.jar!/:4.3.5.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) [spring-web-4.3.5.RELEASE.jar!/:4.3.5.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.5.RELEASE.jar!/:4.3.5.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:106) [spring-boot-actuator-1.4.3.RELEASE.jar!/:1.4.3.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.5.RELEASE.jar!/:4.3.5.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:108) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:349) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:784) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:802) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1410) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_171]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_171]
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_171]

@ukclivecox
Copy link
Contributor

This looks like the JSON returned by your model can't be parsed into a SeldonMessage.
Do you have the raw JSON returned by your model or an example that illustrates it?

@ukclivecox ukclivecox removed this from To do in 0.2.6 Feb 18, 2019
@benoitbayol
Copy link
Contributor Author

benoitbayol commented Feb 18, 2019 •
edited

For example:

2019-02-15 14:56:05.054  INFO 5 --- [nio-8000-exec-1] i.s.e.service.InternalPredictionService  : {"binData":[[0.9999990463256836],[0.0002424759150017053]],"meta":{}}

2019-02-15 14:56:05.091 ERROR 5 --- [nio-8000-exec-1] i.s.e.service.InternalPredictionService  : Couldn't retrieve prediction from external prediction server - 

com.google.protobuf.InvalidProtocolBufferException: null
	at com.google.protobuf.util.JsonFormat$ParserImpl.merge(JsonFormat.java:1084) ~[protobuf-java-util-3.2.0rc2.jar!/:na]
	at com.google.protobuf.util.JsonFormat$Parser.merge(JsonFormat.java:277) ~[protobuf-java-util-3.2.0rc2.jar!/:na]
	at io.seldon.engine.service.InternalPredictionService.queryREST(InternalPredictionService.java:354) [classes!/:0.2.3]
	at io.seldon.engine.service.InternalPredictionService.transformInput(InternalPredictionService.java:216) [classes!/:0.2.3]
	at io.seldon.engine.predictors.PredictiveUnitBean.transformInput(PredictiveUnitBean.java:178) [classes!/:0.2.3]
	at io.seldon.engine.predictors.PredictiveUnitBean.getOutputAsync(PredictiveUnitBean.java:76) [classes!/:0.2.3]
	at io.seldon.engine.predictors.PredictiveUnitBean.getOutput(PredictiveUnitBean.java:60) [classes!/:0.2.3]
	at io.seldon.engine.predictors.PredictiveUnitBean$$FastClassBySpringCGLIB$$ddb0d27b.invoke(<generated>) [classes!/:0.2.3]
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) [spring-core-4.3.5.RELEASE.jar!/:4.3.5.RELEASE]
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:652) [spring-aop-4.3.5.RELEASE.jar!/:4.3.5.RELEASE]
	at io.seldon.engine.predictors.PredictiveUnitBean$$EnhancerBySpringCGLIB$$1d5f6264.getOutput(<generated>) [classes!/:0.2.3]
	at io.seldon.engine.predictors.PredictorBean.predict(PredictorBean.java:54) [classes!/:0.2.3]
	at io.seldon.engine.service.PredictionService.predict(PredictionService.java:84) [classes!/:0.2.3]
	at io.seldon.engine.api.rest.RestClientController.predictions(RestClientController.java:120) [classes!/:0.2.3]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_171]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_171]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_171]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_171]
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:220) [spring-web-4.3.5.RELEASE.jar!/:4.3.5.RELEASE]
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:134) [spring-web-4.3.5.RELEASE.jar!/:4.3.5.RELEASE]
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:116) [spring-webmvc-4.3.5.RELEASE.jar!/:4.3.5.RELEASE]
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) [spring-webmvc-4.3.5.RELEASE.jar!/:4.3.5.RELEASE]
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) [spring-webmvc-4.3.5.RELEASE.jar!/:4.3.5.RELEASE]
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) [spring-webmvc-4.3.5.RELEASE.jar!/:4.3.5.RELEASE]
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963) [spring-webmvc-4.3.5.RELEASE.jar!/:4.3.5.RELEASE]
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897) [spring-webmvc-4.3.5.RELEASE.jar!/:4.3.5.RELEASE]
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) [spring-webmvc-4.3.5.RELEASE.jar!/:4.3.5.RELEASE]
	at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872) [spring-webmvc-4.3.5.RELEASE.jar!/:4.3.5.RELEASE]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:648) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) [spring-webmvc-4.3.5.RELEASE.jar!/:4.3.5.RELEASE]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) [tomcat-embed-websocket-8.5.6.jar!/:8.5.6]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55) [spring-boot-1.4.3.RELEASE.jar!/:1.4.3.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.5.RELEASE.jar!/:4.3.5.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:105) [spring-boot-actuator-1.4.3.RELEASE.jar!/:1.4.3.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.5.RELEASE.jar!/:4.3.5.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) [spring-web-4.3.5.RELEASE.jar!/:4.3.5.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.5.RELEASE.jar!/:4.3.5.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:89) [spring-web-4.3.5.RELEASE.jar!/:4.3.5.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.5.RELEASE.jar!/:4.3.5.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77) [spring-web-4.3.5.RELEASE.jar!/:4.3.5.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.5.RELEASE.jar!/:4.3.5.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) [spring-web-4.3.5.RELEASE.jar!/:4.3.5.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.5.RELEASE.jar!/:4.3.5.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:106) [spring-boot-actuator-1.4.3.RELEASE.jar!/:1.4.3.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.5.RELEASE.jar!/:4.3.5.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:108) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:349) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:784) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:802) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1410) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_171]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_171]
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_171]

This binData has been produce by something like:

output1 = self.model1_api.predict(np.array(x_eval_int))[:, 1]
output2 = self.model2_api.predict(np.array(x_eval_int))[:, 1]
return [output1.tolist(), output2.tolist()]

I tried also with some code like:

return {"strData": json.dumps(value)}

@ukclivecox
Copy link
Contributor

The binData field should be base 64 encoded for decoding the JSON to not fail. How are you creating the binData field?

@benoitbayol
Copy link
Contributor Author

benoitbayol commented Feb 18, 2019 via email •
edited

@ukclivecox
Copy link
Contributor

Can you clarify what your input payload is? Is it using strData?
I think there are two options:

  • You could base64 encode the data before returning it and that should work
  • You could use the undocumented predict_rest endpoint which gives you access to the raw input. This assumes you are using 0.2.5 version. For predict_rest you need to return a valid SeldonMessage JSON as return.

@ukclivecox ukclivecox added this to To do in 0.2.7 Feb 21, 2019
@ukclivecox
Copy link
Contributor

Closing a solutions exist in latest python code or fix via base64. Please reopen if not appropriate.

Bugs automation moved this from Needs triage to Closed Apr 4, 2019
@ukclivecox ukclivecox moved this from To do to Done in 0.2.7 Apr 4, 2019
agrski pushed a commit that referenced this issue Dec 2, 2022
@ukclivecox
docs update (#448)
f2be982
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ukclivecox ukclivecox

Labels
bug
Projects
No open projects
0.2.7
  
Done
Bugs
  
Closed
Milestone
0.2.x
Development

No branches or pull requests
2 participants
@ukclivecox @benoitbayol