Add Image-Recognition and Object-Detection TensorFlow processors #9

tzolov · 2018-02-20T17:35:51Z

Resolves #2

Align with Spring Boot 2.0, Spring Cloud 2.0 and SCSt 2.0 stack
Upgrade to Tensorflow 1.6.0-rc1
Add Image-Recognition processor
Add Object Detection API processor + Protobuf pre-generated labels reader
Extend the common Tensorflow Processor to support multiple evaluation outputs. (FetchModel parameter allow comma-separated out model names)
- Fix the processor output name (in header or tuple) to "result". Use to match the fetch name
- Remove the redundant fetch-index property
- Replace @serviceactivator by @ServiceListener
Refactor the contentType handling to align it with new SCSt 2.0 requirements

Resolves spring-attic#2 - Align with Spring Boot 2.0, Spring Cloud 2.0 and SCSt 2.0 stack - Upgrade to Tensorflow 1.6.0-rc1 - Add Image-Recognition processor - Add Object Detection API processor + Protobuf pre-generated labels reader - Extend the common Tensorflow Processor to support multiple evaluation outputs. (FetchModel parameter allow comma-separated out model names) - Fix the processor output name (in header or tuple) to "result". Use to match the fetch name - Remove the redundant fetch-index property - Replace @serviceactivator by @ServiceListener - Refactor the contentType handling to align it with new SCSt 2.0 requirements

…es in output images. Improve code structure

artembilan

Here is some review.

Not sure that I understand what have you done here (too low programming for me 😉 ), but this is indeed an outstanding work. And we definitely should consider to merge it as fast as possible and start to promote 😄

artembilan · 2018-03-05T22:16:11Z

...a/org/springframework/cloud/stream/app/tensorflow/processor/DefaultOutputMessageBuilder.java

+		}
+
+		// default
+		return MimeTypeUtils.APPLICATION_JSON_VALUE;


Hm. This default is the same as a variant for the String.
Maybe something is missed in the logic?

artembilan · 2018-03-05T22:18:47Z

...ework/cloud/stream/app/image/recognition/processor/ImageRecognitionOutputMessageBuilder.java

+				FontMetrics fm = g.getFontMetrics();
+
+				Tuple resultTuple = new JsonStringToTupleConverter().convert(result.toString());
+				ArrayList<Tuple> labels = (ArrayList) resultTuple.getValues().get(0);


Would be better to have generic args and use @SuppressWarning on the method.

+1. I've replaced with ArrayList<Tuple> labels = resultTuple.getValue("labels", ArrayList.class);

artembilan · 2018-03-05T22:21:00Z

...ework/cloud/stream/app/image/recognition/processor/ImageRecognitionOutputMessageBuilder.java

+			logger.error(e);
+		}
+
+		// Null mend that QR image is found and not output message will be send.


Null means ?
and no output ?

artembilan · 2018-03-05T22:23:37Z

...main/java/org/springframework/cloud/stream/app/object/detection/processor/GraphicsUtils.java

+import java.awt.image.BufferedImage;
+
+/**
+ * @author Christian Tzolov


May we have some JavaDocs on this class to determine the reason of such a big class?
Thanks

artembilan · 2018-03-05T22:24:33Z

...main/java/org/springframework/cloud/stream/app/object/detection/processor/GraphicsUtils.java

+package org.springframework.cloud.stream.app.object.detection.processor;
+
+
+import java.awt.*;


No asterisk imports, please.

artembilan · 2018-03-05T22:30:22Z

...rg/springframework/cloud/stream/app/tensorflow/processor/InvalidTupleTensorflowEncoding.java

+/**
+ * @author Christian Tzolov
+ */
+public class InvalidTupleTensorflowEncoding extends RuntimeException {


Yeah... Exception must have an Exception suffix to be clear...
But that's IMHO

artembilan · 2018-03-05T22:33:34Z

...gframework/cloud/stream/app/tensorflow/processor/TensorflowCommonProcessorConfiguration.java

- * By default the inference result is returned in the outbound Message payload. If the saveResultInHeader property is
- * set to true then the inference result would be stored in the outbound Message header by name as set by
- * the getResultHeader property. In this case the message payload is the same like the inbound message payload.
+ * The {@link OutputMessageBuilder} defines how the computed inference score is arranged withing the output message.
 *
 * @author Christian Tzolov
 * @author Artem Bilan
 */
 @EnableConfigurationProperties(TensorflowCommonProcessorProperties.class)
 public class TensorflowCommonProcessorConfiguration implements AutoCloseable {


We don't need to have here AutoCloseable.
Since you declare TensorFlowService as a bean, Spring will take care about its own AutoCloseable already.

artembilan · 2018-03-05T22:39:31Z

...gframework/cloud/stream/app/tensorflow/processor/TensorflowCommonProcessorConfiguration.java

+	@Bean
+	@StreamMessageConverter
+	public MessageConverter toTensorMessageConverter() {
+		return new AbstractMessageConverter(new MimeType("application", "x-tensor")) {


Where this converter is used?

I thought we need them when we convert input/output from/to external world, therefore it is a bit confusing for me why would one need this custom mime-type?

Thanks

Good spot. I believe this is a left over from my experiments with the new content-type support. I will remove it

artembilan · 2018-03-05T22:45:25Z

...rk/cloud/stream/app/image/recognition/processor/ImageRecognitionProcessorPropertiesTest.java

+/**
+ * @author Christian Tzolov
+ */
+public class ImageRecognitionProcessorPropertiesTest {


Must be plural - Tests

artembilan · 2018-03-05T22:50:47Z

...rk/cloud/stream/app/object/detection/processor/ObjectDetectionTensorflowOutputConverter.java

+			float[] classes = classesTensor.copyTo(new float[1][maxObjects])[0];
+			float[][] boxes = boxesTensor.copyTo(new float[1][maxObjects][4])[0];
+
+			// Work in progress


Should we expect something else here to be done?

Thanks

This is to enable support for image segmentation feature: https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/instance_segmentation.md
While the TF models already support this and we can read the metadata, it turned to be quite involving to convert the mask metadata into mask images.
So this fragment can be regarded as an enabler to support the image segmentation in some of the following up releases.
I will improve the descriptions.

tzolov · 2018-03-07T11:00:04Z

Thanks @artembilan! I've pushed an update to address the review comments.
Also note i've not disabled the integration tests for the Twitter Sentiment, Image Recognition and Object Detection processors. Later pulls large (> 50 MB), pre-build TF models from bintray.

artembilan · 2018-03-09T21:53:19Z

spring-cloud-starter-stream-processor-tensorflow/README.adoc

 image::{image-root}/TensorFlowProcessorArcutectureOverview.png[]

+The `--tensorflow.model` property configures the Processor with the location of the serialized Tensorflow model.
+
+The `TensorflowInputConverter` implementations converts the input data into the format, specific for the given model.


I think to align with the next sentence the implementations word is redundant.

artembilan · 2018-03-09T22:04:00Z

spring-cloud-starter-stream-common-tensorflow/pom.xml

+        </dependency>
+        <dependency>
+            <groupId>org.tensorflow</groupId>
+            <artifactId>libtensorflow_jni_gpu</artifactId>


Would you mind to elaborate more here, please?

Looks like using this here we don't have choice in the target application: https://www.tensorflow.org/versions/master/install/install_linux#determine_which_tensorflow_to_install

I wonder what is going to happen if I don't have the proper GPU or I am just on Windows (although my Nvidia GPU is pretty good 😄 )

I mean shouldn't we leave such a choice for end-user?
Or doesn't it just have any effect if we don't meet GPU requirements and it just fallback to CPU mode?

Thanks

Good point!
According to this doc: https://www.tensorflow.org/install/install_java

GPU acceleration is available via Maven only for Linux and only if your system meets the requirements for GPU.

So my assumption is that those dependencies will only be activated if the above conditions hold. For example it works find on my MacBook laptop using only the CPU there (e.g. the GPU dependencies are ignored).
Also with those additional maven dependencies i hoped we could address issues like this: https://stackoverflow.com/questions/46430512/tensorflow-train-a-model-with-gpus-and-inference-with-java-api

But having said this i acknowledge that i have not tested this functionality, so i will comment out those dependencies and open a separate issue for tackling this.

… in the src

artembilan · 2018-03-15T20:52:22Z

Merged as cd1ea72

tzolov added the in pr label Feb 20, 2018

tzolov added 4 commits February 20, 2018 18:50

Improve docs

ec67d2f

Generate the label map Protobuf class on the fly

f3a4ebb

Allow drowing the image recontintion labels and object detection bbox…

347c004

…es in output images. Improve code structure

Update to TF 1.6.0. Add README. Improve structure

ff54449

artembilan suggested changes Mar 5, 2018

View reviewed changes

sabbyanandan assigned artembilan Mar 6, 2018

Address the review comments

48feba4

Add GPU support. Improve doc

8ccc3df

artembilan suggested changes Mar 9, 2018

View reviewed changes

tzolov added 3 commits March 15, 2018 19:03

disable the gup dependecies. Fix SCSt 2.0 test support dependecy

692e216

Ignore integration tests to prevent downloading large TF models

61cd287

pre-complie the protobuf definitions and copy the result java classes…

9318e6a

… in the src

artembilan closed this Mar 15, 2018

artembilan removed the in pr label Mar 15, 2018

Add Image-Recognition and Object-Detection TensorFlow processors #9

Add Image-Recognition and Object-Detection TensorFlow processors #9

tzolov commented Feb 20, 2018

artembilan left a comment

artembilan Mar 5, 2018

tzolov Mar 7, 2018

artembilan Mar 5, 2018

tzolov Mar 7, 2018

artembilan Mar 5, 2018

tzolov Mar 7, 2018

artembilan Mar 5, 2018

artembilan Mar 5, 2018

artembilan Mar 5, 2018

tzolov Mar 7, 2018

artembilan Mar 5, 2018

tzolov Mar 7, 2018

artembilan Mar 5, 2018

tzolov Mar 7, 2018

artembilan Mar 5, 2018

tzolov Mar 7, 2018

artembilan Mar 5, 2018

tzolov Mar 7, 2018

tzolov commented Mar 7, 2018

artembilan Mar 9, 2018

tzolov Mar 15, 2018

artembilan Mar 9, 2018

tzolov Mar 15, 2018 •

edited

artembilan commented Mar 15, 2018

		package org.springframework.cloud.stream.app.object.detection.processor;


		import java.awt.*;

Add Image-Recognition and Object-Detection TensorFlow processors #9

Add Image-Recognition and Object-Detection TensorFlow processors #9

Conversation

tzolov commented Feb 20, 2018

artembilan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tzolov commented Mar 7, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tzolov Mar 15, 2018 • edited

Choose a reason for hiding this comment

artembilan commented Mar 15, 2018

tzolov Mar 15, 2018 •

edited