Init Commit for iOS AICamera Demo #26

nickyfantasy · 2017-11-09T02:10:33Z

Integrate with Paddle libs
Integrate with C++ code from liuyiqun
Create Objective C wrapper and bridging header to call from client in Swift
Able to capture video from device camera and output image buffer
Create hardcode data to draw bounding box and label

-Integrate with Paddle libs -Integrate with C++ code from liuyiquan -Create Objective C wrapper and bridging header to call from client in Swift -Able to capture video from device camera and output image buffer -Create hardcode data to draw bounding box and label

hedaoyuan · 2017-11-13T03:49:38Z

Demo/iOS/AICamera/AICamera/paddle_image_recognizer.cpp

+
+namespace image {
+
+#if 0


Remove this code, if it is not needed.

Yes, it is removed in the newer version.

hedaoyuan · 2017-11-13T03:54:14Z

Demo/iOS/AICamera/AICamera/paddle_image_recognizer.cpp

+                    const size_t nChannels,
+                    const size_t dstHeight,
+                    const size_t dstWidth) {
+        float ratioX = srcWidth * 1.0 / dstWidth;


use static_cast<float> replace * 1.0

Done. The code is copied from another project, not refined but just for advanced use then.

hedaoyuan · 2017-11-13T03:57:56Z

Demo/iOS/AICamera/AICamera/paddle_image_recognizer.cpp

+                float a2 = newX - x1;
+
+                unsigned char* ptDst = outputImage + (i * dstWidth + j) * nChannels;
+                unsigned char* ptSrc = const_cast<unsigned char*>(inputImage) +


why need const_cast? I think ptSrc can be defined as const unsigned char*.

hedaoyuan · 2017-11-13T07:04:49Z

Demo/iOS/AICamera/AICamera/paddle_image_recognizer.h

+    normed_channel_(0) {
+        // Initalize Paddle
+        char* argv[] = {const_cast<char*>("--use_gpu=False")};
+        CHECK(paddle_init(1, (char**)argv));


The function which calls paddle_init should be a static function and need to ensure that paddle_init will only be called once.

void InitPaddle() { static paddle_error ret = paddle_init(1, (char**)argv); }

hedaoyuan · 2017-11-13T07:17:25Z

Demo/iOS/AICamera/AICamera/paddle_image_recognizer.h

+
+private:
+    paddle_gradient_machine gradient_machine_;
+    paddle_error error_;


error_ does not seem to be used.

I am considering to record the error instead of exiting directly...

hedaoyuan · 2017-11-13T07:18:41Z

Demo/iOS/AICamera/AICamera/paddle_image_recognizer.cpp

+                                 const size_t height,
+                                 const size_t width,
+                                 const size_t channel) {
+    // if (height == normed_height && width == normed_width && channel ==


Remove the useless code (line 205-219).

It is refined. When the input's size is the same as the normed, then resize is not needed.

hedaoyuan · 2017-11-13T07:32:05Z

Demo/iOS/AICamera/AICamera/paddle_image_recognizer.cpp

+    // Bilinear Interpolation Resize
+    unsigned char* resized_pixels = (unsigned char*)malloc(
+                                                           resized_height * resized_width * channel * sizeof(unsigned char));
+    image::resize_hwc(pixels,


From 228 to 255 is it possible to merge into one function? Convert the hwc format data with char type to chw format data with float type. This can improve image preprocessing performance.

It will make the code not readable and difficult to debug. A rotation is also needed now. We profile the time then.

Xreki

@hedaoyuan The c++ code is not the final version. It is for the first use in the demo and I'll refine it.

Xreki · 2017-11-13T09:06:21Z

Demo/iOS/AICamera/AICamera/paddle_image_recognizer.cpp

+
+namespace image {
+
+#if 0


Yes, it is removed in the newer version.

Xreki · 2017-11-13T09:09:31Z

Demo/iOS/AICamera/AICamera/paddle_image_recognizer.cpp

+                    const size_t nChannels,
+                    const size_t dstHeight,
+                    const size_t dstWidth) {
+        float ratioX = srcWidth * 1.0 / dstWidth;


Done. The code is copied from another project, not refined but just for advanced use then.

Xreki · 2017-11-13T09:11:08Z

Demo/iOS/AICamera/AICamera/paddle_image_recognizer.cpp

+                float a2 = newX - x1;
+
+                unsigned char* ptDst = outputImage + (i * dstWidth + j) * nChannels;
+                unsigned char* ptSrc = const_cast<unsigned char*>(inputImage) +


Xreki · 2017-11-13T09:12:26Z

Demo/iOS/AICamera/AICamera/paddle_image_recognizer.cpp

+                                 const size_t height,
+                                 const size_t width,
+                                 const size_t channel) {
+    // if (height == normed_height && width == normed_width && channel ==


It is refined. When the input's size is the same as the normed, then resize is not needed.

Xreki · 2017-11-13T09:14:39Z

Demo/iOS/AICamera/AICamera/paddle_image_recognizer.cpp

+    // Bilinear Interpolation Resize
+    unsigned char* resized_pixels = (unsigned char*)malloc(
+                                                           resized_height * resized_width * channel * sizeof(unsigned char));
+    image::resize_hwc(pixels,


It will make the code not readable and difficult to debug. A rotation is also needed now. We profile the time then.

Xreki · 2017-11-13T09:35:15Z

Demo/iOS/AICamera/AICamera/paddle_image_recognizer.h

+    normed_channel_(0) {
+        // Initalize Paddle
+        char* argv[] = {const_cast<char*>("--use_gpu=False")};
+        CHECK(paddle_init(1, (char**)argv));


Xreki · 2017-11-13T10:13:29Z

Demo/iOS/AICamera/AICamera/paddle_image_recognizer.h

+
+private:
+    paddle_gradient_machine gradient_machine_;
+    paddle_error error_;


I am considering to record the error instead of exiting directly...

…curacy threshold and time refresh rate

Xreki · 2017-11-29T09:20:40Z

@nickyfantasy @hedaoyuan @NHZlX
Let's finish this PR and merge it as soon as possible. I don't familiar with swift, so do not have so many comments. Here are several points of mine:

License and copyright: should we keep the same as Paddle?
Models: there are four models in the demo, I think
- face_mobilenet_160_91.paddle and pascal_mobilenet_300_66.paddle are prerequisite
- mobilenet_ssd_160.paddle can be removed
- vgg_ssd_net.paddle is so large, and will not include in the demo. However, users are free to download it for test.
Demo's name: how about rename the demo's name to PDCamera?
Icon of the demo: I think we may use the following picture as the demo's icon.
Documentation: need to write the README, it must tell users the following things:
- How to build the demo
  - how to get the models, and where to put the models
  - where to put paddle libs
- The QR code to download the app
- A simple shown of the app
Change the min-ios-target to a lower version (iOS 10.0)

Add paddle logo

Xreki · 2017-12-05T09:53:39Z

For the icon, I have tested, and it is well shown.
For the app's name, it seems it still AICamera?

Xreki · 2017-12-05T11:59:15Z

I have uploaded the models to a server:

pascal_mobilenet_300_66.paddle, http://cloud.dlnel.org/filepub/?uuid=39c325d9-b468-4940-ba47-d50c8ec5fd5b
face_mobilenet_160_91.paddle, http://cloud.dlnel.org/filepub/?uuid=038c1dbf-08b3-42a9-b2dc-efccd63859fb

Xreki · 2017-12-06T06:53:29Z

Demo/iOS/AICamera/AICamera/ImageRecognizer.swift

+
+
+
+


Remove these blank lines.

Xreki · 2017-12-06T06:55:06Z

Demo/iOS/AICamera/AICamera/SSDData.h

+@end
+
+
+


Remove these blank lines too.

Xreki · 2017-12-06T06:56:10Z

Demo/iOS/AICamera/AICamera/SSDData.m

+    self = [super init];
+    if (self) {
+
+    }


Nothing between the if statement? Then remove the line 17-19.

Xreki · 2017-12-06T06:59:59Z

Demo/iOS/AICamera/AICamera/SSDModel.swift

+            case .PascalMobileNet160:
+                return (160, 160)
+            case .PascalVGG300:
+                return (300, 300)


Remove line 14-15 and line 25-28.

Xreki · 2017-12-06T07:01:39Z

Demo/iOS/AICamera/AICamera/ViewController.swift

+        } else if ssdModel == SSDModel.PascalMobileNet300 {
+            highlightBtn = pascalMobileNetBtn;
+        } else if ssdModel == SSDModel.PascalVGG300 {
+            highlightBtn = pascalVgg300Btn;


Can this also be removed?

We need to keep this. We give users an option to download and try VGG model, for now we hide the button when the model cannot be found

Xreki · 2017-12-06T07:02:56Z

Demo/iOS/AICamera/AICamera/ViewController.swift

+        previewLayer.frame = previewContainer.bounds
+        previewLayer.contentsGravity = kCAGravityResizeAspect
+        previewLayer.videoGravity = AVLayerVideoGravityResizeAspect
+        //        previewLayer.connection.videoOrientation = .portrait


Remove this line since it is of no use.

Xreki · 2017-12-06T07:05:26Z

Demo/iOS/AICamera/AICamera/ViewController.swift

+//                if (isRestarting) {
+//                    restart()
+//                    return;
+//                }


Remove all unused codes in this file.

Xreki · 2017-12-06T07:10:53Z

Demo/iOS/AICamera/AICamera/paddle_image_recognizer.cpp

@@ -0,0 +1,360 @@
+/* Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserve.


The cpp file is updated in https://github.com/PaddlePaddle/Mobile/tree/develop/Demo/linux . However, the paddle_image_recognizer.cpp is split into two files. I found the format of c++ codes in this project is
disorderly. Did you copy the origin file to this project, or copy and paste the codes?

Yes, I put image_utils in there as well. Now I copied paddle_image_recognizer and image_utils from linux Demo to here

Xreki · 2017-12-06T11:27:53Z

Demo/iOS/AICamera/README.md

+
+//QR code link
+
+//screenshot link


You can make a directory images and put the images here, then use following codes to link the images:

![image](./images/QR_code.jpg) ![image](./images/demo_screenshot.jpg)

Xreki · 2017-12-06T12:23:57Z

Demo/iOS/AICamera/README.md

+| ------------------------ |:----------:| --------:|
+| face_mobilenet_160_91    | 160x160    | 71%      |
+| pascal_mobilenet_300_66  | 300x300    | 91%      |
+| vgg_ssd_net              | 300x300    | 66%      |


The model and accuracy does not match.

…e README

Xreki

Thanks for the work. I'll approve it and merge it first. The README still need to refine, but we can do in next PR.

Xreki requested review from Xreki, hedaoyuan and llxxxll November 9, 2017 02:34

nickyfantasy added 3 commits November 9, 2017 14:29

fall back to swift 3 and finish objective C wrapper

056bf8b

Integrate with latest C++ code with resize and pass in byte array

d0dd12b

successfully get results from infer() and draw bounding box

ba62f76

hedaoyuan reviewed Nov 13, 2017

View reviewed changes

Xreki reviewed Nov 13, 2017

View reviewed changes

nickyfantasy added 5 commits November 13, 2017 14:50

Integrate with rotate logic, finalized and clean up code

6bd3279

Add front camera and new model, add settings for different models, ac…

9c5ee0e

…curacy threshold and time refresh rate

Fix issue where default is front camera

3bdcfd8

fix iphone X layout issue

f931ce9

format settings accuracy label

73402fa

Xreki mentioned this pull request Nov 21, 2017

Undefined symbols for architecture x86_64: PaddlePaddle/Paddle#5795

Closed

yukang2017 mentioned this pull request Nov 25, 2017

paddle配置IOS移动端时出现的配置问题 PaddlePaddle/Paddle#5909

Closed

change app name to PDCamera

bcd36e1

Add paddle logo

Add README, remove 2 models and update settings, rename to PDCamera

da6123e

Xreki mentioned this pull request Dec 6, 2017

请问有paddlepaddle应用到Android的Demo吗 #43

Closed

Xreki reviewed Dec 6, 2017

View reviewed changes

clean up code, add image_utils.cpp, add screenshot and QR code, updat…

f8c3400

…e README

Xreki approved these changes Dec 7, 2017

View reviewed changes

Xreki merged commit 5ec3a4c into develop Dec 7, 2017

Xreki deleted the AICameraiOSDemo branch December 14, 2017 02:54

		@@ -0,0 +1,360 @@
		/* Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserve.


		namespace image {

		#if 0


		namespace image {

		#if 0

		@end

Init Commit for iOS AICamera Demo #26

Init Commit for iOS AICamera Demo #26

Conversation

nickyfantasy commented Nov 9, 2017 • edited by Xreki

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Xreki left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Xreki commented Nov 29, 2017 • edited

Xreki commented Dec 5, 2017

Xreki commented Dec 5, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Xreki left a comment

Choose a reason for hiding this comment

nickyfantasy commented Nov 9, 2017 •

edited by Xreki

Xreki commented Nov 29, 2017 •

edited