Ideas to Reduce iOS App Power Consumption

[andi00123456]

Hi Very good work

I have downloaded your App and have tested. I am thinking if I can just push a "button" and let the image pixel changed like from 416 to 320, or go to 608. Because if the video at the meaning time doesn't have sufficient objects to detect, we may reduce energy cost by using lower accuracy models.

I am interested in this part, Do you have any idea for how to do it and hope we can work together.

[glenn-jocher]

There are a few enhancements to the app we had in mind, mostly iOS development work, such as updating the camera functionality with pinch and zoom, swapping rear and front cameras etc.

Adjusting the model input (or output) size is not possible in CoreML. Swapping models is possible though, so you could swap between two seperate models: yolov3-spp-416, which currently runs about 10 FPS, and yolov3-spp-320, which runs about 20 fps. yolov3-tiny certainly runs much faster than 30fps also, though bear in mind you would need to have all these models physically located in your iPhone app at their time of use (we had wanted to add a model download functionality as well, as we currently also have working SSD MobileNet models).

I assume these FPS specs will get a nice bump later on in the year with the A13 chip in the new iPhones. If you have expertise in GCP, iOS or CoreML we could certainly use some help.

[glenn-jocher]

Ah, I suppose you could also run the standard yolov3-spp-416 at a reduced rate, say 1 FPS, and increase the response rate to full speed once an object is detected, returning to 1 FPS if nothing was observed for the trailing 5s. This would cut your energy use by 90% (from >10 FPS to 1).

[andi00123456]

Hi Glenn,
Thanks for your reply.
It's obviously that loading three models at the same time would cost a lot. I am thinking that we could load one model and change the resolution, which means reload feature map size. it could cost few seconds as long as it could be accepted. Do you have any comments about my idea?

[glenn-jocher]

@andi00123456 I think you misunderstood. I was saying that having 3 models available, as part of the app bundle, is the only solution to your request. It is not possible to change the input or output dimensions of a CoreML model. But having 3 models available does not cost anything in terms of power consumption, only storage space (your app will have more MB to download, which is why I was saying a good feature to have would be to download models on demand rather than prepackage them in the app bundle).

Power is only required when a model processes input frames. If you want to switch between yolov3-spp-320 and yolov3-spp-416 you simply pass input frames to one or the other, certainly not to both of them.

[andi00123456]

@glenn-jocher Thanks for you reply. One more question, if I have three models as app bundle, when I open app, Does all 3 models must be loaded in Hot Memory(RAM)? If it is, then it cost a lot and also cost three times power than just loading one model.
Storage space is not my concerned.

[glenn-jocher]

@andi00123456 I did a quick google search and it seems that now CoreML 2 does allow flexibility in the input and output shapes. You might start by looking at:
https://forums.developer.apple.com/thread/110407

If you don't have experience in CoreML, Swift and iOS though, this will be a very major undertaking indeed. The current models we use do not support flexibility in the input dimension, as you can see from the Xcode screenshot below:

[andi00123456]

@glenn-jocher I appreciated that what you did.
It is true that i don't have lots of experience about CoreML. furthermore, implemented my idea in the IOS App is the big step.
What I am thinking is that I should implement switch input size at the Server by revising Pytorch code, If I success, then we can think more about it. Thanks.

[glenn-jocher]

@andi00123456 ah of course, resizing in PyTorch is easy :). If you know anyone with iOS expertise, we have internships open:
https://startupmatcher.com/job/ultralytics-llc/ios-developer-internship

We could actually take several steps that would knock the power consumption down and increase the framerate:

• implement input size flexibility in the CoreML model (416 to 320 = +100% faster)
• fuse batch_norm and conv2d layers in PyTorch prior to ONNX export (+30% faster). UPDATE: Done, improvement on CPU inference is small, perhaps 10%.
• implement inference on non-square images (eliminate about 44% of the area = +44% faster) Training on non-squared images #126 UPDATE: Implemented. Now detect.py automatically processes single image batches in rectangular shapes. Speedup on zidane.jpg is about 30-35% (1s before, about 650ms now).

[andi00123456]

@glenn-jocher Thanks, Once I made it, I will let you know.

[glenn-jocher]

UPDATE: I implemented batch_norm fusion in detect.py now, including fusion prior to ONNX export. This reduces the layer count by about 1/3, from 225 to 150 in PyTorch and from 700 to 400 in ONNX, but the new fused CoreML model only runs a bit faster, perhaps 10-20%. The actual model weight count dropped negligibly, < 1%.

yolov3/detect.py

Lines 37 to 51 in f9d25f6

	# Fuse batchnorm
	fuse = True
	if fuse:
	fused_list = nn.ModuleList()
	for a in list(model.children())[0]:
	for i, b in enumerate(a):
	if isinstance(b, nn.modules.batchnorm.BatchNorm2d):
	# fuse this bn layer with the previous conv2d layer
	conv = a[i - 1]
	fused = torch_utils.fuse_conv_and_bn(conv, b)
	a = nn.Sequential(fused, *list(a.children())[i + 1:])
	break
	fused_list.append(a)
	model.module_list = fused_list
	#model_info(model) # yolov3-spp reduced from 225 to 152 layers

[glenn-jocher]

@andi00123456 Rectangular inference is now working in our latest iDetection iOS App build! This is a screenshot recorded yesterday at 192x320, inference on vertical 4k format 16:9 aspect ratio iPhone video. This pushes the performance to realtime 30 FPS!! This means that we now have YOLOv3-SPP running in realtime on an iPhone Xs using rectangular inference! This is a worldwide first as far as we know.

This latest version iDetection, v6 has been pushed to the Apple store for review, and should be available for download worldwide later on in the week.

[andi00123456]

@glenn-jocher Hi Glenn, I am very excited and glad that you reached it out. Really Amazing Work. I will learning this tutorial as you posted
By the way, I finished "Resizing" input size by Pytorch. Just like you said, it's pretty easy.

[glenn-jocher]

@andi00123456 great! Remember inference and training is done at the image size you specify in the argparser argument img_size, which is set at 416 default, but can also be typically run at 320 or 608.

[glenn-jocher]

@andi00123456 iDetection v6 is now available on the app store. You should get 30 FPS now, full realtime inference.

[glenn-jocher]

Support added for YOLOv3-tiny now as well, allowing the user the switch back and forth between SPP and tiny instantly.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove Stale label or comment or this will be closed in 5 days.