Use pychubby in android using Chaquopy sdk

[bharatkhadodra1996]

import matplotlib.pyplot as plt
import numpy as np
import urllib.request as urllib
from pychubby.actions import Smile
from pychubby.detect import LandmarkFace

def makeSmile(filePath):
    
img = plt.imread(filePath)
lf = LandmarkFace.estimate(img)
a = Smile(scale=0.2)
new_lf, df = a.perform(lf) 
new_lf.plot(show_landmarks=False)
return "Task done"

This is my .py code. I am calling makeSmile method from java file and passing the image file as a parameter.
Once the work is done on python side, I need output image in return. I read all the documentation but did not understand what the plot method do ? why it does not return anything ?

[jankrepl]

@bharatkhadodra1996 thank you for your interest!

So full transparency, I have no experience with Chaquopy. However, you can access new_lf.img and it will give you the actual image as a numpy array. Hope that helps!

[bharatkhadodra1996]

Ok so I tried your given solution. I converted the new_lf.img to byte_array and passed it to my android file. When I try to create a bitmap with that byte array, null bitmap is created. So there maybe some issue with byte_array.

[jankrepl]

Hmm, again, I am not an expert on Chaquopy. Maybe consider creating an issue in their issue tracker: https://github.com/chaquo/chaquopy/issues and ask how to work with images that are stored in a numpy array.

[bharatkhadodra1996]

Ok I solve that issue by saving file to device. One more issue I faced is my app crash because PyChubby is not able to handle some images. Maybe Dlib is not able to detecting the face or something I do not know the clear reason because there is no error log message about this. So can you tell me what can be the reason and how can I identify the error ?

[jankrepl]

I think one possible reason is that the image is too big. Try resizing it (making it smaller)

let me know if it helps:)

@bharatkhadodra1996

Another possibility is that it did not find any face in the image and then this exception would be raised
https://github.com/jankrepl/pychubby/blob/master/pychubby/detect.py#L221-L222

[bharatkhadodra1996]

Image is not too big size, its around 100 kb.
I traced my code and I found that below line causing an error.
lf = LandmarkFace.estimate(img, model_path=None, n_upsamples=1, allow_multiple=False)

This is the error I got = There are some duplicates.
Below is the image I am using.

I do not understand image is very clear and it is very easy to identify face landmarks then why it is throwing such error.

[jankrepl]

Hmm, interesting.

This is the part of the code that raises that exception:

pychubby/pychubby/detect.py

Lines 255 to 256 in 3d7916f

	if np.unique(points, axis=0).shape != (68, 2):
	raise ValueError("There are some duplicates.")

It basically means that there are duplicate landmark points detected by dlib. Honestly, I don't remember whether this would lead to some crazy problems inside of pychubby if it is just one or two duplicates. However, maybe dlib just failed to do the detection correctly and there are way more duplicates. But what you can do is to

• clone the repo and cd to it
• pip install -e . (editable installation
• comment out those lines and just see whether something breaks + inspect what the 68 detected landmarks look like lf.points

Hope that helps

[jankrepl]

I actually just checked and it seems like there is just a single duplicate landmark in your image. So I guess in this specific case it would make sense to not raise the ValueError. Again, feel free to adjust it.

I am a bit afraid to allow for duplicates in general because if there are 2 duplicate landmark points and there is an action that moves each of the duplicates to a new position that is different then it will basically become "unsolvable". Hope that makes sense:)

[bharatkhadodra1996]

Okay so basically you want me to clone pychubby project to my account. Comment the lines in detect.py file and then use it.
Is that correct ?

[jankrepl]

Okay so basically you want me to clone pychubby project to my account. Comment the lines in detect.py file and then use it. Is that correct ?

yes:)

[bharatkhadodra1996]

Ok so I forked this project and made necessary changes. But I do not know how to use it now ? I mean I was writing pip install pychubby for the original project. So what I need to write to use this forked project ?

[jankrepl]

https://packaging.python.org/en/latest/tutorials/installing-packages/#installing-from-a-local-src-tree

[bharatkhadodra1996]

Thank you for the response. It helped me a lot in my project. After completing all the process, my application size is around 200 MB. From 200 MB, 150+ MB is occupied by PyChubby. So Do you have any idea if I want to reduce the size or if I can use lite version of PyChubby ?

[jankrepl]

Thank you for the response. It helped me a lot in my project. After completing all the process, my application size is around 200 MB. From 200 MB, 150+ MB is occupied by PyChubby. So Do you have any idea if I want to reduce the size or if I can use lite version of PyChubby ?

So I would guess the space is taken by your dependencies (things you have in your virtual environment)
So you would probably have to remove some dependencies (e.g. matplotlib) and make sure to modify the source code of pychubby accordingly - assuming your project does not really need them. Not trivial.

Also, a completely different paradigm would be to write a HTTP server around pychubby and make it possible to send requests to it. You can host it on the cloud. This way the actual processing would be done on the server rather than on the cellphone. However, that would be a lot of work and it would require a lot of domain knowledge. Also maybe you would not wanna go for this solution since it would require constant internet connection.

[bharatkhadodra1996]

Thank you for the response. It helped me a lot in my project. After completing all the process, my application size is around 200 MB. From 200 MB, 150+ MB is occupied by PyChubby. So Do you have any idea if I want to reduce the size or if I can use lite version of PyChubby ?

So I would guess the space is taken by your dependencies (things you have in your virtual environment) So you would probably have to remove some dependencies (e.g. matplotlib) and make sure to modify the source code of pychubby accordingly - assuming your project does not really need them. Not trivial.

Also, a completely different paradigm would be to write a HTTP server around pychubby and make it possible to send requests to it. You can host it on the cloud. This way the actual processing would be done on the server rather than on the cellphone. However, that would be a lot of work and it would require a lot of domain knowledge. Also maybe you would not wanna go for this solution since it would require constant internet connection.

Ok I will try this solution once my application is completed.
One thing I noticed that when I compress my JPG file, its face landmark points get changed. That's why I got the previous error "There are some duplicates".
If I do not compress the image file, application takes too much time to process and sometimes crashes. So do you have any idea or solution how this situation can be handled ?

[jankrepl]

@bharatkhadodra1996 I think you just need to find the right balance between speed and quality. The bigger the image the slower PyChubby is going to be. So try to find the image resolution that represents the right tradeoff for you

[bharatkhadodra1996]

Yes that's right. Currently I am working on it.
I want to add one more feature to this project which is currently not available in PyChubby.
This feature will whiten teeth part. So do you have any plan to add such feature ?

[jankrepl]

The teeth whitening is definitely not something PyChubby can do (it just does warping). I would encourage you to look into generative models that could take your original image + prompt and generate a new one. Or a bit more oldschool approach would be using semantic segmentation to find the pixels that correspond to teeth and then manually overwrite those pixels:D

[bharatkhadodra1996]

@jankrepl I was thinking that I am getting the landmarks of face. So it is easy to recognize the teeth area using those landmark points. So can I do something with that in python ?

[jankrepl]

Well, getting the mouth landmarks will still leave you with the problem of trying to segment things like gums, lips, teeth...

Anyway, as mentioned above this is a problem unrelated to puchubby. I suggest we close this thread and in case you have another specific question related to pychubby feel free to open a new issue:)

Good luck @bharatkhadodra1996

[bharatkhadodra1996]

Ok sure. And thank you for all the help. @jankrepl