[DMP 2024]: Identify and retrieve beneficiary details through face recognition

[drtechie]

Ticket Contents

Description

Digital Bharat Collaborative (DBC) is an initiative of Piramal Swasthya, the healthcare arm of the Piramal Foundation. DBC's focus areas include building a robust digital public health delivery platform, improving data-driven governance and empowering healthcare workers.

Health and Wellness Centre (HWC) Mobile Application is developed by DBC to meet all Comprehensive Primary Health Care (CPHC) services as per the guidelines of Ayushman Bharat Digital Mission (ABDM). Health and Wellness Centres are aimed to deliver Comprehensive Primary Health Care, that includes preventive, promotive, curative, rehabilitative and palliative care, with a focus on wellness and the delivery of an expanded range of services closer to the community.

There are challenges in the field during the revisit of patients. When a patient revisits a centre, he/she must be looked up from the database in the most efficient and easiest way possible. The health worker should be able to fetch the details of the patient to prevent re-registration and thereby data duplication. By integrating facial recognition into the app, Community Health Officers (CHO) can quickly retrieve the registered patient's details, prefill the revisit form, look up medical history and prevent duplicate registrations.

The addition of this feature will ease the workflow of hundreds of health workers and indirectly ease the process of revisiting for thousands of beneficiaries.

Goals & Mid-Point Milestone

Goals

• Research and identify face recognition models that can work offline
• Collect and store face recognition data from a picture
• Search the face with saved facial biometric data
• [Goals Achieved By Mid-point Milestone are above]
• Lookup for the existing beneficiary in DB with the matched face
• Implement UI-UX elements for integrating the capture and recognition of images in the registration form

Setup/Installation

https://github.com/PSMRI/HWC-Mobile-App

Expected Outcome

• Capture and store facial recognition data in case of first registration
• Capture face and search beneficiaries from the existing registry of facial data in case of re-visit
• User-intuitive UI-UX to prefill data for a beneficiary in case of a revisit.

Acceptance Criteria

• Implement robust security measures to protect patient data, especially facial biometrics.
• The solution must work offline.

Implementation Details

This repo is a good starting point for the implementation.

At a high level, the following may be the implementation steps:

1. Launch Camera with lower resolution setting - Take a photo (you could also take a frame from the camera feed.
2. Find the face in the photo and collect bounding box coordinates ( as a Rect ) using MLKit.
3. Crop the face from the photo using the bounding box coordinates.
4. Feed the cropped face to the FaceNet model to generate embeddings. Your output will be an 128 dimensional vector of the given face
5. Load the list of patient IDs, names and their vectors to perform the face search on. This data set should be specific to the health workers' location and be cached for offline use.
6. Compare the photo vector with the list using  either L2norm or cosine similarity
7. If there are close matches - show the list of patients that matched closely along with a thumbnail of their photo.
8. if the user chooses an already registered patient -> start a new visit for that patient.
9. If there are no matches or the user chooses -register this as a new patient.
10. Save a thumbnail of the photo and features against a new patient ID

Mockups/Wireframes

NA

Product Name

AMRIT Mobile

Organisation Name

Piramal Swasthya Management Research Institute

Domain

⁠Healthcare

Tech Skills Needed

Computer Vision, Machine Learning, Material-UI

Mentor(s)

@drtechie

Category

Machine Learning, Mobile

[Abinash-bit]

Hi @drtechie, The project problem statement is pretty clear and doable with in the timeframe of the of C4GT 2024. I am intrested in working on this project.

My Background :
I am Abinash Mahapatra from Odisha University of Technology and Research, Currently pursuing Btech in Electrical Engineering (Final Year).

I have worked on NLP, Transformers, Various CNN Architectures such as LeNet, Alexnet, VGG16, VGG19, Resnet18, Resnet 50, Computer vision and currently interning as a Machine Learning Developer @ZCLAP INC.

I will go through the installation and setup and start updating you.

Please can you attach your discord user id so that i can contact and update you abot the project.

[MadhukeshSingh]

Hi there, @drtechie , I'm passionate about machine learning and keen on joining this project.

Here's a bit about myself:
I am Madhukesh Singh, currently studying at the National Institute of Technology, Hamirpur, in my third year.

My experience includes working on image processing, computer vision, and object detection in satellite imagery during my internship as an AI developer at DRDO DYSL.AI.

Is there a preferred method for communicating with the mentors? I'm eager to contact you and explore how I can contribute.

[Sayanjones]

@drtechie I'm interested in contributing to the facial recognition feature for your mobile app. I have the listed skills (Computer Vision, Machine Learning, material UI) and can help research offline models, develop recognition logic, and implement a user-friendly UI/UX.

I'd love to discuss how I can contribute further. Could we schedule a meeting?

[mrchandanm]

Hii @drtechie I am interested in working on this project! I have a strong background in computer vision, image processing, machine learning, and Android development. I believe my skills and experience make me well-suited for this project, and I am eager to contribute to its success.

[drtechie]

@Abinash-bit @MadhukeshSingh @mrchandanm @Sayanjones
I appreciate your interest in this issue.

Please follow this link to register for the DMP 2024 program.
You'll need to submit the design proposal on Unstop to apply for the project and contribute.

If you have any specific questions or doubts about the project, feel free to ask.

Thanks.

[skushagra]

Hi @drtechie, I am interested in working on this project. I will try to complete it and raise a PR in the given timeframe.

My Background :
I am Kushagra, from Scaler School of Technology, pursuing Bachelors in Computer Science(First Year).

I am new to Machine Learning. I have explored PyTorch, FaceNet etc. I believe with my background in development and ML, I can contribute to this project.

Can you share your discord or some way to contact you?

[Thejas775]

Hi @drtechie I am interested in working on this project. Let me introduce myself. I am Thejas Elandassery from Indian Institute of Technology Madras.I have experience with deepface, ML kit, Tensorflow Lite, CameraX and native android development in kotlin. I have done internships and freelance projects in android development. I have a question regarding this project.

1. Do we save the face details of the person on device or on some cloud database ?

For offline loading of the model. We can train ml model and use tensorflow lite and open-cv for the work (another way of implementing this feature) or the FaceNet.

Thanks waiting for your reply.

[Unnatidharsharma]

This is my Discord server link- https://discord.gg/rT9ZkvrB

…

________________________________ From: Thejas Elandassery ***@***.***> Sent: 12 April 2024 14:57 To: PSMRI/HWC-Mobile-App ***@***.***> Cc: UNNATIDHAR SHARMA ***@***.***>; Manual ***@***.***> Subject: Re: [PSMRI/HWC-Mobile-App] [DMP 2024]: Identify and retrieve beneficiary details through face recognition (Issue #1) Hi @drtechie<https://github.com/drtechie> I am interested in working on this project. Let me introduce myself. I am Thejas Elandassery from Indian Institute of Technology Madras.I have experience with deepface, ML kit, Tensorflow Lite, CameraX and native android development in kotlin. I have done internships and freelance projects in android development. I have a question regarding this project. 1. How will we get the face image data of the person that we want to detect the face like do we will already have the model trained on the person's image? For offline loading of the model. We can train ml model and use tensorflow lite and open-cv for the work. Thanks waiting for your reply. — Reply to this email directly, view it on GitHub<#1 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BCKU66NNHXYHP5MOS534SO3Y46SHBAVCNFSM6AAAAABETNWAFGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANJRGM4DOMRWGE>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[AyushSarangi]

Hi @drtechie I am Ayush Sarangi and I am interested to be a part of this project. Currently I am an undergraduate at IIT-BHU, Varanasi. I have prior experience in computer vision domain. I would love to connect with you and start contributing to this project as soon as possible. waiting for your reply .
Thank you.

[drtechie]

@AyushSarangi @Unnatidharsharma @Thejas775 @skushagra

Please follow this link to register for the DMP 2024 program.
You'll need to submit the design proposal on Unstop to apply for the project and contribute.

If you have any specific questions or doubts about the project, feel free to ask here or message me on Discord (ID: drtechie).

Thanks.

[parth721]

Hello @drtechie, I have a question:
don't we need to fix issues, to earn point? Because I found the selection criteria is 25% on issues fixed, 75% on proposal

[Thejas775]

@AyushSarangi @Unnatidharsharma @Thejas775 @skushagra

Please follow this link to register for the DMP 2024 program. You'll need to submit the design proposal on Unstop to apply for the project and contribute.

If you have any specific questions or doubts about the project, feel free to ask here or message me on Discord (ID: drtechie).

Thanks.

Sir the discord id you shared is not working.

[LisysTL]

Hello @drtechie , I have already achieved a similar kind of functionality with my team not only using face net , also another related algorithms , I have a team and we can contribute over this .

[drtechie]

Hello @drtechie, I have a question: don't we need to fix issues, to earn point? Because I found the selection criteria is 25% on issues fixed, 75% on proposal

Hello @parth721, I checked the proposal template here

You have more chances of getting selected if you have previously solved C4GT community tickets.

Applicants who have contributed to C4GT’s open community projects or successfully completed the GitHub classroom project listed on C4GT’s website will boost their selection chances in DMP 2024 by 25%.

If you have queries regarding the selection process for the DMP 2024 program, I recommend getting in touch with the C4GT team or message in Discord.

cc: @SaumyaSamagra

[drtechie]

Sir the discord id you shared is not working.

@Thejas775 fairly certain drtechie is my username. You can DM me.
Also, I joined the C4GT server today.

[drtechie]

Hello @drtechie , I have already achieved a similar kind of functionality with my team not only using face net , also another related algorithms , I have a team and we can contribute over this .

As mentioned in my previous replies, @LisysTL please follow the proposal process.

[barkhaaroraa]

hello @drtechie , i am barkha arora currently a second year student at upes dehradun and i am interested in taking up this task. i am particularly interested in this task because i have experience working with real time face recognition using python libraries and fetching information about the person through databases (you can checkout fall detection on my github pinned repositories), while building ui/ux is something i have little experience with, but i'd love to contribute to this task.

[Vaibhavsun]

Hello @drtechie, I'm Vaibhav Suntwal, currently a second-year B.Tech student at IIIT Naya Raipur. I'm keen on collaborating on this project, bringing my expertise in machine learning, CNNs, and computer vision to the table. I have experience working with face recognition using Haar cascade classifier and have also tackled image classification tasks.

[dhanushgithubrepo]

hi @drtechie , I'm Dhanush , currently a second year student in Rajalakshmi Institute of Technology.

I have deep interest on the field of machine learning , deep learning and computer vision ,I have created two project's similar to the project :

             -->1. Identification of dog breed by using computer vision with the help of Convolutional Neural Networks 
             -->2. Identification of bird species using deep learning and computer vision

I want to contribute to this project idea , I have registered in the Unstop and in the status of writing my proposal where I am asked to find for the issues and contribute to it please help me with this part on where to find as I am new to open source

Thank you.

[AbhimanyuSamagra]

Do not ask process related questions about how to apply and who to contact in the above ticket. The only questions allowed are about technical aspects of the project itself. If you want help with the process, you can refer instructions listed on Unstop and any further queries can be taken up on our Discord channel titled DMP queries. Here's a Video Tutorial on how to submit a proposal for a project.

[akashmeruva9]

hi @drtechie i am trying to open the app, i need login details to open the app, username and other details

[aaradhyasinghgaur]

@drtechie sir , I'm also having an issue with logging into the app via username/password when debugging it.

[PriyalPB]

Hi @drtechie! I'm a third year student from Cummins Pune.

I'm thrilled to join your Identify and retrieve beneficiary details through face recognition project and offer my skill sets which has a strong background in Machine Learning ,deep learning (CNN), NLP, Computer Vision and Python, which seem to fit perfectly with what you're looking for.
I'm excited to explore how my expertise can elevate the project. Furthermore, the integration DL with CV advancements could lead to a seamlessly automated system.
I'm eager to discuss further avenues where I can make meaningful contributions. Could we schedule a meeting to delve into this in more detail?

[drtechie]

base URL: https://amritdemo.piramalswasthya.org/
Username: 7000000006
PW: Test@123

@akashmeruva9 @kyra-09 try these details.

[Mann14]

Hi, @drtechie I'm Mann Mittal, currently a third-year student in LNCTE Bhopal.
I'm keen on collaborating with you on this project because I have a deep interest in Machine learning and my college branch is also AIML I have good experience on working with this type of project, and I also created a similar type of project named "Facial recognition app" as a minor project in my college.
Github link-https://github.com/Mann14/face_recognition
I would love to work with you on this project and learn some new things from you.

[memyselfandglitch]

Hi @drtechie ,
I am Deveshi Singh, a third year undergraduate student at IIT(ISM) Dhanbad. I have worked on a product development facility in my college where I worked extensively with computer vision and Raspberry Pi to develop a blind aid for visually impaired individuals. I am very passionate about machine learning and the field of computer vision. I have worked extensively with the MERN stack and I am familiar with NoSQL databases. I am really looking forward to making a contribution to this project. I would love to work on this project and learn new things in the process.
Regards
Deveshi

[Mansi08git]

Hello @drtechie
I am Mansi Soni , a second year undergraduate student currently pursuing B.Tech . I am currently working on Machine Learning and Artificial Intelligence and in that right now I am exploring Computer Vision . I have also made a model for face-recognition using open cv and face-recognition modules. And I am interested to collaborate in the above project . As I found myself best for this project . Please allow me to work on this project and to explore more about computer vision.

[hlukare]

Wow, this project seems like a game-changer! Integrating facial recognition into the HWC Mobile App could truly revolutionize healthcare delivery, making it more efficient and convenient for both health workers and patients. I'm excited to be a part of this innovative journey towards improving public health in our community!

Please allow me to work on this project and learn new things in the process.

can you please tell me how can I get access for this project?

[drtechie]

Do not ask process related questions about how to apply and who to contact in the above ticket. The only questions allowed are about technical aspects of the project itself. If you want help with the process, you can refer instructions listed on Unstop and any further queries can be taken up on our Discord channel titled DMP queries. Here's a Video Tutorial on how to submit a proposal for a project.

Locked the conversation.
Please refer to the above quoted reply.

[drtechie]

Weekly Goals

Week 1

• Identify offline face recognition model
• Create a POC for recognition and tagging of persons with collected images

Week 2

• Create UI for capturing photographs of beneficiaries
• Compute face vectors and store photographs and face vectors locally

Week 3

• Compute similarity results when new photographs are taken
• Create UI for suggestions of matched beneficiaries

Week 4

• Pull data and pre-fill/continue registration flow for selected suggestion

Week 5

• Create APIs to accept face vectors and photographs for beneficiaries
• Create APIs to send face vectors and photographs to client side.

Week 6

• Up-sync face vector and photograph
• Down-sync face vector and photograph

Week 7

• Create a fallback UI for searching patients when suggestions are not available

Week 8

• Review complete user flow and polishing of UI-UX

[Thejas775]

Weekly Learnings & Updates

Week 1

• I researched various face recognition models for our project. FaceNet model by Google is the best fir for our purpose.
• Created a document containing all the research I have done so for.

Week 2

• Added FaceNet model into the app.
• Room database cannot store vectors so the vector needs to be converted to string.

Week 3

• Calculated Face vector for comparison using L2Norm.
• Implemented UI suggestions for matching of person and updated the recycler view.

Week 4

• Pulled Data of the user whose face recognition had matching results and continue the flow for the registration.

Week 5

• Facenet Model was having issues working with all phones, so this week tried to resolve this issue by finding different ways to solve this.

Week 6

• Switched the facenet model from gpu to cpu which allowed it to run on all devices and for a faster initialisation.
• Along with this, added the faceEmbeddings into the PatentNetwork (PatNet) which allows us to send the faceEmbeddigns to the server along with the patients.

Week 7

• Worked with the API changes on the server code.

Week 8

• Resolving issues of the API.

Week 9

• Updating the code for the retreival of FaceEmbedding from the server to app.

Week 10

• Completed all the development.

[drtechie]

This ticket is completed.