-
Notifications
You must be signed in to change notification settings - Fork 733
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Using Recognition Service with more than 20k images registered #621
Comments
BTW if you want to run it in Kubernetes, check out this repository: |
Hi @pospielov , Thank you so much for the quick reply, yes I was checking the kubernetes repository and the stats you have given helps a lot. Again, thank you so much for the quick turn around. |
Hi @pospielov , I tried the suggestion (compreface_api_java_options=-Xmx3g, compreface_admin_java_options=-Xmx1g), but now the container is crashing even with 1k images when I try to run the recognition service. My current laptop is not very high end and I have an intel graphics card(Intel UHD Graphics 630 1536 MB), not Nvidia. I saw some of the env file has runtime key set as Nvidia in .env file. Can I set the runtime to intel, if so how, can you please help/point me to a link where I can understand how to set it up ? |
Hi here is our hardware: The question here is who is the initiator of OOM error - Linux or Java |
looks like OS kills it. Try to reduce the compreface_api_java_options number to -Xmx2g |
Sure checking now. Based on my understanding, it should not be an issue but please let me know if I need to delete the images too. |
hah, is this configuration is for all containers? So 3Gb is a sum of all RAM consumption? Probably it can be not enough. Could you try 6Gb with the same configuration? |
When you say all containers, what do you mean ? I have updated RAM to 6 GB overall, now uploading images and checking, will keep you posted with results. |
I was successfully able to upload 15k images after increasing overall RAM to 6 GB and kept the configuration as - This is my current docker stats I will also try to add Kubernetes into the mix and upload another 10k images to check if the container fails. |
yes, this is what I was talked about - the sum of RAM consumption is more than 3Gb which leads to OOM. |
Yes, also can you please tell me what is the total number of connections possible ? and once I get this, I am not able to proceed further without killing all the containers and deleting the complete volume. |
Hi, this is a bug. I already fixed it, just need to publish a hotfix. I'll try to do it today |
Sure, once you do I will upload lots of images and test it and will keep you posted here. |
hi, I uploaded the new 0.6.1 version to dockerhub, could you check if it fixes your problem? |
Hi , I downloaded the latest version 0.6.1. Though the service is a little slow compared to previous version(0.6.0) in terms of how much time it takes for new registration. (Previous was less than 1 sec, this one took 3-4 secs) After the registration, I ran the service in recognition mode which failed It was timing out most of the time. I tried checking the container logs but there was nothing there in the API container and Core container said the request was success. The CPU stats for API container went more than 200% and RAM at 2.5 GB for a single request which is not correct. Core container stats was completely fine. I also tried this service to find total number of subjects - I am not sure of the reason behind some services passing, some failing and the logs are not helpful so was not able to debug further. Total RAM is 6 GB and CPU cores are 3 |
UPDATE :- I will create a new service(new API Key) and check what happens then. |
UPDATE :- Also, I do get this error sometimes - |
Hi
|
Hey, Also, yeah the error I mentioned above keeps on happening every few API calls, I was not able to check the container logs. |
I'm looking at what we can do with this error. |
Hi @AnkurChatter |
Hi @pospielov , |
Hi,
I tested the application and it works really amazing.
Thank you so much for the build and documentation.
But when I upload images in excess of 1k and test the recognition services on it, the API container fails.
After that I looked into custom builds and tried mobile net/face net/insight face, all of them with no GPU support but after uploading 3k images, the container again failed during a call to recognition service for just 1 patient.
My current laptop configuration is -
CPU - iIntel Core i3-1115G4 with 4.1 Ghz clock speed (Dual Core ) ,
OS - Win 10 64 bit,
RAM - 8GB DDR4 2666 Mhz
Hard Disk - 1TB HDD
Graphic Card - Intel UHD Graphics 630 1666 MB
The current laptop doesn't have nvidia graphics so no additional GPU support.
This will act as a standalone system which should be capable of using the face recognition algorithm upto a scale of 20k images. I cannot host this on cloud, need this to be on-premise
Can you please suggest a custom build and if such a build is not possible, what kind of system configurations will be needed to scale up to that level ?
Update :- I will try running the docker container in a kubernetes cluster and see if that can help me in solving the scaling issue.
Thank you.
The text was updated successfully, but these errors were encountered: