Nvidia Jetson Nano SBC - No CUDA device detected #387
Comments
|
Hi @ManuSB90 Sorry.. 😞 |
|
Hi @ManuSB90 , So, before testing with php, you must make sure this test pass to make sure you compiled dlib correctly. 😉 On the other hand, change the size of the images since the jpg was very small. and no faces were detected. 😅 |
|
Hi Matias, Thanks for this, hope my dream is not this short lived... 😞 This is exactly what i was going to ask next, help to determine if the issue is with pdlib / dlib compilation or CUDA drivers, so then to raise the issue in the according forums, but before raising an issue there just needed to dig a bit deeper somehow in the pdlib calls to see what dlib modules we are calling where the GPU validation is happening and failing and if this is cumming from my compiled library's, vs the pip ones working fine... Run the test again with the below not so promising result: |
|
Maybe like you should update CUDNN... 🤔 See But I really can't help you beyond that because i don't have any experience with CUDNN. Observe how pip3 compiles, and try to reproduce it in your installation, then try the cpp test before trying php.. 😉 |
|
Hi @ManuSB90 You still need to install Pdlib for face clustering, but the image analysis task (Where you get all the CUDA advantage) is done a with service in python... 🤔 Just replace your facerecognition folder with that, and follow the instructions of https://github.com/matiasdelellis/facerecognition-external-model |
|
Hi @matiasdelellis , Thanks for the effort in building an external model 🙇 I have not manage to exactly know what the dlib python compile property's are or how they are different to the manual compile. or if the repository it is pulling dlib sources are different in any way prepared for arm/jetson nano. Just wanted to confirm if the install is different... Also trying to pas the --install-option="--set BUILD_SHARED_LIBS=ON" but did not quite work :)
In an other note, i did manage to run the cpp-test finally with the same result :
Then all 3 test failed consistently on the second .png image with, great 😅
CUDNN not being up to date wouldn't be the issue as currently the machine is at cuDNN 8.0.0. I raised the issue in the Nvidia forums where very vauge and not that helpfull 🤦♂️
But did point to the documentation of the failing function:
And looks to be that the function is dealing with rerunning the amount of GPU memory workspace the user needs to allocate to be able to call. Then is this a memory issue? Also noticed that in the first pull of the test suite the python test was completing the scan for both images fine. Does that give any meaning full info? is the image to small? |
|
Hi @ManuSB90
This is strange .. If you use upsample, it can detect more faces, but i should consume 4 times more memory in that test. 😕 you can do the same test with php? Just add an 🤔 |
|
Well this is counter-intuitive... So yes, ran the php test as indicated with the 1 and it completed this time the scan for both photos. But have monitored the GPU memory allocation and notice some differences:
I feel like we could inquire this in the NVIDIA forums why that would be? (also am not able to pin exactly waht/how to ask... ) But at the same time I think we have proven some instances where the php dlib use is working correctly? Again not sure if the upscaling is needed because the images are quite small in size in this sample test? and that if in the real NC implementation this would be needed as most of my pictures would be bigger, circa 4-6MB files. Also coming back to the original issue, while running the NC background job the error doesn't seem to be the same facing here with the sample test. Thanks :) |
Maybe we should report it to Dlib https://github.com/davisking/dlib/blob/master/python_examples/cnn_face_detector.py#L56
Great.. 😄
The test images are small, so found few faces.
Try to reproduce it with the official examples, and report it... 😉
As for the test, no matter how many faces it reports, the idea is just to check that pdlib is working correctly. The images are small just to make sure that it doesn't fail due to lack of memory. As for our application, we have a more precise control of the size of the images. All images are standardized to one size so that they all take the same time, and memory consumption is controlled.
No idea!. 😅 |
|
Hi, Apologies for the delay, got bit busy these last days... 😅
Tried dlib cnn_face_detector.py script and it's working with both. So not able to reproduce the error we where getting with the pdlib test suite png picture: may be something to do with the .png file format?
Tried changing the upscale property to '1' on line 197 as indicated but still failed once submitted: The point where the background job code seems to fail seams to be even before than the pdlib test script, on the open call : Without even getting to the detect call, so our upscale change having no effect relay: Also tried installing the external module as indicated, but after chaining and re-running the background job it is getting the same failure... i think the module change didn't work and still running module 1? I tried restarting Apache and still no change Not sure what to do next... 😅 any suggestions? Thanks again for the support 💪 |
You did not set the model URL or API key.. |
Means that the kernel killed the process due consuming a lot of memory. You must make the image smaller.. |
Hi,
First of all thanks for the amazing work putting together this app, it is through app that I am for the first time learning about ML models and the python world. The wiki is very well documented and relay helpful and giving great insight on the working of the models for a noob like me 😃.
Wanted to build a self hosted NC server after on a Raspberry pi after having successfully installed this in an VM and run a scan.
But after finding the recommendation of having a machine with a dedicated GPU looked for a small SBC and found the Nvidia Jetson nano with some examples of face recognition projects using dlib/python built on it and working.
I have manage to install dlib + face_recognition via pip3 on the device but this did not build the needed shared library, as when installed pdlib it did not find dlib installed. So also installed dlib manually as per the instructions. It automatically detected as CUDA device and installed in CUDA mode.
Some details of the system:
After installing everything setup the pdlib-min-test-suite with the below result:
Note: PHP test is not so successful also seeing very small spikes of GPU use through jtop.
Python test is more successful also registering prolonged 100% use of GPU while running scan on both pics.
Also ran pdlib test suite after build:
I believe face recognition NC app should work on the system as python test suites seem to run fine on the device and it is used out there with real time projects like Doorbell Camera Python. (Have not manage yet to build a small python test script myself and test yet, need to learn bit more about python scripting first)
But not sure if the device detection is failing as this not being an conventional pcie GPU device, but a Tegra X1 SoC architecture... but again, out of depth here with my limited knowledge.
Hope can get some clarification if pdlib is working as expected and the reason of the scan failing to detec the device when in the python environment dlib config/projects are running fine?
Expected behaviour
Face scan to complete.
Actual behaviour
Getting the flowing error:
Steps to reproduce
Server configuration
Operating system:
Ubuntu 18.04.5 LTS
Pdlib version:
v1.0.2
How is DLib installed: Make sure it is working correctly with this tool
git clone https://github.com/davisking/dlib.git
cd dlib/dlib
mkdir build
cd build
cmake -DBUILD_SHARED_LIBS=ON ..
make
sudo make install
Also tried forcing cuda flag: cmake -DBUILD_SHARED_LIBS=ON -D DLIB_USE_CUDA=1 ..
Both methods detect cuda library's and compile successfully.
How is PDlib installed: Make sure it is working correctly with this tool
git clone https://github.com/goodspb/pdlib.git
cd pdlib
phpize
./configure
make
sudo make install
PHP version:
nvjet@nvjet-desktop:~$ php -v
PHP 7.3.24-3+ubuntu18.04.1+deb.sury.org+1 (cli) (built: Oct 31 2020 16:59:59) ( NTS )
Web server:
nvjet@nvjet-desktop:~$ /usr/sbin/apache2 -v
Server version: Apache/2.4.29 (Ubuntu)
Database:
nvjet@nvjet-desktop:~$ mysql -V
mysql Ver 15.1 Distrib 10.1.47-MariaDB, for debian-linux-gnu (aarch64) using readline 5.2
Nextcloud version:
nextcloud-20.0.2.tar.bz2
Client configuration
Browser:
Operating system:
Logs
Background task log with debug.
sudo -u apache php occ -vvv face:background_job
The text was updated successfully, but these errors were encountered: