DVFS causing consistent repeatable classification failures. Expected behavior?

[HypervisorX]

I'm completely new to AI and the jetson ecosystem, but love this outstanding resource you've created here for people to get started. I just recently got a 4gb Jetson devkit bundle from sparkfun that included a power supply etc as it seemed to be literally the only place who had any jetsons in stock.

Following the tutorial and getting to the part of Using the ImageNet Program on Jetson as soon as I tried to classify an image it was never working, always failing with Exception: jetson.inference -- imageNet.Classify() encountered an error classifying the image. if running the python code, or simply not labeling the output image if using the C code.

I tried many things attempting to figure out what was wrong, I'll spare you the details but can assure you there were many times I overwrote the sd card with the factory image thinking maybe it was update related. Sometimes it did work if I ran imagenet right after initial setup, as long as I didn't reboot. Knowing the issue was DVFS, this makes sense as the clock and voltage were probably still higher from being busy with the initial setup.

Eventually after eliminating so many other things, I found that if I disabled DVFS the classification always worked every time, but as soon as I restored the default settings with DVFS turned on, or rebooted which does the same thing, the classification would basically nearly always fail (unless other system activity had already push clocks/voltage up enough to get lucky).

Is this expected behavior that the nano fails to classify images when DVFS is on? If it is then I would highly suggest adding some kind of warning or note telling people to run jetson_clocks before opening the docker container for the examples. I'm sure it would save some other person from this situation if jetson really is supposed to fail to classify images when DVFS is enabled.

I'm assuming this probably isn't expected behavior or I would find more people asking about this in the closed issues. I did find some issues with similar classification problems, but never saw any mention of jetson_clocks or DVFS.

Is this some kind of hardware defect that is only apparent when DVFS is maybe too aggressively, at least for my board, scales the voltage down to the point where its making incorrect calculations?

Heres are two examples showing a run with and without DVFS on. This is just with default options downloaded/installed for the docker container. To save on unnecessary lines of text in the logs, the initial TensorRT network optimization was pre run with DVFS disabled just so I could be sure it wasn't introducing any errors at that point. I have tested many times including deleting the cached network and even starting with a fresh SD card image. Once I realized DVFS was the issue it has been consistently reproducible.

I have attached two files that show the full output from running of the docker container and attempting the first imagenet example with DVFS on and off, including with the output of tegrastats to confirm the clock frequency before running the docker container.
dvfs-off.txt
dvfs-on.txt

The TLDR is that when DVFS is on it will fail with

Traceback (most recent call last):
  File "./imagenet.py", line 68, in <module>
    class
_id, confidence = net.Classify(img)
Exception: jetson.inference -- imageNet.Classify() encountered an error classifying the image

[dusty-nv]

Hmm interesting - I have not seen this behavior before, or been able to reproduce it, and I've not heard others report it either.

Is your Nano running in 10W mode or 5W mode? You can check that with sudo nvpmodel -q

[HypervisorX]

It was in 10W mode, but switching it to 5W mode still results in the same error imageNet.Classify() exception as before as long as I haven't run jetson_clocks to disable DVFS.
The VERY interesting thing is that in 5W mode after I run jetson_clocks with the cached TensorRT network it will result in the following crazy classification for the orange example.

[image] loaded 'images/orange_0.jpg' (1024x683, 3 channels)
class 0410 - 0.010277 (apiary, bee house)
class 0427 - 0.025345 (barrel, cask)
class 0478 - 0.013329 (carton)
class 0489 - 0.010521 (chainlink fence)
class 0539 - 0.017624 (doormat, welcome mat)
class 0580 - 0.021011 (greenhouse, nursery, glasshouse)
class 0611 - 0.048462 (jigsaw puzzle)
class 0646 - 0.014671 (maze, labyrinth)
class 0738 - 0.012352 (pot, flowerpot)
class 0741 - 0.017624 (prayer rug, prayer mat)
class 0750 - 0.027832 (quilt, comforter, comfort, puff)
class 0794 - 0.047363 (shower curtain)
class 0858 - 0.018326 (tile roof)
class 0911 - 0.014839 (wool, woolen, woollen)
class 0921 - 0.012207 (book jacket, dust cover, dust jacket, dust wrapper)

if I delete data/networks/bvlc_googlenet.caffemodel.1.1.8001.GPU.FP16.engine and let Tensor RT re-optimize the network in 5W mode after running jetson_clocks it will simply fail with the classification exception. If I switch back to 10W mode, run jetson_clocks, and delete the same previosly mentioned network file so Tensor RT will re-optimize the network again, it will then properly come up with the classification of class 0950 - 0.966797 (orange)

[HypervisorX]

For anyone else who may locate this issue later with the same or similar problem I wanted to post an update as I close this issue. It was indeed a hardware defect with my Jetson nano devkit board. While I can't say for sure what the problem was I can confirm that after receiving the RMA replacement, using the same exact setup as before, everything is working properly without needing to disable DVFS.