About Determinism of TensorRT

[hibestil]

Hi,
I'm using TensorRT FP16 precision mode to optimize my deep learning model which takes an image input and outputs a steering angle. While testing my model, I have observed that FPS(frames per second) of TensorRT inference is different for same inputs. For example, when I run the model at time t for input A, FPS of inferencing is X. But at time t+1, FPS of inferencing for input A is Y.

My observation from results, TensorRT inference engine is non-deterministic if FP16 precision mode used.

I started to think that the source of the non-determinism is floating point operations when I see this comment about CUDA:

"If your code uses floating-point atomics, results may differ from run to run because floating-point operations are generally not associative, and the order in which data enters a computation (e.g. a sum) is non-deterministic when atomics are used."

Does type of precision such as FP16, FP32 and INT8 affect determinism of TensorRT?
Why the FPS values are varying even if the calculations are same?

Do you have any thoughs?

Best regards.

NOTES:

• Hardware : Jetson TX2
• I expect same FPS value on every execution.
• I measure FPS like that:

clock_t beginExecuteEngine = clock();
context->execute(kBatchSize,bindings);
double deltaTimeExecuteEngine = double( clock() - beginExecuteEngine) /double(CLOCKS_PER_SEC);

[rajeevsrao]

You are conflating deterministic execution with variations in execution time. Execution time varies based on many factors. Check out Duncan's talk at GTC 2019 about Determinism in Deep Learning: https://www.youtube.com/watch?v=TB07_mUMt0U

[hibestil]

Thank you for suggestion @rajeevsrao . I watched Duncan's whole talk. But he focused on accuracy determinism. In time critical systems, timing determinism is important. The system should give me guarantee to do inference in stable X time. For example, a self-driving car and ADAS, any delay in the expected time processing can have devastating consequences. Given the fact that there are multiple sensors and decisions based on these sensors must be made in a given time frame to make a safe next manoeuvre of a car, it is critical that the behavior of the system is independent of its workload. Would you give me a suggestion about timing determinism?

I understand difference between deterministic execution and execution time. To eliminate factors that can be affect execution time, I also run the model in an isolated CPU enviroment. At the end of my experiments, I observed/"started to think" the main reason is about non-(timing)-determinism of TensorRT. I could not find anything about timing determinsim on developer guide. The only thing that I found about determinism on this guide is "reproducability" (accuracy determinism). Could you clarify timing determinism of TensorRT?

By the way, When I doing research, I see discussions about non-determinism of TensorFlow and CuDNN. What's your thoughts on these discussions?

I've listed below some of them (By Duncan's talk I understand non-determinism on the accuracy side. But timing determinism is still struggling my mind) :

• "Some ops seem to be non-deterministic on GPU. This can be annoying when debugging instability issues, and there are also some niche applications where deterministic functions are absolutely necessary." : GPU determinism flag jax-ml/jax#565
• "GPU calculations are non-deterministic because the thread scheduling is non-deterministic. Floating-point errors are accumulated in unpredictable ways for operations that are not associative -- a consequence of the GPU hardware itself, not TensorFlow." : Non-deterministic behaviour when ran on GPU openai/baselines#805
• "On GPU, small amount of non-deterministic results is expected. TensorFlow uses the Eigen library, which uses Cuda atomics to implement reduction operations, such as tf.reduce_sum etc. Those operations are non-determnistical." (zheng-xq) : Mention that GPU reductions are nondeterministic in docs tensorflow/tensorflow#2732 (comment)
• "... anything using cuda atomics is non-deterministic, so a way to narrow it down is to see which CuDNN algorithms use CUDA atomics..." Non-deterministic mean and sum reduction tensorflow/tensorflow#3103 (comment)
• "You can't until every operation on cuDNN is not completely deterministic..." : https://stackoverflow.com/questions/44800055/how-to-create-a-cnn-with-deterministic-operations-in-tensorflow-on-a-gpu

As far as I know, if wrong please inform me, TensorRT uses CUDA and CuDNN at backend. Can non-determinism(timing) of CUDA and CuDNN affect my TensorRT engine?

[DilipSequeira]

You're likely to have better luck with this question on the Jetson forums on NVIDIA devtalk, where the Jetson specialists hang out.

As you note, automotive applications have demanding requirements, and the intended platform for those is NVIDIA's DRIVE platform rather than Jetson.

[hibestil]

After talk with @duncanriach from Nvidia, I just figured out that I was using clock() function to measure latency. But unluckly, "clock()" function was measuring CPU clock.
For real time latency measurements, it is need to use chronometer functions such as std::chrono or CudaEvents. When I changed my latency measurement function, problem solved.
Actually Cuda and TensorRT had already behaved latency deterministic from the beginning. It is a tiny-stupid thing but it took my lots of time. Because of I have solved problem I'm closing this issue, thank you for your help.