-
Notifications
You must be signed in to change notification settings - Fork 47
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
lavfi/dnn: Async Support for Native Backend #425
Closed
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit adds an async execution mechanism for common use in the TensorFlow and Native backends. Signed-off-by: Shubhanshu Saxena <shubhanshu.e01@gmail.com>
This commits refactors the get async result function for common use in all three backends. Signed-off-by: Shubhanshu Saxena <shubhanshu.e01@gmail.com>
This commit adds the documentation of typedefs and functions in the async module for common use in DNN backends. Signed-off-by: Shubhanshu Saxena <shubhanshu.e01@gmail.com>
This commit adds a function for execution of TFInferRequest and documentation for functions related to TFInferRequest. Signed-off-by: Shubhanshu Saxena <shubhanshu.e01@gmail.com>
This commit enables async execution in the TensorFlow backend and adds function to flush extra frames. The async execution mechanism executes the TFInferRequests on a detached thread. The following is the comparison of this mechanism with the existing sync mechanism on TensorFlow C API 2.5 GPU variant. Async Mode: 0m57.064s Sync Mode: 1m1.959s The above was performed on super resolution filter using ESPCN model. Signed-off-by: Shubhanshu Saxena <shubhanshu.e01@gmail.com>
This patch adds error handling for cases where the execute_model_tf fails, clears the used memory in the TFRequestItem and finally pushes it back to the request queue. Signed-off-by: Shubhanshu Saxena <shubhanshu.e01@gmail.com>
Since requests are running in parallel, there is inconsistency in the status of the execution. To resolve it, we avoid using mutex as it would result in single TF_Session running at a time. So add TF_Status to the TFRequestItem Signed-off-by: Shubhanshu Saxena <shubhanshu.e01@gmail.com>
The frame allocation and filling the TaskItem with execution parameters is common in the three backends. This commit shifts this logic to dnn_backend_common. Signed-off-by: Shubhanshu Saxena <shubhanshu.e01@gmail.com>
This commit unifies the async and sync inference mechanism in the DNN module. For now, the execution is disabled in all three backends temporarily. Signed-off-by: Shubhanshu Saxena <shubhanshu.e01@gmail.com>
This commit unifies the inference functions in the TensorFlow backend and introduces async flag in the TFOptions to be used to switch between the modes. Signed-off-by: Shubhanshu Saxena <shubhanshu.e01@gmail.com>
This commit unifies the execution functions in the OpenVINO backend and introduces async flag in the OVOptions to be used to select the execution mode. Signed-off-by: Shubhanshu Saxena <shubhanshu.e01@gmail.com>
This commit rearranges the code in Native Backend to use the TaskItem for inference and enables the unified inference in the backend. It also adds flush function as required in the unified mechanism. Signed-off-by: Shubhanshu Saxena <shubhanshu.e01@gmail.com>
Remove async flag from filter's perspective after the unification of async and sync modes in the DNN backend. Signed-off-by: Shubhanshu Saxena <shubhanshu.e01@gmail.com>
This commit rearranges the existing code to create two separate functions for filling native model operands with execution data and the completion callback. Signed-off-by: Shubhanshu Saxena <shubhanshu.e01@gmail.com>
This commit defines NativeRequestItem for the Native Backend and also adds a function to copy operands from the Native Model, to be used for creating DnnOperands in NativeRequestItem. Signed-off-by: Shubhanshu Saxena <shubhanshu.e01@gmail.com>
This patch changes the current synchronous execution in the Native Backend to use NativeRequestItem and separate set of operands from the request, which will help in implementing asynchronous execution. Signed-off-by: Shubhanshu Saxena <shubhanshu.e01@gmail.com>
This patch introduces a separate function to execute the NativeRequestItem and free its operands. Signed-off-by: Shubhanshu Saxena <shubhanshu.e01@gmail.com>
The patch adds the documentation for the newly created functions for NativeRequestItems in the Native backend. Signed-off-by: Shubhanshu Saxena <shubhanshu.e01@gmail.com>
This patch adds async support to the Native Backend by executing the NativeRequestItem on detached threads. Signed-off-by: Shubhanshu Saxena <shubhanshu.e01@gmail.com>
Switch flush frame function to use asynchronous mode of execution. Signed-off-by: Shubhanshu Saxena <shubhanshu.e01@gmail.com>
Reduce default value of conv2d_threads in Native Backend for async mode. Signed-off-by: Shubhanshu Saxena <shubhanshu.e01@gmail.com>
shubhanshu02
force-pushed
the
native_async
branch
from
July 24, 2021 11:34
e4504dc
to
8be3238
Compare
Close. No activity. Feel free to reopen when you fix conflicts and need more pre-check |
Sure, thanks. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
TODO
This pull request will be reopened after the pull request #423 gets merged.
Patch Set Description
This patchset is a part of optional deliverables in the GSoC project Async Support for TensorFlow Backend in FFmpeg.
Objective: Extend the asynchronous inference support to the Native Backend.
Parts under this deliverable
NativeRequestItem
based.Notable Points
NativeRequestItem
has its own set of operands compared to a single operand set perNativeModel
earlier.conv2d_threads
only for async executionPatches
1f177d1 lavfi/dnn_backend_native: Separate Functions for Filling Model with Input
2200a04 lavfi/dnn_backend_native: Separate Functions Completion Callback
848a820 lavfi/dnn_backend_native: Add NativeRequestItem and copy_operands
d70712a lavfi/dnn_backend_native: Request-based Inference
05f28d8 lavfi/dnn_backend_native: Add Inference Function
e42c729 lavfi/dnn_backend_native: Documentation for New Functions
5431d81 lavfi/dnn: Async Support for Native Backend
bf69c63 lavfi/dnn_backend_native: Switch Flushing to Async
1fb16f6 lavfi/dnn_backend_native: Reduce default value of conv2d_threads