Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Added latency evaluation scripts for all tasks #746

Merged
merged 7 commits into from
Dec 24, 2021
Merged

Conversation

fg-mindee
Copy link
Contributor

As per #699, some latency benchmarking was required. This PR introduces scripts in PyTorch & TensorFlow to benchmark latency on CPU & GPU for text recognition and text detection.

The results were added in a csv file in the same folder to keep track of latency changes for each model.

Finally, a CI job was added to ensure those scripts can run properly.

Any feedback is welcome!

@fg-mindee fg-mindee added topic: ci Related to CI ext: references Related to references folder framework: pytorch Related to PyTorch backend framework: tensorflow Related to TensorFlow backend topic: text detection Related to the task of text detection topic: text recognition Related to the task of text recognition labels Dec 23, 2021
@fg-mindee fg-mindee added this to the 0.5.0 milestone Dec 23, 2021
@fg-mindee fg-mindee self-assigned this Dec 23, 2021
@codecov
Copy link

codecov bot commented Dec 23, 2021

Codecov Report

Merging #746 (8766582) into main (a1291f0) will increase coverage by 0.04%.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #746      +/-   ##
==========================================
+ Coverage   96.07%   96.12%   +0.04%     
==========================================
  Files         129      129              
  Lines        4794     4794              
==========================================
+ Hits         4606     4608       +2     
+ Misses        188      186       -2     
Flag Coverage Δ
unittests 96.12% <100.00%> (+0.04%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
doctr/models/detection/linknet/tensorflow.py 100.00% <100.00%> (ø)
doctr/models/builder.py 99.12% <0.00%> (+1.75%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 03c0ad9...8766582. Read the comment docs.

@fg-mindee fg-mindee changed the title feat: Added latency evaluation scripts for text detection & recognition feat: Added latency evaluation scripts for all tasks Dec 23, 2021
@fg-mindee fg-mindee linked an issue Dec 23, 2021 that may be closed by this pull request
3 tasks
Copy link
Collaborator

@charlesmindee charlesmindee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@fg-mindee fg-mindee merged commit 0192ecf into main Dec 24, 2021
@fg-mindee fg-mindee deleted the latency branch December 24, 2021 10:29
@fg-mindee fg-mindee added the benchmark This issue reports a benchmark label Dec 24, 2021
@fg-mindee fg-mindee added the type: new feature New feature label Dec 31, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
benchmark This issue reports a benchmark ext: references Related to references folder framework: pytorch Related to PyTorch backend framework: tensorflow Related to TensorFlow backend topic: ci Related to CI topic: text detection Related to the task of text detection topic: text recognition Related to the task of text recognition type: new feature New feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[scripts] Adding scripts to easily evaluate the latency of models
2 participants