Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[tools] new api and support dynamic models in python benchmark app #8582

Merged
merged 246 commits into from
Dec 2, 2021
Merged

[tools] new api and support dynamic models in python benchmark app #8582

merged 246 commits into from
Dec 2, 2021

Conversation

AlexeyLebedev1
Copy link
Contributor

@AlexeyLebedev1 AlexeyLebedev1 commented Nov 14, 2021 •
edited
Loading

Tickets:

  • 68569

Details:

  • rewrite benchmark app with API 2.0.
  • support dynamic models.

Support dynamic models includes:

  • -shape parameter can be dynamic
  • -data_shape parameter represents input data shape
  • specifying -data_shape parameter isn't required in case image input, images will be processed with original shapes
  • -batch_size can be dynamic
  • -pcseq tool reports performance for each shape (group of shapes in case multi input network)
  • -legacy_mode - enable legacy scenario with inputs filling only once before measurements, default for static case
  • -full mode - include inputs filling into loop measurement, default for dynamic models.
  • Max/Avg/Min latency for both cases - dynamic and static, median latency is valid for static case only

Some changes in benchmark app:

  • Number of processed input files doesn't depend on number of infer requests anymore. Max number of images that is multiple number of data shapes will be used.
  • User able to pass path to directory in input file mapping.

nosovmik and others added 30 commits September 6, 2021 21:18
@nosovmik
Preprocessing API - base classes
9218954 
Includes API definition for trivial mean/scale operations (which don't require layout)

Mean/scale with 'layout' support will be done under separate task together
 with Layout

Current test code coverage: 100%
@nosovmik
Python bindings for base preprocessing API
ba84f8a
@pszmel
Merge remote-tracking branch 'upstream/master' into ov20/preprocess_b…
4c9d8bd 
…ase_python
@pszmel
Merge remote-tracking branch 'upstream/master' into ov20/preprocess_b…
96138b8 
…ase_python
@pszmel
remove pre_post_process directory from ngraph/core
2593ff6
@pszmel
remove files from ngraph/python dir
3b8fd4c
@pszmel
move pyngraph pre_post_process files from ngraph/python to runtime
81b89d2
@pszmel
remove pre_post_process test from CMakeList
3edb5a9
@pszmel
move include to the header
54c8fe9
@pszmel
update include path for pre_post_process
8d7fb91
@pszmel
style fix
a2dcadf
@pszmel
bind InputTensorInfo::set_layout
0676b5d
@pszmel
cleaned test_preprocess
eaf08db
@pszmel
Merge remote-tracking branch 'upstream/master' into ov20/preprocess_b…
e77ad6b 
…ase_python
@pszmel
fix test expected output
1fcfd55
@pszmel
Merge remote-tracking branch 'upstream/master' into ov20/preprocess_b…
b3da1c2 
…ase_python
@pszmel
Merge remote-tracking branch 'upstream/master' into ov20/preprocess_b…
de5b498 
…ase_python
@pszmel
Merge remote-tracking branch 'upstream/master' into ov20/preprocess_b…
4dab7a4 
…ase_python
@pszmel
remove duplicate test
bc5e618
@pszmel
update description of set_element_type
34908aa
@pszmel
fix style
07d5166
@pszmel
Merge remote-tracking branch 'upstream/master' into ov20/preprocess_b…
02675cc 
…ase_python
@pszmel
move preprocess from pyngraph to pyopenvino/graph
448273a
@pszmel
update test_preprocess imports and remove unnecessary test
859784b
@pszmel
remove duplicate import
a8fa793
@pszmel
update custom method
12fe584
@pszmel
update test
18c2608
@pszmel
update test
73417e8
@pszmel
Merge remote-tracking branch 'upstream/master' into ov20/preprocess_b…
de244f0 
…ase_python
@pszmel
create decorator that changes Node into Output<Node>
6215244
@AlexeyLebedev1 AlexeyLebedev1 requested a review from a team November 28, 2021 17:24
akuporos and others added 13 commits November 29, 2021 12:32
@akuporos
Merge remote-tracking branch 'upstream/master' into akup/quick-fix-pa…
f711506 
…ckaging
Merge remote-tracking branch 'akuporos/akup/quick-fix-packaging' into…
525fa81 
… alexeyl1/tools/update_benchmark
Merge remote-tracking branch 'upstream/master' into alexeyl1/tools/up…
7e5c92e 
…date_benchmark
set layout in runtime
ec92cd7
static mode for dynamic models with all equal data shapes
b36785f
use get_tensor instead set_tensor in legacy mode
337020e
benchmarking dynamic model available in full mode only
c1942c9
fix layout detection
bf48319
Merge remote-tracking branch 'upstream/master' into alexeyl1/tools/up…
63020c7 
…date_benchmark
use batch_size * iteration instead processed_frames in legacy mode
6735bd6
fix tensor naming
22fb833
Merge remote-tracking branch 'upstream/master' into alexeyl1/tools/up…
5a40b30 
…date_benchmark
represent --inference_only
bd8fb33
@akuporos akuporos merged commit f9bd740 into openvinotoolkit:master Dec 2, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ilyachur ilyachur ilyachur left review comments

@AlexKoff88 AlexKoff88 AlexKoff88 left review comments

@jiwaszki jiwaszki jiwaszki left review comments

@nkogteva nkogteva nkogteva approved these changes

@avladimi avladimi Awaiting requested review from avladimi avladimi was automatically assigned from openvinotoolkit/openvino-docs-maintainers

@fzhar fzhar Awaiting requested review from fzhar

@ivikhrev ivikhrev Awaiting requested review from ivikhrev

@akuporos akuporos Awaiting requested review from akuporos

Assignees

@nkogteva nkogteva

Labels
category: Core OpenVINO Core (aka ngraph) category: inference OpenVINO Runtime library - Inference category: tools OpenVINO C++ / Python tools
Projects
None yet
Milestone
2022.1
Development

Successfully merging this pull request may close these issues.

None yet