-
Notifications
You must be signed in to change notification settings - Fork 1.2k
/
render_benchmark_doxygen.h
165 lines (138 loc) · 8.33 KB
/
render_benchmark_doxygen.h
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
namespace drake {
namespace geometry {
namespace render {
/** @defgroup render_engine_benchmarks Render Engine Benchmarks
@ingroup render_benchmarks
The benchmark consists of a scene with one or more spheres and one or more
cameras above the spheres, looking down at the spheres.
If there are multiple spheres, they are positioned in a regular grid placed
at a uniform height above the ground plane. Increasing the number of spheres
provides an approximate measure of how the renderer performs with increased
scene complexity.
If there are multiple cameras, they are all at the same position, looking in
the same direction, with the same intrinsic properties. In other words, each
should produce the same output image. This provides a measure of the
scalability as a simulation includes an increasing number of cameras.
The output image can be configured to an arbitrary size; larger images take
more rendering time.
<h2>Warm-starting RenderEngine Implementations</h2>
Some RenderEngine implementations may defer some of the initialization work
until the rendering API is actually invoked (e.g., RenderEngineVtk). In these
cases, the first rendering will be misleadingly expensive. The benchmark
"warm starts" each render engine under evaluation by performing two renderings
outside of the timed loop. This allows the benchmark to better report the
expected results in the engine's "steady state".
When comparing these benchmark results with observed performance in
applications, remember that the first and possibly second renderings may
deceptively impact any analysis of *average* render performance.
<h2>Benchmarks</h2>
The benchmarks have been configured to examine the following characteristics of
the included RenderEngine implementations:
- Scalability w.r.t. scene complexity
- For a fixed image size, we render a sequence of images with varying
numbers of spheres.
- Scalability w.r.t. number of cameras
- For a fixed image size, render multiple cameras at the simplest and most
complex scene complexities.
- Scalability w.r.t. image size
- With a single camera, we render multiple output images of varying sizes
with both simple and complex scenes.
We examine those same properties for all three image types: color, depth, and
label.
<h2>Running the benchmark</h2>
The benchmark can be executed as:
```
bazel run //geometry/benchmarking:render_experiment -- \
--output_dir=trial1
```
The output will be something akin to the following (although not identical;
lines have been removed for brevity):
```
Run on (12 X 4400 MHz CPU s)
CPU Caches:
L1 Data 32K (x6)
L1 Instruction 32K (x6)
L2 Unified 256K (x6)
L3 Unified 12288K (x1)
Load Average: 9.75, 4.65, 3.26
------------------------------------------------------------------------------------ // NOLINT(*)
Benchmark Time CPU Iterations // NOLINT(*)
------------------------------------------------------------------------------------ // NOLINT(*)
RenderBenchmark/VtkColor/1/1/640/480 0.641 ms 0.641 ms 1026 // NOLINT(*)
RenderBenchmark/VtkColor/12/1/640/480 0.843 ms 0.843 ms 882 // NOLINT(*)
RenderBenchmark/VtkColor/120/1/640/480 2.58 ms 2.58 ms 280 // NOLINT(*)
RenderBenchmark/VtkColor/240/1/640/480 5.30 ms 5.30 ms 100 // NOLINT(*)
RenderBenchmark/VtkColor/480/1/640/480 9.10 ms 9.10 ms 79 // NOLINT(*)
RenderBenchmark/VtkColor/1200/1/640/480 23.4 ms 23.4 ms 30 // NOLINT(*)
RenderBenchmark/VtkColor/1/10/640/480 6.36 ms 6.36 ms 117 // NOLINT(*)
RenderBenchmark/VtkColor/1200/10/640/480 179 ms 179 ms 4 // NOLINT(*)
RenderBenchmark/VtkColor/1/1/320/240 0.267 ms 0.267 ms 2318 // NOLINT(*)
RenderBenchmark/VtkColor/1/1/1280/960 2.39 ms 2.39 ms 317 // NOLINT(*)
RenderBenchmark/VtkColor/1/1/2560/1920 11.5 ms 11.5 ms 65 // NOLINT(*)
RenderBenchmark/VtkColor/1200/1/320/240 24.1 ms 24.0 ms 30 // NOLINT(*)
RenderBenchmark/VtkColor/1200/1/1280/960 27.0 ms 26.9 ms 26 // NOLINT(*)
RenderBenchmark/VtkColor/1200/1/2560/1920 39.8 ms 39.7 ms 19 // NOLINT(*)
RenderBenchmark/GlColor/1/1/640/480 0.523 ms 0.480 ms 1382 // NOLINT(*)
RenderBenchmark/GlColor/12/1/640/480 0.553 ms 0.511 ms 1236 // NOLINT(*)
RenderBenchmark/GlColor/120/1/640/480 0.833 ms 0.790 ms 884 // NOLINT(*)
RenderBenchmark/GlColor/240/1/640/480 1.20 ms 1.15 ms 591 // NOLINT(*)
RenderBenchmark/GlColor/480/1/640/480 1.87 ms 1.83 ms 384 // NOLINT(*)
RenderBenchmark/GlColor/1200/1/640/480 3.76 ms 3.72 ms 188 // NOLINT(*)
RenderBenchmark/GlColor/1/10/640/480 4.98 ms 4.56 ms 137 // NOLINT(*)
RenderBenchmark/GlColor/1200/10/640/480 37.1 ms 36.6 ms 19 // NOLINT(*)
RenderBenchmark/GlColor/1/1/320/240 0.151 ms 0.147 ms 4830 // NOLINT(*)
RenderBenchmark/GlColor/1/1/1280/960 1.93 ms 1.88 ms 335 // NOLINT(*)
RenderBenchmark/GlColor/1/1/2560/1920 9.14 ms 9.08 ms 72 // NOLINT(*)
RenderBenchmark/GlColor/1200/1/320/240 3.46 ms 3.42 ms 199 // NOLINT(*)
RenderBenchmark/GlColor/1200/1/1280/960 5.29 ms 5.25 ms 127 // NOLINT(*)
RenderBenchmark/GlColor/1200/1/2560/1920 12.5 ms 12.5 ms 57 // NOLINT(*)
RenderBenchmark/VtkDepth/1/1/640/480 1.89 ms 1.89 ms 391 // NOLINT(*)
RenderBenchmark/GlDepth/1/1/640/480 0.486 ms 0.486 ms 1578 // NOLINT(*)
RenderBenchmark/VtkLabel/1/1/640/480 1.35 ms 1.35 ms 513 // NOLINT(*)
RenderBenchmark/GlLabel/1/1/640/480 1.34 ms 1.30 ms 521 // NOLINT(*)
```
Additional configuration is possible via the following flags:
- __save_image_path__: Enables saving the rendered images in the given
location. Defaults to no saving.
- __show_window__: Whether to display the rendered images. Defaults to false.
For example:
```
bazel run //geometry/benchmarking:render_experiment -- \
--output_dir=trial2 --save_image_path="/tmp" --show_window=true
```
<h4>Interpreting the benchmark</h4>
Each line in the table represents a particular configuration of the benchmark
parameters of the form:
```
RenderBenchmark/TestName/sphere_count/camera_count/image_width/image_height
```
- __TestName__: One of
- __VtkColor__: Renders the color image from RenderEngineVtk.
- __VtkDepth__: Renders the depth image from RenderEngineVtk.
- __VtkLabel__: Renders the label image from RenderEngineVtk.
- __GlColor__: Renders the color image from RenderEngineGl.
- __GlDepth__: Renders the depth image from RenderEngineGl.
- __GlLabel__: Renders the label image from RenderEngineGl.
- __sphere_count__: The total number of spheres.
- __camera_count__: Simply the number of independent cameras being rendered.
The cameras are all co-located (same position, same view direction) so
they each render the same image.
- __image width__ and __image_height__: The dimensions of the output image
in pixels.
The `Time` and `CPU` columns are measures of the average time it took to create
a single frame for all the specified cameras. The `Iterations` indicates how
often the action was performed to compute the average value. For more
information see the [google benchmark
documentation](https://github.com/google/benchmark).
Now we can analyze the example output and draw some example inferences (not a
complete set of valid inferences):
- Frame cost scales linearly with the number of cameras.
- For RenderEngineVtk a 10X increase in number of cameras leads to a 10X
increase in time for both color and depth, but a (roughly) 20X increase
for label.
- RenderEngineGl scales better with respect to scene complexity than
RenderEngineVtk.
*/
} // namespace render
} // namespace geometry
} // namespace drake