Why is the first video especially time consuming when batching video?

[dlliyang]

Selected framework: Tensorflow
2019-05-29 16:59:48.336944: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA
2019-05-29 16:59:48.799442: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1392] Found device 0 with properties:
name: Tesla V100-PCIE-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.38
pciBusID: 0000:d9:00.0
totalMemory: 15.75GiB freeMemory: 15.34GiB
2019-05-29 16:59:48.799528: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1471] Adding visible gpu devices: 0
2019-05-29 16:59:49.216287: I tensorflow/core/common_runtime/gpu/gpu_device.cc:952] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-05-29 16:59:49.216360: I tensorflow/core/common_runtime/gpu/gpu_device.cc:958] 0
2019-05-29 16:59:49.216371: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0: N
2019-05-29 16:59:49.216689: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1084] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14517 MB memory) -> physical GPU (device: 0, name: Tesla V100-PCIE-16GB, pci bus id: 0000:d9:00.0, compute capability: 7.0)

CNN model has been built and initialized
Architecture used: vgg

Number of videos: 12
Storage directory: test/
CPU cores: 1
Batch size: 32

Feature Extraction Process

0%| | 0/12 [00:00<?, ?video/s]('get Feature Time:', 1.6686129570007324)
8%|8 | 1/12 [00:02<00:22, 2.01s/video, video=2451831]('get Feature Time:', 0.2662198543548584)
8%|8 | 1/12 [00:02<00:22, 2.01s/video, video=1457058]('get Feature Time:', 0.28127503395080566)
8%|8 | 1/12 [00:02<00:22, 2.01s/video, video=2354797]('get Feature Time:', 0.020145893096923828)
8%|8 | 1/12 [00:02<00:22, 2.01s/video, video=1526389]('get Feature Time:', 0.22216320037841797)
42%|####1 | 5/12 [00:03<00:10, 1.48s/video, video=1180428]('get Feature Time:', 0.26467204093933105)
42%|####1 | 5/12 [00:03<00:10, 1.48s/video, video=1774533]('get Feature Time:', 0.01830887794494629)
42%|####1 | 5/12 [00:03<00:10, 1.48s/video, video=1451504]('get Feature Time:', 0.024552106857299805)
67%|######6 | 8/12 [00:04<00:04, 1.14s/video, video=1466764]('get Feature Time:', 0.023871183395385742)
67%|######6 | 8/12 [00:05<00:04, 1.14s/video, video=1957997]('get Feature Time:', 0.018220186233520508)
83%|########3 | 10/12 [00:05<00:01, 1.00video/s, video=1191604]('get Feature Time:', 0.023998022079467773)
83%|########3 | 10/12 [00:05<00:01, 1.00video/s, video=1469600]('get Feature Time:', 0.024385929107666016)
100%|##########| 12/12 [00:06<00:00, 1.15video/s, video=2163757]
('costTime:', 6.613268136978149)

the first video cost most time,but now i only need process one video and it's must in a fast time,i don't know why,hope to receive your reply,thank u.

[gkordo]

This is a Tensorflow issue, not one of this implementation. The reason why this happens is that TF needs to allocate more GPU memory. Most of the DL frameworks needs significantly more time for the first batch than the rest, due to this issue.

If you can't initialize the network in a constant variable in order to reuse it, then I do not think there is a workaround.

[dlliyang]

This is a Tensorflow issue, not one of this implementation. The reason why this happens is that TF needs to allocate more GPU memory. Most of the DL frameworks needs significantly more time for the first batch than the rest, due to this issue.

If you can't initialize the network in a constant variable in order to reuse it, then I do not think there is a workaround.

Okay,Thank you.