~~Cross driver version engine cache could impact performace?~~

[dtlnor]

Setup

• vsmlrt v13.1
• Nvidia 4090
• processor clock @ 2675~2775 MHz
• memory clock @ 1281.5 MHz
• CUDA driver: 12.0.147
• Windows 10 22H2
• VapourSynth R60 (py 3.10 64bit)

FP16
1920x1080 rgbs

Measurements: FPS / Device Memory (MB)

using 528.24 cache

• driver 528.24

model		TRT 1 stream	TRT 2 streams
waifu2x upconv7		39.87 / 2478	64.64 / 4214
waifu2x upresnet10		31.88 / 2130	46.92 / 3615
waifu2x cunet			26.05 / 9256
cugan		20.41 / 4927	26.56 / 9252
rife I.O. RGBH (3840x2176, model=46)		54.62 / 1637	68.73 / 2600
rife I.O. RGBS (3840x2176, model=46)		35.90 / 1960	51.78 / 3281

using 528.24 cache

• driver 528.49

model		TRT 1 stream	TRT 2 streams
waifu2x upconv7		39.91 / 2408	65.52 / 4252
waifu2x upresnet10		29.17 / 2100	41.78 / 3642
waifu2x cunet			23.15 / 9479
cugan		19.25 / 4942	23.95 / 9253
rife I.O. RGBH (3840x2176, model=46)		54.37 / 1621	67.99 / 2584
rife I.O. RGBS (3840x2176, model=46)		35.78 / 1975	51.29 / 3301

using 528.49 cache

• driver 528.49

model		TRT 1 stream	TRT 2 streams
waifu2x upconv7		40.89 / 2441	66.81 / 2468
waifu2x upresnet10		32.08 / 2113	46.49 / 3617
waifu2x cunet			25.88 / 9440
cugan		21.10 / 4955	26.48 / 9306
rife I.O. RGBH (3840x2176, model=46)		56.03 / 1687	69.43 / 2694
rife I.O. RGBS (3840x2176, model=46)		37.24 / 1954	54.15 / 3233

For 2 streams trt

• waifu2x upresnet10
• waifu2x cunet
• cugan

this three model seems to have performace impact for using the cache not generated by the current driver version.
(idk if this is just 528.49's issue)

[dtlnor]

OK, nothing happens. It turns out my performance impact takes from my home temperature.
(I didn't turn on my air Air Conditioner when running 528.24 cache with driver 528.49 )

[WolframRhodium]

Just for my curiosity, how does hardware-accelerated GPU scheduling impact performance?

[dtlnor]

@WolframRhodium

vsmlrt v13.1

• AMD Ryzen™ 9 7950X
• DDR5 2x32 6000mhz
• Nvidia 4090 (wddm, ecc disabled, rBar enabled)
• GPU processor clock @ 2490 MHz
• GPU memory clock @ 1469 MHz
• driver 528.49
• CUDA driver: 12.0.147
• Windows 10 22H2
• VapourSynth R60 (py 3.10 64bit)
• backend:
  fp16 = True, use_cuda_graph = True, static_shape = False (RIFE using the script from discussion)

Hardware accelerated GPU scheduling OFF

model		TRT 1 stream	TRT 2 streams
dpir color		22.57 / 1961	25.45 / 3317
dpir grey		24.45 / 1938	25.81 / 3268
waifu2x upconv7		40.48 / 2430	68.58 / 4282
waifu2x upresnet10		33.09 / 2113	49.51 / 3631
waifu2x cunet		21.28 / 5020	27.31 / 9441
cugan		21.75 / 4925	28.10 / 9265
RealESRGANv2 xsx2		30.62 / 1284	43.58 / 1958
rife io-fp16 peak static (3840x2176, model=46)		63.34 / 1687	121.22 / 2708
rife io-fp16 overhead static (3840x2176, model=46)		55.65 / 1695	69.14 / 2706
rife io-fp32 peak static (3840x2176, model=46)		39.06 / 2110	74.39 / 3331
rife io-fp32 overhead static (3840x2176, model=46)		36.73 / 1999	52.09 / 3312

Hardware accelerated GPU scheduling ON

model		TRT 1 stream	TRT 2 streams
dpir color		22.61 / 1962	23.96 / 3437
dpir grey		24.41 / 1936	25.16 / 3268
waifu2x upconv7		41.12 / 2432	52.84 / 4269
waifu2x upresnet10		33.52 / 2104	40.51 / 3620
waifu2x cunet		21.67 / 5007	24.35 / 9439
cugan		22.18 / 4923	25.11 / 9258
RealESRGANv2 xsx2		31.35 / 1264	37.13 / 1956
rife io-fp16 peak static (3840x2176, model=46)		65.26 / 1700	94.79 / 2771
rife io-fp16 overhead static (3840x2176, model=46)		57.31 / 1686	72.78 / 2701
rife io-fp32 peak static (3840x2176, model=46)		40.24 / 1993	65.72 / 3320
rife io-fp32 overhead static (3840x2176, model=46)		37.93 / 1987	55.15 / 3311

[WolframRhodium]

Thanks! It does show that HAGS is beneficial for low latency (single stream) inferences while bad for throughput-oriented workloads.