Performance Regression in DML

[contentis]

Comparing performance from 0.3.0rc2 to the current release 0.3.0, there seems to be a 7-10% drop in performance for a token generation when using the benchmark script (benchmark.py).
Profiling indicated that the inferences themselves haven't changed, but the delta originates during the sampling process. Notably, in the rc2, there doesn't seem to be any PCI traffic.

Set-Up:

• RTX4090
• DML 1.14.1
• ORT 1.16
• Intel 13900k

Model:

• LLame2 7b int4

---

Raw Data

0.3.0rc2

100/100
Average Tokenization Latency (per token): 0.9491149859968573 ms
Average Tokenization Throughput (per token): 1053.6131182774425 tps
Average Prompt Processing Latency (per token): 0.45547810004791245 ms
Average Prompt Processing Throughput (per token): 2195.495238727852 tps
Average Token Generation Latency (per token): 5.24359868934632 ms
Average Token Generation Throughput (per token): 190.7087210986893 tps
Average Sampling Latency (per token): 0.007989048026502132 ms
Average Sampling Throughput (per token): 125171.35917604853 tps
Average Wall Clock Time: 0.5673357248306274 s
Average Wall Clock Throughput: 352.52495347390305 tps
Results saved in genai_e2e!

1000/100
Average Tokenization Latency (per token): 38.520145014626905 ms
Average Tokenization Throughput (per token): 25.96044224704448 tps
Average Prompt Processing Latency (per token): 0.14640189501224085 ms
Average Prompt Processing Throughput (per token): 6830.512678243603 tps
Average Token Generation Latency (per token): 5.873470604854796 ms
Average Token Generation Throughput (per token): 170.2570877214294 tps
Average Sampling Latency (per token): 0.0082304006209597 ms
Average Sampling Throughput (per token): 121500.7684380977 tps
Average Wall Clock Time: 0.7697359204292298 s
Average Wall Clock Throughput: 1429.0615402053788 tps
Results saved in genai_e2e!

2000/100
Average Tokenization Latency (per token): 57.92622998706065 ms
Average Tokenization Throughput (per token): 17.26333649235202 tps
Average Prompt Processing Latency (per token): 0.17663256999803706 ms
Average Prompt Processing Throughput (per token): 5661.470022267768 tps
Average Token Generation Latency (per token): 6.459128232367073 ms
Average Token Generation Throughput (per token): 154.81965429776434 tps
Average Sampling Latency (per token): 0.008355751167982817 ms
Average Sampling Throughput (per token): 119678.04927362534 tps
Average Wall Clock Time: 1.0549691677093507 s
Average Wall Clock Throughput: 1990.5795015409972 tps
Results saved in genai_e2e!

---

0.3.0

100/100
Average Tokenization Latency (per token): 0.9885049948934466 ms
Average Tokenization Throughput (per token): 1011.6286768058188 tps
Average Prompt Processing Latency (per token): 0.7301077501033433 ms
Average Prompt Processing Throughput (per token): 1369.6608478111002 tps
Average Token Generation Latency (per token): 5.755390152876087 ms
Average Token Generation Throughput (per token): 173.7501669630997 tps
Average Sampling Latency (per token): 0.008945500769186765 ms
Average Sampling Throughput (per token): 111788.04024527628 tps
Average Wall Clock Time: 0.6463512778282166 s
Average Wall Clock Throughput: 309.4292637155656 tps
Results saved in genai_e2e!

1000/100
Average Tokenization Latency (per token): 36.456640012329444 ms
Average Tokenization Throughput (per token): 27.42984541805842 tps
Average Prompt Processing Latency (per token): 0.1962642399885226 ms
Average Prompt Processing Throughput (per token): 5095.171693317536 tps
Average Token Generation Latency (per token): 6.411659899649841 ms
Average Token Generation Throughput (per token): 155.96585215859827 tps
Average Sampling Latency (per token): 0.008837299712467939 ms
Average Sampling Throughput (per token): 113156.73707310943 tps
Average Wall Clock Time: 0.8736567020416259 s
Average Wall Clock Throughput: 1259.0757873538178 tps
Results saved in genai_e2e!

2000/100
Average Tokenization Latency (per token): 271.83306501829065 ms
Average Tokenization Throughput (per token): 3.6787283398828383 tps
Average Prompt Processing Latency (per token): 0.21109468999493403 ms
Average Prompt Processing Throughput (per token): 4737.21058556233 tps
Average Token Generation Latency (per token): 7.096416060283611 ms
Average Token Generation Throughput (per token): 140.91620213711576 tps
Average Sampling Latency (per token): 0.0104329998139292 ms
Average Sampling Throughput (per token): 95849.70936785509 tps
Average Wall Clock Time: 1.4074696063995362 s
Average Wall Clock Throughput: 1492.0393239410928 tps
Results saved in genai_e2e!

[zhangxiang1993]

Hi @contentis, thanks for reporting this, we are looking into it.

[zhangxiang1993]

Quick updates: this performance regression issue is related to a directml dependency on a lower level d3d library which was intended to fix some memory leak issue, and the performance regression is kinda the sacrifice from that fix. We are still looking into a proper fix for this.