Significant performance difference between DirectX 11 and OpenCL pipelines

[sutajo]

On my machine with an RTX 3080 GPU, I get the following results with DirectX and OpenCL:

DirectX 11:

Processed to : BC2        with  1 iteration(s) in 2.848 seconds
Using        : DXC
Quality      : 1.00
KPerf(ms)    : 3.797
MTx/s        : 4.214
MSE          : 12.1288
PSNR         : 37.29
SSIM         : 0.9590
Total time(s): 3.122

OpenCL:

Processed to : BC2        with  1 iteration(s) in 0.238 seconds
Using        : OCL
Quality      : 1.00
KPerf(ms)    : 0.013
MTx/s        : 1189.248
MSE          : 12.1298
PSNR         : 37.29
SSIM         : 0.9590
Total time(s): 0.511

I used the same 4K image for both measurements. I ran the CLI tool multiple times to ensure that the shaders are already compiled.
Any ideas what might cause the performance issue with the DirectX 11 backend?