[ROCm] Generalize FastGeLU

[cloudhan]

Allow the whole pipeline to be parameterized with unary elementwise functor.

[cloudhan]

This also very slightly improved the perf,

Before:

FastGeluStaticSelection_half                       float16  batch_size=256  seq_len=128  hidden_size=192  84.35 us, 1193.48 GB/s
FastGeluTunable_half                               float16  batch_size=256  seq_len=128  hidden_size=192  81.46 us, 1235.72 GB/s
FastGelu_half_128_1                                float16  batch_size=256  seq_len=128  hidden_size=192  263.70 us, 381.74 GB/s
FastGelu_half_128_16                               float16  batch_size=256  seq_len=128  hidden_size=192  89.04 us, 1130.62 GB/s
FastGelu_half_128_2                                float16  batch_size=256  seq_len=128  hidden_size=192  137.21 us, 733.64 GB/s
FastGelu_half_128_4                                float16  batch_size=256  seq_len=128  hidden_size=192  95.07 us, 1058.88 GB/s
FastGelu_half_128_8                                float16  batch_size=256  seq_len=128  hidden_size=192  83.13 us, 1210.96 GB/s
FastGelu_half_192_1                                float16  batch_size=256  seq_len=128  hidden_size=192  185.82 us, 541.74 GB/s
FastGelu_half_192_16                               float16  batch_size=256  seq_len=128  hidden_size=192  89.24 us, 1128.00 GB/s
FastGelu_half_192_2                                float16  batch_size=256  seq_len=128  hidden_size=192  115.59 us, 870.90 GB/s
FastGelu_half_192_4                                float16  batch_size=256  seq_len=128  hidden_size=192  95.67 us, 1052.25 GB/s
FastGelu_half_192_8                                float16  batch_size=256  seq_len=128  hidden_size=192  83.21 us, 1209.82 GB/s
FastGelu_half_256_1                                float16  batch_size=256  seq_len=128  hidden_size=192  176.06 us, 571.76 GB/s
FastGelu_half_256_16                               float16  batch_size=256  seq_len=128  hidden_size=192  88.44 us, 1138.29 GB/s
FastGelu_half_256_2                                float16  batch_size=256  seq_len=128  hidden_size=192  114.57 us, 878.63 GB/s
FastGelu_half_256_4                                float16  batch_size=256  seq_len=128  hidden_size=192  95.85 us, 1050.20 GB/s
FastGelu_half_256_8                                float16  batch_size=256  seq_len=128  hidden_size=192  84.07 us, 1197.38 GB/s
FastGelu_half_320_1                                float16  batch_size=256  seq_len=128  hidden_size=192  195.94 us, 513.75 GB/s
FastGelu_half_320_16                               float16  batch_size=256  seq_len=128  hidden_size=192  88.60 us, 1136.19 GB/s
FastGelu_half_320_2                                float16  batch_size=256  seq_len=128  hidden_size=192  123.44 us, 815.49 GB/s
FastGelu_half_320_4                                float16  batch_size=256  seq_len=128  hidden_size=192  100.72 us, 999.45 GB/s
FastGelu_half_320_8                                float16  batch_size=256  seq_len=128  hidden_size=192  85.40 us, 1178.68 GB/s
FastGelu_half_384_1                                float16  batch_size=256  seq_len=128  hidden_size=192  195.97 us, 513.69 GB/s
FastGelu_half_384_16                               float16  batch_size=256  seq_len=128  hidden_size=192  88.81 us, 1133.55 GB/s
FastGelu_half_384_2                                float16  batch_size=256  seq_len=128  hidden_size=192  123.81 us, 813.03 GB/s
FastGelu_half_384_4                                float16  batch_size=256  seq_len=128  hidden_size=192  101.11 us, 995.58 GB/s
FastGelu_half_384_8                                float16  batch_size=256  seq_len=128  hidden_size=192  84.75 us, 1187.78 GB/s
FastGelu_half_448_1                                float16  batch_size=256  seq_len=128  hidden_size=192  190.69 us, 527.89 GB/s
FastGelu_half_448_16                               float16  batch_size=256  seq_len=128  hidden_size=192  88.89 us, 1132.47 GB/s
FastGelu_half_448_2                                float16  batch_size=256  seq_len=128  hidden_size=192  121.14 us, 831.00 GB/s
FastGelu_half_448_4                                float16  batch_size=256  seq_len=128  hidden_size=192  99.56 us, 1011.11 GB/s
FastGelu_half_448_8                                float16  batch_size=256  seq_len=128  hidden_size=192  84.68 us, 1188.74 GB/s
FastGelu_half_512_1                                float16  batch_size=256  seq_len=128  hidden_size=192  178.75 us, 563.15 GB/s
FastGelu_half_512_16                               float16  batch_size=256  seq_len=128  hidden_size=192  89.07 us, 1130.19 GB/s
FastGelu_half_512_2                                float16  batch_size=256  seq_len=128  hidden_size=192  116.16 us, 866.63 GB/s
FastGelu_half_512_4                                float16  batch_size=256  seq_len=128  hidden_size=192  97.61 us, 1031.26 GB/s
FastGelu_half_512_8                                float16  batch_size=256  seq_len=128  hidden_size=192  83.87 us, 1200.29 GB/s
FastGelu_half_64_1                                 float16  batch_size=256  seq_len=128  hidden_size=192  507.46 us, 198.37 GB/s
FastGelu_half_64_16                                float16  batch_size=256  seq_len=128  hidden_size=192  90.69 us, 1109.93 GB/s
FastGelu_half_64_2                                 float16  batch_size=256  seq_len=128  hidden_size=192  258.26 us, 389.77 GB/s
FastGelu_half_64_4                                 float16  batch_size=256  seq_len=128  hidden_size=192  136.29 us, 738.63 GB/s
FastGelu_half_64_8                                 float16  batch_size=256  seq_len=128  hidden_size=192  81.08 us, 1241.62 GB/s

After:

FastGeLUStaticSelection_half                       float16  batch_size=256  seq_len=128  hidden_size=768  82.98 us, 1213.06 GB/s
FastGeLUTunable_half                               float16  batch_size=256  seq_len=128  hidden_size=768  79.81 us, 1261.24 GB/s
FastGeLU_half_128_1                                float16  batch_size=256  seq_len=128  hidden_size=768  263.25 us, 382.39 GB/s
FastGeLU_half_128_16                               float16  batch_size=256  seq_len=128  hidden_size=768  88.85 us, 1133.02 GB/s
FastGeLU_half_128_2                                float16  batch_size=256  seq_len=128  hidden_size=768  136.95 us, 735.05 GB/s
FastGeLU_half_128_4                                float16  batch_size=256  seq_len=128  hidden_size=768  91.05 us, 1105.62 GB/s
FastGeLU_half_128_8                                float16  batch_size=256  seq_len=128  hidden_size=768  82.29 us, 1223.25 GB/s
FastGeLU_half_192_1                                float16  batch_size=256  seq_len=128  hidden_size=768  183.90 us, 547.38 GB/s
FastGeLU_half_192_16                               float16  batch_size=256  seq_len=128  hidden_size=768  89.22 us, 1128.26 GB/s
FastGeLU_half_192_2                                float16  batch_size=256  seq_len=128  hidden_size=768  113.38 us, 887.88 GB/s
FastGeLU_half_192_4                                float16  batch_size=256  seq_len=128  hidden_size=768  91.38 us, 1101.63 GB/s
FastGeLU_half_192_8                                float16  batch_size=256  seq_len=128  hidden_size=768  81.84 us, 1229.99 GB/s
FastGeLU_half_256_1                                float16  batch_size=256  seq_len=128  hidden_size=768  171.83 us, 585.85 GB/s
FastGeLU_half_256_16                               float16  batch_size=256  seq_len=128  hidden_size=768  88.62 us, 1135.88 GB/s
FastGeLU_half_256_2                                float16  batch_size=256  seq_len=128  hidden_size=768  112.62 us, 893.85 GB/s
FastGeLU_half_256_4                                float16  batch_size=256  seq_len=128  hidden_size=768  91.45 us, 1100.75 GB/s
FastGeLU_half_256_8                                float16  batch_size=256  seq_len=128  hidden_size=768  82.86 us, 1214.94 GB/s
FastGeLU_half_320_1                                float16  batch_size=256  seq_len=128  hidden_size=768  190.37 us, 528.79 GB/s
FastGeLU_half_320_16                               float16  batch_size=256  seq_len=128  hidden_size=768  88.14 us, 1142.07 GB/s
FastGeLU_half_320_2                                float16  batch_size=256  seq_len=128  hidden_size=768  121.30 us, 829.90 GB/s
FastGeLU_half_320_4                                float16  batch_size=256  seq_len=128  hidden_size=768  96.51 us, 1043.06 GB/s
FastGeLU_half_320_8                                float16  batch_size=256  seq_len=128  hidden_size=768  84.25 us, 1194.77 GB/s
FastGeLU_half_384_1                                float16  batch_size=256  seq_len=128  hidden_size=768  190.03 us, 529.73 GB/s
FastGeLU_half_384_16                               float16  batch_size=256  seq_len=128  hidden_size=768  88.76 us, 1134.08 GB/s
FastGeLU_half_384_2                                float16  batch_size=256  seq_len=128  hidden_size=768  121.92 us, 825.65 GB/s
FastGeLU_half_384_4                                float16  batch_size=256  seq_len=128  hidden_size=768  96.68 us, 1041.23 GB/s
FastGeLU_half_384_8                                float16  batch_size=256  seq_len=128  hidden_size=768  83.34 us, 1207.89 GB/s
FastGeLU_half_448_1                                float16  batch_size=256  seq_len=128  hidden_size=768  184.39 us, 545.95 GB/s
FastGeLU_half_448_16                               float16  batch_size=256  seq_len=128  hidden_size=768  88.48 us, 1137.77 GB/s
FastGeLU_half_448_2                                float16  batch_size=256  seq_len=128  hidden_size=768  119.05 us, 845.55 GB/s
FastGeLU_half_448_4                                float16  batch_size=256  seq_len=128  hidden_size=768  95.27 us, 1056.67 GB/s
FastGeLU_half_448_8                                float16  batch_size=256  seq_len=128  hidden_size=768  83.50 us, 1205.53 GB/s
FastGeLU_half_512_1                                float16  batch_size=256  seq_len=128  hidden_size=768  174.79 us, 575.91 GB/s
FastGeLU_half_512_16                               float16  batch_size=256  seq_len=128  hidden_size=768  88.76 us, 1134.14 GB/s
FastGeLU_half_512_2                                float16  batch_size=256  seq_len=128  hidden_size=768  114.32 us, 880.58 GB/s
FastGeLU_half_512_4                                float16  batch_size=256  seq_len=128  hidden_size=768  93.12 us, 1081.02 GB/s
FastGeLU_half_512_8                                float16  batch_size=256  seq_len=128  hidden_size=768  83.00 us, 1212.87 GB/s
FastGeLU_half_64_1                                 float16  batch_size=256  seq_len=128  hidden_size=768  506.36 us, 198.80 GB/s
FastGeLU_half_64_16                                float16  batch_size=256  seq_len=128  hidden_size=768  91.16 us, 1104.24 GB/s
FastGeLU_half_64_2                                 float16  batch_size=256  seq_len=128  hidden_size=768  257.95 us, 390.25 GB/s
FastGeLU_half_64_4                                 float16  batch_size=256  seq_len=128  hidden_size=768  135.42 us, 743.36 GB/s
FastGeLU_half_64_8                                 float16  batch_size=256  seq_len=128  hidden_size=768  79.90 us, 1259.82 GB/s

[PeixuanZuo]

LGTM. Thanks.