Discussion for new level-1v/-1m-like operations

[fgvanzee]

Let's discuss new operations that we might like to add to BLIS, specifically those that would fall into level-1v or level-1m families (and perhaps level-2):

• element-wise vector/matrix multiplication
  • fused element-wise vector ops
• element-wise vector/matrix division
• vector reduction
• partial matrix reductions
  • reduce rows to vector
  • reduce columns to vector
• "matrix-times-diagonal-matrix", e.g. C_ij = A_ij * b_j
• c = diag(A*trans?(B)) (c_i = A_ij*B_ji or A_ij*B_ij)
• mixed-precision/mixed-domain level1/2
• parallelization for level1/2

We will add to this list as the discussion unfolds!

[ct-clmsn]

Not sure if these are in scope but, a function like einsum would be useful; ensum is a dependency function for gaussian mixture modeling (and adjacent algorithms).

Generally, any functionality in BLIS that could reflect, or be utilized to implement, functionality found in numpy would be especially valuable: https://numpy.org/doc/stable/reference/routines.linalg.html

[devinamatthews]

@ct-clmsn some of the things I have in mind can be used as kernels for various combinations of einsum parameters. Providing einsum itself is almost certainly out-of-scope, but yes we'd definitely like to be able to provide all of the basic operations that einsum would use internally. Is there some taxonomy/list of these?

[ct-clmsn]

@devinamatthews I'll have to dig through the source to find them, shouldn't take too much time. Will get back with a list soon.

Source implementation is here.

Looks like dot product is the only thing happening under the hood get_sum_of_products_function ; the indexing mechanisms, which work a little bit like a simple domain specific language for describing looping mechanics/loads/stores, is where the 'magic' happens. So it's out of scope.

There's an optimization effort for the functionality that's worth noting

[realab-ai]

I found the performance of operators in 1f and 2 (axpyf, gemv, etc.) are lower than expected at SKX. It seems the current kernels are not designed under the context of AVX512. Shall we take AVX512 feature in the the planning level-1v/-1m?
And, how about multi-threading in the planning work? I'm not sure if it is a good feature for level-1f/-1m but very interested in this issue.

[fgvanzee]

I found the performance of operators in 1f and 2 (axpyf, gemv, etc.) are lower than expected at SKX. It seems the current kernels are not designed under the context of AVX512. Shall we take AVX512 feature in the the planning level-1v/-1m?

Fair question. I think for now we're only gathering candidate operations to add to BLIS. Microarchitectures and instruction sets for which to optimize those (or existing) operations is mostly orthogonal IMO.

And, how about multi-threading in the planning work? I'm not sure if it is a good feature for level-1f/-1m but very interested in this issue.

Multithreading for non-level-3 operations is one of those action items we've wanted for a long time, but not badly enough relative to everything else we want. 😂 It will happen someday!

[hominhquan]

Good initiative, I would add to the original list:

• a, b, c, d, e are vectors (real or complex)
• element-wise dual-multiply dual-add/sub:
  • e = (conj?(a)*b) + (conj?(c)*d)
  • e = (conj?(a)*b) - (conj?(c)*d)
  • e = -(conj?(a)*b) - (conj?(c)*d)
• element-wise dual-multiply dual-add/sub accumulate:
  • e += (conj?(a)*b) + (conj?(c)*d)
  • e += (conj?(a)*b) - (conj?(c)*d)
  • e += -(conj?(a)*b) - (conj?(c)*d)
• element-wise vector type-conversion (narrow, widen). Does copyv kernel already support type-conversion ?
  • a = type(conj?(b)) where type() can be S, D, C, Z and Half (H) and Complex-half (CH)
• Ideally, I would like to see fp16 Half (H) and Complex-half (CH) in the BLAS/BLIS API as well.

[devinamatthews]

@hominhquan for 2./3., are these building blocks for something like e = \sum_i conj?(a_i)*b_i?

[devinamatthews]

@hominhquan 4./5. are on the radar!

[devinamatthews]

@realab-ai thanks for letting us know about this! Would you mind opening an issue for this with some basic performance comparison data? I think that actually using the AVX2 level1v/f kernels from Zen could help a lot. I've heard conflicting things about the benefits of AVX512 for such operations due to the more severe throttling.

[devinamatthews]

@ct-clmsn I created a taxonomy of "feasible" unary and binary tensor operations in TBLIS, and these reduce naturally to matrix ops. That actually is sort of what prompted this issue. Hopefully those same operations would be those that slot into opt_einsum as basic kernels.

[hominhquan]

@hominhquan for 2./3., are these building blocks for something like e = \sum_i conj?(a_i)*b_i?

Kind of. I've seen some DSP algorithms on which they can be useful. They are like the next-gen of FMA operations with more flops.

[realab-ai]

@realab-ai thanks for letting us know about this! Would you mind opening an issue for this with some basic performance comparison data? I think that actually using the AVX2 level1v/f kernels from Zen could help a lot. I've heard conflicting things about the benefits of AVX512 for such operations due to the more severe throttling.

Ok. I'll do this recently.

[devinamatthews]

@hominhquan Cool. I ask because if the goal is N simultaneous element-wise multiplications where N > 1 but not necessary just 2 then this is more like a level1f kernel than level1. I'll add to the list.