aie2p transformer/conv kernels from an engine build: which to upstream, and as kernels or example edits?

[atassis]

While building an open-source inference engine on aie2p (NPU2), I've written a set of small, generic
transformer/conv kernels. A few look useful beyond my engine, so before I open a stack of PRs I want to
check appetite and preferred shape.

As a concrete example I just opened #3410, a LayerNorm numerical fix. The exported bf16 layer_norm in
aie_kernels/aie2p/layer_norm.cc accumulates the mean sum in bf16 and computes the variance as
E[x^2] - mean^2, so its error grows with the row length: measured rel-L2 rises from 1.1% at cols=2048
to 1.9% at 4096 (input uniform[-1,1], vs an f32 host reference). The file already has a
layer_norm_welford, but it's float-only and reduces over the row axis rather than the feature axis, so
it can't back the per-row bf16 path as-is. The fix accumulates the mean in f32 and centers the variance
(two-pass); the interface and the eps/gamma/beta constants are unchanged, the error goes flat (~0.6%),
and the PR has the before/after plus a host-reference test.

Beyond that, in rough order of readiness, I also have: an extension of basic/transposes to runtime-dim
bf16/f32 (it currently covers compile-time int8/16/32); a depthwise 1D conv on sliding_mul; a
higher-precision f32 2^x as an alternative to the aie::exp2-based bf16_exp path, for
softmax/sigmoid cases that need more than bf16 precision (the note in softmax.cc already flags that
sensitivity); and a couple of fused mmul epilogues (LayerNorm, activations) that are parameterized
rather than tied to my model. Full list if it's useful.

Two questions:

1. Do you want reference kernels at this granularity in aie_kernels/aie2p/, or keep it lean?
2. For things that extend an existing example (transpose, activations), do you prefer editing the
   example or a separate kernel?

Each would come with a host-reference test and its aie.iron.kernels factory. #3410 is the first; I'll
follow your steer on the rest.

[hunhoffe]

@atassis Thank you for #3410! I'm excited to add to the open source kernels here and in amd/IRON. To be honest, we still need to do a great deduplication of kernels between here and amd/IRON, but it's just never made it to the top of the list so far.

To answer your questions:

1. Yes, add to aie_kernels/aie2p -- we are eager to add to our library of tested/useful kernels. I do want to try to keep the kernels there (when possible) well tested and somewhat robust to different datatypes. We also want to continue to support aie2 kernels, so (as your time allows) checking to see if your improvements port well would be very appreciated. If you don't have time/interest in porting, and if you find other issues/gaps/inconsistencies in the kernels, please do create issues so these things don't fall off the radar.
2. Extend the existing example when there is one but please also edit/update the pedagogical materials and documentation that reference it.

I look forward to seeing your future PRs!

[atassis]

This all makes sense, I can do all that.
On IRON <-> kernel duplication you mentioned: I think that's one of the no-contract boundaries from the contract-first view #3390 - the copies drift because nothing frozen sits between them. Whenever it's worth priotitizing, the low-risk first step is singe-sourcing the kernels here (IRON already depends on mlir-aie) rather than anything structural. Happy to help with that when the time comes!

[andrej]

We would strongly welcome reducing the kernel code duplication between IRON and MLIR-AIE. We had plans to have IRON be the "operator library", but before we can migrate all operators there, getting rid of the code duplication first is more important.

So yes -- if you wanted to migrate some kernels, even incrementally, from IRON back to MLIR-AIE -- that would be a great contribution aiding in maintainability.