Another interesting paper related to this idea

[jukofyork]

Not all Layers of LLMs are Necessary during Inference (v2)

[shamanez]

Interesting!

[jukofyork]

I've also read another paper along these lines too (but can't find it atm) that suggests the layers marked by the yellow areas at the bottom:

are just doing some kind of "averaging", and IIRC the paper suggests replacing these penultimate layers with some other much cheaper operation (IIRC, it was a training time modification rather than post-training though).

[jukofyork]

ShortGPT: Layers in Large Language Models are More Redundant Than You Expect

Can't find the paper about "averaging" though.