Use prefetching

[mratsim]

Proposed by @data-man

Examples:

• https://gist.github.com/osimola/7917568
• In depth discussion on Stack Overflow.
• Torvalds thoughts.

Software prefetching is moronic.

It's a great way to generate almost-optimal behavior on one particular CPU with one particular cache setup and memory subsystem (and one particular load), but then it falls flat on its face whenever there is some other micro-architecture or cache layout, or when you have other things going on on that same machine.

I am still not sure where software prefetching is advantageous. The main use case would be tensor iterations but if tensors are contiguous, the next memory locations are loaded in cache anyway.
If it's not contiguous, what is the difference between prefetching and just plain loading the data at that location.

[mratsim]

Prefetching is probably useful when you can predict where to load the data but it looks random for the hardware prefetcher like for binary trees. This would be useful for tree algorithms and strided iteration for example.

Some links:

• https://stackoverflow.com/questions/7327994/prefetching-examples
• https://scripts.mit.edu/~birge/blog/accelerating-code-using-gccs-prefetch-extension/
• https://www.naftaliharris.com/blog/2x-speedup-with-one-line-of-code/
• https://lemire.me/blog/2018/04/30/is-software-prefetching-__builtin_prefetch-useful-for-performance/

Closing as this kind of low-level details will be handled in Laser