Moving function into a separate crate results in a more effective code for some reason

[newpavlov]

While working on optimizations for crypto-hashes I've notice a very strange behaviour described in the title. I've isolated the relevant code into this repository, so you can run it yourself.

Enabling lto produces the same slow result for separate case as for in-crate one. Optimal code generated only if #[inline] or #[inline(always)] used for compress function. Also generated assembly for two cases is quite different despite the identical code.

Probably it's due to some mis-optimization which gets turned off when function is in a different crate, but available for inlining.

UPD: It was reported on the reddit that 64-bit ARM shows the same performance for both cases.

[newpavlov]

Citing doener:

It's tied to the RC array. Moving just out of the original crate makes all the difference. Not sure why (yet).

OK, so with the array values available, LLVM can collapse two add instructions into a single lea instruction.
For example:

add    0x4(%r10),%edx # %r10 is the base address of RC
add    %ecx,%edx

Becomes:

lea    -0x173848aa(%rdi,%rax,1),%eax

Now the problem is that this causes frontend stalls. perf stat shows:

1,946,734,091 stalled-cycles-frontend # 65.05% frontend cycles idle ( +- 15.66% )

vs.

1,062,250,896 stalled-cycles-frontend # 44.28% frontend cycles idle ( +- 21.81% )

Which is an issue in Intel CPUs since Sandy Bridge, where the three operand LEA instruction has higher latency and limited dispatch port choices, see https://software.intel.com/en-us/node/544484

[samlh]

Possible LLVM fix: https://reviews.llvm.org/rL303333

[newpavlov]

The issue is still reproducible on Nightly 2018-10-28.