Measure execution speed

[bjorn3]

[bjorn3]

As of b533f91:

Benchmark #1: ./target/mod_bench
  Time (mean ± σ):      5.644 s ±  0.043 s    [User: 5.617 s, System: 0.013 s]
  Range (min … max):    5.578 s …  5.696 s
 
Benchmark #2: ./target/mod_bench_llvm_0
  Time (mean ± σ):      3.474 s ±  0.030 s    [User: 3.459 s, System: 0.007 s]
  Range (min … max):    3.436 s …  3.516 s
 
Benchmark #3: ./target/mod_bench_llvm_1
  Time (mean ± σ):      1.220 s ±  0.016 s    [User: 1.211 s, System: 0.004 s]
  Range (min … max):    1.196 s …  1.242 s
 
Benchmark #4: ./target/mod_bench_llvm_2
  Time (mean ± σ):     351.7 ms ±   1.6 ms    [User: 347.3 ms, System: 1.4 ms]
  Range (min … max):   350.2 ms … 355.0 ms
 
Benchmark #5: ./target/mod_bench_llvm_3
  Time (mean ± σ):     356.1 ms ±   4.2 ms    [User: 350.9 ms, System: 1.8 ms]
  Range (min … max):   351.2 ms … 363.0 ms
 
Summary
  './target/mod_bench_llvm_2' ran
    1.01 ± 0.01 times faster than './target/mod_bench_llvm_3'
    3.47 ± 0.05 times faster than './target/mod_bench_llvm_1'
    9.88 ± 0.09 times faster than './target/mod_bench_llvm_0'
   16.05 ± 0.14 times faster than './target/mod_bench'

mod_bench is using cg_clif, mod_bench_llvm_* is using cg_llvm using the respective opt-level.

Comparing only mod_bench and mod_bench_llvm_0 the last is ~1.65 times faster.

cc @sunfishcode (You may be interested in the speed of cranelift. Please note that cg_clif itself is not yet optimized for output quality.)

[sunfishcode]

Currently in Cranelift the IR verifier is enabled by default, which can take a lot of time. Can you benchmark with the "enable_verifier" setting disabled?

[bjorn3]

This is just execution speed.

[sunfishcode]

Ah, please update the issue title then :-). Also, you may want to try setting Cranelift's opt_level to best.

[bjorn3]

Compilation speed is at decent level already. Running hyperfine with opt_level set to best right now.

Edit: doesn't seem to change much: flags_builder.set("opt_level", "best").unwrap();

[bjorn3]

Compilation speed of libcore without optimizations for cg_clif and cg_llvm:

Benchmark #1: rustc -Zalways-encode-mir -Cpanic=abort -Zcodegen-backend=/Users/bjorn/Documents/rustc_codegen_cranelift/target/debug/librustc_codegen_cranelift.dylib -L crate=target/out --out-dir target/out target/libcore/src/libcore/lib.rs --crate-type lib --crate-name core
  Time (mean ± σ):     19.574 s ±  0.519 s    [User: 17.951 s, System: 1.171 s]
  Range (min … max):   18.884 s … 20.433 s
 
Benchmark #2: rustc -Zalways-encode-mir -Cpanic=abort target/libcore/src/libcore/lib.rs --crate-type lib --crate-name core
  Time (mean ± σ):     14.237 s ±  0.258 s    [User: 14.313 s, System: 0.539 s]
  Range (min … max):   13.845 s … 14.677 s
 
Summary
  'rustc -Zalways-encode-mir -Cpanic=abort target/libcore/src/libcore/lib.rs --crate-type lib --crate-name core' ran
    1.37 ± 0.04 times faster than 'rustc -Zalways-encode-mir -Cpanic=abort -Zcodegen-backend=/Users/bjorn/Documents/rustc_codegen_cranelift/target/debug/librustc_codegen_cranelift.dylib -L crate=target/out --out-dir target/out target/libcore/src/libcore/lib.rs --crate-type lib --crate-name core'

Edit: wait that's with semi debug mode compiled cg_clif. (cranelift itself is optimized but cg_clif is not)

Edit2: Using release mode for cg_clif:

Benchmark #1: rustc -Zalways-encode-mir -Cpanic=abort -Zcodegen-backend=/Users/bjorn/Documents/rustc_codegen_cranelift/target/release/librustc_codegen_cranelift.dylib -L crate=target/out --out-dir target/out target/libcore/src/libcore/lib.rs --crate-type lib --crate-name core
  Time (mean ± σ):     15.521 s ±  0.471 s    [User: 14.646 s, System: 0.600 s]
  Range (min … max):   14.824 s … 16.147 s
 
Benchmark #2: rustc -Zalways-encode-mir -Cpanic=abort target/libcore/src/libcore/lib.rs --crate-type lib --crate-name core
  Time (mean ± σ):     14.492 s ±  0.351 s    [User: 14.464 s, System: 0.590 s]
  Range (min … max):   13.761 s … 14.818 s
 
Summary
  'rustc -Zalways-encode-mir -Cpanic=abort target/libcore/src/libcore/lib.rs --crate-type lib --crate-name core' ran
    1.07 ± 0.04 times faster than 'rustc -Zalways-encode-mir -Cpanic=abort -Zcodegen-backend=/Users/bjorn/Documents/rustc_codegen_cranelift/target/release/librustc_codegen_cranelift.dylib -L crate=target/out --out-dir target/out target/libcore/src/libcore/lib.rs --crate-type lib --crate-name core'

[sunfishcode]

At a high level, it's not too surprising that Cranelift's execution speed on Rust would be in the ballpark of LLVM's O0 on Rust, because it's not doing any inlining. The rough short-term plan is to enable the MIR inliner to help with this.

There's probably a bunch of low-hanging fruit too, just making sure common Rust constructs are compiled well.

[lachlansneff]

Are you multithreading compilation? Cranelift is inherently very good at parallel compilation,

[bjorn3]

@lachlansneff No, rustc's TyCtxt is not thread safe (!Send + !Sync) and I believe cranelift's Module isn't either (!Sync)

[sunfishcode]

That's true, cranelift-codegen can be run with multiple instances in parallel, but cranelift-module doesn't yet make use of that.

[bjorn3]

As of ef5d161

mod_bench_inline is build with -Zmir-opt-level=3 for both itself and sysroot.

Note: this is using a different computer than the previous benchmark

Benchmark #1: ./target/out/mod_bench
  Time (mean ± σ):      6.790 s ±  0.195 s    [User: 6.774 s, System: 0.008 s]
  Range (min … max):    6.548 s …  7.159 s
 
Benchmark #2: ./target/out/mod_bench_inline
  Time (mean ± σ):      5.600 s ±  0.136 s    [User: 5.589 s, System: 0.004 s]
  Range (min … max):    5.422 s …  5.887 s
 
Benchmark #3: ./target/out/mod_bench_llvm_0
  Time (mean ± σ):      4.402 s ±  0.197 s    [User: 4.391 s, System: 0.004 s]
  Range (min … max):    4.165 s …  4.784 s
 
Benchmark #4: ./target/out/mod_bench_llvm_1
  Time (mean ± σ):      1.627 s ±  0.036 s    [User: 1.624 s, System: 0.000 s]
  Range (min … max):    1.589 s …  1.693 s
 
Benchmark #5: ./target/out/mod_bench_llvm_2
  Time (mean ± σ):     423.6 ms ±   2.6 ms    [User: 422.0 ms, System: 0.0 ms]
  Range (min … max):   420.0 ms … 428.3 ms
 
Benchmark #6: ./target/out/mod_bench_llvm_3
  Time (mean ± σ):     422.9 ms ±   2.5 ms    [User: 420.8 ms, System: 0.4 ms]
  Range (min … max):   420.5 ms … 429.3 ms
 
Summary
  './target/out/mod_bench_llvm_3' ran
    1.00 ± 0.01 times faster than './target/out/mod_bench_llvm_2'
    3.85 ± 0.09 times faster than './target/out/mod_bench_llvm_1'
   10.41 ± 0.47 times faster than './target/out/mod_bench_llvm_0'
   13.24 ± 0.33 times faster than './target/out/mod_bench_inline'
   16.05 ± 0.47 times faster than './target/out/mod_bench'

[bjorn3]

Flamegraph for mod_bench_inline

[sunfishcode]

I took a quick look at the code. Here are some notes:

Some of the code will get better once Cranelift has more support for i8 and the associated workarounds are removed.

Is -Zmir-opt-level=3 in use when building libcore? I'm seeing things like core::cmp::impls::<impl core::cmp::PartialOrd for u32>::lt not being inlined, which is the kind of thing we're really going to want to inline.

If I'm reading this correctly, there's a small memmove in there, which the small memcpy/memmove/memset optimization should help with, once bytecodealliance/cranelift#586 is fixed.

There's a codegen abort when I enable opt_level=best. I'll investigate that.

[bjorn3]

Is -Zmir-opt-level=3 in use when building libcore?

Yes, the whole sysroot.

If I'm reading this correctly, there's a small memmove in there

I am currently using my own code for copying locals:

https://github.com/bjorn3/rustc_codegen_cranelift/blob/29b4c34c540db66fd13b63037109528eeb97146b/src/common.rs#L416-L442

So that memmove comes from the copy_nonoverlapping intrinsic:
https://github.com/bjorn3/rustc_codegen_cranelift/blob/e1fc9a56d9ec243562e88740c3c86895ab3fd846/src/intrinsics.rs#L142

Which likely came fromcore::mem::swap: https://github.com/rust-lang/rust/blob/b76ee83254ec0398da554f25c2168d917ba60f1c/src/libcore/iter/range.rs#L228

There's a codegen abort when I enable opt_level=best. I'll investigate that.

😭