Other timing tools

[aminya]

Is there any other timing measure that shows the effect of using precompiles (in a dynamic manner without sysimage) other than what we have in @snoopi and snoopi_bot?

Is @timev a good measure?

[aminya]

I used @timev, and clearly shows the improvements 🚀 . People thought that the only improvement that we get is in the inference time.

For example, for Zygote:

Running @timev in running the tests shows that we get:

• 10s raw improvement
• 23 MB less memory usage
• 1GiB less memory allocation.

Details

│ Precompile Deactivated Benchmark
# for using Zygote:
└ Inference time (ms): 	23654.842
@timev result (This has some noise).
391.374697 seconds (628.72 M allocations: 31.522 GiB, 4.16% gc time)
elapsed time (ns): 391374697041
gc time (ns):      16295209095
bytes allocated:   33846029551
pool allocs:       628513544
non-pool GC allocs:190530
malloc() calls:    11983
realloc() calls:   85
GC pauses:         503
full collections:  9

│ Precompile Activated Benchmark
┌ Info: 
└ Inference time (ms): 	22565.431
@timev result (This has some noise).
382.785964 seconds (605.11 M allocations: 30.360 GiB, 3.93% gc time)
elapsed time (ns): 382785964109
gc time (ns):      15042733929
bytes allocated:   32598547445
pool allocs:       604912736
non-pool GC allocs:184276
malloc() calls:    11737
realloc() calls:   81
GC pauses:         467
full collections:  6

https://github.com/aminya/Zygote.jl/runs/646094164?check_suite_focus=true#step:6:123

[aminya]

If this is the case, should I just use @timev for benchmarking isntead of @snoopi? People seem to have better intuition for it.

    julia_code_timev = """
    using $package_name
    @timev begin
        $(string(snoop_script));
    end
    @info("The above is @timev result (This has some noise).")
    """
    julia_cmd_timev = `julia --project=@. -e $julia_code_timev`

https://github.com/aminya/SnoopCompile.jl/blob/4c5556de806fd2c9fcd7c82e606c6162f102b195/src/bot/snoopi_bench.jl#L33-L40

[timholy]

@snoopi is for discovery and for measuring just one thing, inference time. @timev measures a whole bunch of things, including load time, LLVM time, execution time, etc. In the end it's closer to what people will want to reduce, so it's not bad to use it to decide whether the precompiles are worth it. E.g., #74.

In general I worry that too many people use SnoopCompile unthinkingly; for me, the "discovery" part is by far the most important and I generally prefer to write the precompile file by hand. But I recognize I'm probably an outlier. Recommending @timev might help people realize that, for example, setting a higher tmin (and thus precompiling less) might actually reduce times they care about.

[aminya]

People are less concerned with the precompiling time, but with its effect on the performance. Using @timev helps with this for sure. I think I can close this, as this will be solved in #71