@btime alters subsequent @time result

[chunjiw]

Also posted in discource
Minimal example:

using BenchmarkTools
struct S
end
(s::S)() = 1
let
    s = S()
    # @btime $s()
    @time s()
end

From which I get output 0.000000 seconds. No surprise here since the function should not have any allocations. But things change if I add @btime before @time like so:

let
    s = S()
    @btime $s()
    @time s()
end

From which I get output

  1.266 ns (0 allocations: 0 bytes)
  0.000006 seconds (8 allocations: 384 bytes)

So @time depends on previous seemingly unrelated calls.

I tried @allocated and it correctly prints out 0:

let
    s = S()
    @btime $s()
    println(@allocated s())
end

If the code is in a function, then @time output is correct, as pointed out in discourse. But personally I think this is still a bug.

julia> versioninfo()
Julia Version 1.10.2
Commit bd47eca2c8a (2024-03-01 10:14 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 12 × Intel(R) Core(TM) i7-6850K CPU @ 3.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, broadwell)
Threads: 1 default, 0 interactive, 1 GC (on 12 virtual cores)

Julia is installed via juliaup.

[chunjiw]

@mbauman just provided a much better MWE:
https://discourse.julialang.org/t/btime-alters-subsequent-time-result/111384/5

[Zentrik]

I suspect this is due to the compiler not optimising the code as well in the second case, e.g. replace @time with @timed and you will see less allocations (@time calls @timed internally). Given wrapping the code in a function makes the allocation disappear I'm not sure how to look at the compiled code and verify.