Allocation tests fail in Julia v1.5+

[nickrobinson251]

• the tests:
  

  Nabla.jl/test/sensitivities/functional/functional.jl

  Lines 230 to 238 in da2b7ef

  	# Check that the number of allocations which happen in the reverse pass of `map` and
  	# `broadcast` is invariant of the size of the data structure of which we are
  	# mapping / broadcasting.
  	for f in [:map, :broadcast]
  	let
  	@eval foo_small() = sum($f(tanh, Leaf(Tape(), randn(10, 10))))
  	@eval foo_large() = sum($f(tanh, Leaf(Tape(), randn(10, 100))))
  	@test allocs(@benchmark foo_small()) == allocs(@benchmark foo_large())
  	@test allocs(@benchmark ∇(foo_small())) == allocs(@benchmark ∇(foo_large()))

• the failures:

Functional: Test Failed at /home/pkgeval/.julia/packages/Nabla/D60dX/test/sensitivities/functional/functional.jl:237
  Expression: allocs(#= /home/pkgeval/.julia/packages/Nabla/D60dX/test/sensitivities/functional/functional.jl:237 =# @benchmark(foo_small())) == allocs(#= /home/pkgeval/.julia/packages/Nabla/D60dX/test/sensitivities/functional/functional.jl:237 =# @benchmark(foo_large()))
   Evaluated: 15 == 14
Stacktrace:
 [1] top-level scope at /home/pkgeval/.julia/packages/Nabla/D60dX/test/sensitivities/functional/functional.jl:237
 [2] top-level scope at /workspace/srcdir/usr/share/julia/stdlib/v1.5/Test/src/Test.jl:1114
 [3] top-level scope at /home/pkgeval/.julia/packages/Nabla/D60dX/test/sensitivities/functional/functional.jl:5
Functional: Test Failed at /home/pkgeval/.julia/packages/Nabla/D60dX/test/sensitivities/functional/functional.jl:238
  Expression: allocs(#= /home/pkgeval/.julia/packages/Nabla/D60dX/test/sensitivities/functional/functional.jl:238 =# @benchmark(∇(foo_small()))) == allocs(#= /home/pkgeval/.julia/packages/Nabla/D60dX/test/sensitivities/functional/functional.jl:238 =# @benchmark(∇(foo_large())))
   Evaluated: 53 == 52
Stacktrace:
 [1] top-level scope at /home/pkgeval/.julia/packages/Nabla/D60dX/test/sensitivities/functional/functional.jl:238
 [2] top-level scope at /workspace/srcdir/usr/share/julia/stdlib/v1.5/Test/src/Test.jl:1114
 [3] top-level scope at /home/pkgeval/.julia/packages/Nabla/D60dX/test/sensitivities/functional/functional.jl:5

• It's unclear if the failures are with the map or broadcast version of the test

[nickrobinson251]

using #186 shows it's f = broadcast that fails

[nickrobinson251]

I don't know what allocs(@benchmark(... is returning, and if it's trustworthy?

If we switch to @ballocated:

diff --git a/Project.toml b/Project.toml
index e4aa578..c3545aa 100644
--- a/Project.toml
+++ b/Project.toml
@@ -11,6 +11,7 @@ SpecialFunctions = "276daf66-3868-5448-9aa4-cd146d93841b"
 Statistics = "10745b16-79ce-11e8-11f9-7d13ad32a3b2"

 [compat]
+BenchmarkTools = "0.5"
 DiffRules = "^0.0"
 DualNumbers = ">=0.6.0"
 FDM = "^0.6"
diff --git a/test/sensitivities/functional/functional.jl b/test/sensitivities/functional/functional.jl
index 7a4b6b4..8e462ba 100644
--- a/test/sensitivities/functional/functional.jl
+++ b/test/sensitivities/functional/functional.jl
@@ -234,8 +234,8 @@ using DiffRules: diffrule, hasdiffrule
         let
             @eval foo_small() = sum($f(tanh, Leaf(Tape(), randn(10, 10))))
             @eval foo_large() = sum($f(tanh, Leaf(Tape(), randn(10, 100))))
-            @test allocs(@benchmark foo_small()) == allocs(@benchmark foo_large())
-            @test allocs(@benchmark ∇(foo_small())) == allocs(@benchmark ∇(foo_large()))
+            @test (@ballocated foo_small()) == (@ballocated foo_large())
+            @test (@ballocated ∇(foo_small())) == (@ballocated ∇(foo_large()))
         end
     end

then tests fail on Julia 1.4 (and 1.5 but with slightly different numbers):

f = map: Test Failed at /Users/nick/repos/Nabla.jl/test/sensitivities/functional/functional.jl:237
  Expression: #= /Users/nick/repos/Nabla.jl/test/sensitivities/functional/functional.jl:237 =# @ballocated(foo_small()) == #= /Users/nick/repos/Nabla.jl/test/sensitivities/functional/functional.jl:237 =# @ballocated(foo_large())
   Evaluated: 2144 == 16608 
...
...
f = map: Test Failed at /Users/nick/repos/Nabla.jl/test/sensitivities/functional/functional.jl:238
  Expression: #= /Users/nick/repos/Nabla.jl/test/sensitivities/functional/functional.jl:238 =# @ballocated(∇(foo_small())) == #= /Users/nick/repos/Nabla.jl/test/sensitivities/functional/functional.jl:238 =# @ballocated(∇(foo_large()))
   Evaluated: 5184 == 34112
...
...
f = broadcast: Test Failed at /Users/nick/repos/Nabla.jl/test/sensitivities/functional/functional.jl:237
  Expression: #= /Users/nick/repos/Nabla.jl/test/sensitivities/functional/functional.jl:237 =# @ballocated(foo_small()) == #= /Users/nick/repos/Nabla.jl/test/sensitivities/functional/functional.jl:237 =# @ballocated(foo_large())
   Evaluated: 2144 == 16608
...
...
f = broadcast: Test Failed at /Users/nick/repos/Nabla.jl/test/sensitivities/functional/functional.jl:238
  Expression: #= /Users/nick/repos/Nabla.jl/test/sensitivities/functional/functional.jl:238 =# @ballocated(∇(foo_small())) == #= /Users/nick/repos/Nabla.jl/test/sensitivities/functional/functional.jl:238 =# @ballocated(∇(foo_large()))
   Evaluated: 5232 == 34160

[nickrobinson251]

...any objections to just deleting these test, @willtebbutt ?

[willtebbutt]

IIRC these were designed to ensure that the number of allocations isn't proportional to the size of the inputs. i.e. if we change something and all of a sudden these operations produce an allocation for each element of the input matrices, that would be very bad, which is why I was attempting to count the total number of allocations as opposed to the total amount of memory allocated.

It might be possible to make these tests pass by just making foo_large operate on a 10 x 11 matrix, rather than 10 x 100. Either way, having some kind of test like this seems like a good idea to me.

[nickrobinson251]

well, i guess i don't understand what @ballocated outputs and what allocs outputs. Since for all Julia versions the ballocated numbers are very different between the _small and _large cases (see above), whereas for allocs there's only a change in v1.5, where for broadcast the _small case is 1 bigger than the _large cases.

[iamed2]

I answered on Slack so I'll answer here:

allocated = amount of memory allocated
allocs = number of times memory is allocated