Use ChainRulesTestUtils.jl

[mattBrzezinski]

ChainRulesTestUtils.jl is now in the general registry, this is working towards using that package as a test dependency, see #166

[oxinabox]

I suspect this is because ChainRules only allows FiniteDifferences v0.7
but ChainRulesTestUtils allows FiniteDifferences v0.7, 0.8, and 0.9.

We should add tests to ChainRulesTestUtils to prove it supports everything.

In the short term maybe a patch release that cuts support down to v0.7 in ChainRulesTestUtils and a then lower bounding to that patch.

[mattBrzezinski]

I suspect this is because ChainRules only allows FiniteDifferences v0.7
but ChainRulesTestUtils allows FiniteDifferences v0.7, 0.8, and 0.9.

We should add tests to ChainRulesTestUtils to prove it supports everything.

In the short term maybe a patch release that cuts support down to v0.7 in ChainRulesTestUtils and a then lower bounding to that patch.

When running this branch and doing add FiniteDifferences@0.7 the tests still fail. Looking into the Manifest.toml I can see that the FiniteDifferences 0.7.2 is set, but ChainRulesTestUtils does not appear.

This is probably because it's a test only dependency. But I wonder how Julia handles this under the hood, when does it use the highest version of FiniteDifferences for ChainRulesTestUtils? Or the one already in the Manifest.toml?

EDIT: Just discussed with EricD about this, everything should be using FiniteDifferences@0.7 since it's being set in ChainRules.

[oxinabox]

Hmm, this will require further investigation then.

I was fairly sure we were careful not to change anything in the source, right?

[mattBrzezinski]

@willtebbutt

[nickrobinson251]

The stacktrace leads to the rrule_test() and more specifically this line:

x̄s_fd = j′vp(fdm, f, ȳ, xs...)

Shouldn't that j'vp be _make_fdm_call?

[mattBrzezinski]

Using _make_fdm_call fails the existing test in ChainRulesTestUtils.jl with

  MethodError: no method matching _make_fdm_call(::FiniteDifferences.var"#30#31"{FiniteDifferences.var"#42#43"{FiniteDifferences.var"#55#56"{FiniteDifferences.var"#57#58"{var"#fst#5"},FiniteDifferences.var"#77#79"{Tuple{Float64,Float64},Tuple{Array{Float64,1},Array{Float64,1}},Tuple{typeof(first),typeof(first)},Array{Int64,1}}},Int64},Array{Float64,1},Array{Float64,1},Int64}, ::Float64)
  Closest candidates are:
    _make_fdm_call(::Any, ::Any, !Matched::Any, !Matched::Any, !Matched::Any) at /Users/mattbr/.julia/dev/ChainRulesTestUtils.jl/src/ChainRulesTestUtils.jl:18

[oxinabox]

Other than this last thing it LGTM

[mattBrzezinski]

@nickrobinson251 any comments?

[nickrobinson251]

🚀

[nickrobinson251]

we should open an issue on ChainRulesTestUtils about whether or not we want this to be exported / part of the public API