How to compare Arrays of Numbers, e.g. Floats

[mauro3]

Is there (or should there be) a way to compare arrays of floats? This would entail first parsing the string into floats to then compare the floats approximately. A hack looks like:

function comp_str_reprs_of_arrays(ref, test_)
    arr_ref = eval(Meta.parse(ref)) # note that this makes a global variable, a bit sub-optimal
    arr_test = eval(Meta.parse(test_)) # note that this makes a global variable, a bit sub-optimal
    return size(arr_ref)==size(arr_test) && arr_ref ≈ arr_test
end

X = [1.56, 1.87]

@test_reference "reftest-files/X.txt" X by=comp_str_reprs_of_arrays

[mauro3]

I guess another way would be to store the reference array in a, say, JLD or BSON file. Which is what is done already with pictures.

I'd be happy to make a PR if this deemed a useful add-on.

[johnnychen94]

I haven't tried it but my guess is that JLD/BSON should just work. If it isn't then we might need to add some adaptors to https://github.com/JuliaTesting/ReferenceTests.jl/blob/master/src/fileio.jl

[mauro3]

Cool. So how do I use those files? Just by specifying the file extension?

[johnnychen94]

Yes, ReferenceTest delegates almost all the IO work to FileIO so as long as the format is registered in FileIO, it should be recognizable.

[mauro3]

Cool, this works:

comp(d1, d2) = all([ v1≈v2 for (v1,v2) in zip(values(d1), values(d2))])
d = Dict(:X=> X[inds])
@test_reference "reftest-files/X.bson" d by=comp

So, I guess it's a documentation issue? I can add an example for this and maybe other backends.

[johnnychen94]

That would be great! (and if possible also a piece of test)

We definitely need some rewrite of the docs using Documenter, which is currently an empty project.

[mauro3]

Do you know how to deal with JLD? This, and permutations, does not work:

@test_reference "x.jld" [1,2]
@test_reference "x.jld" ("x", [1,2])

the command for jld is

JLD.save("t.jld", "x", [1,2]) #or 
using FileIO; save(File(format"JLD","t"),"x", [1,2]) 

i.e. it needs both a name and a value.

[johnnychen94]

It seems that JLD.save also supports Dict input

save("x.jld", Dict("x"=>[1, 2]))
@test_reference "x.jld" Dict("x"=>[1,2])

[adrhill]

JLD2 worked out of the box for me:

julia> A = rand(20);

julia> @test_reference "myref.jld2" Dict("A" => A) by=(r,a)->isapprox(r["A"], a["A"], rtol=0.1)
┌ Info: Reference file for "myref.jld2" did not exist. It has been created
└   new_reference = "/Users/funks/myref.jld2"
- NEW CONTENT -----------------
Dict{String, Vector{Float64}} with 1 entry:
  "A" => [0.968553, 0.439909, 0.555604, 0.151386, 0.687479, 0.14697, 0.408348, …
-------------------------------
[ Info: Please run the tests again for any changes to take effect

julia> @test_reference "myref.jld2" Dict("A" => 0.95 * A) by=(r,a)->isapprox(r["A"], a["A"], rtol=0.1)
Test Passed
  Expression: true

julia> @test_reference "myref.jld2" Dict("A" => 0.8 * A) by=(r,a)->isapprox(r["A"], a["A"], rtol=0.1)
┌ Info: Reference Test for "myref.jld2" failed.
│   reference = "/Users/funks/myref.jld2"
└   actual = "/var/folders/74/wcz8c9qs5dzc8wgkk7839k5c0000gn/T/jl_aRRSda/myref.jld2"
- REFERENCE -------------------
Dict{String, Vector{Float64}} with 1 entry:
  "A" => [0.968553, 0.439909, 0.555604, 0.151386, 0.687479, 0.14697, 0.408348, …
-------------------------------
- ACTUAL ----------------------
Dict{String, Vector{Float64}} with 1 entry:
  "A" => [0.774842, 0.351927, 0.444483, 0.121109, 0.549983, 0.117576, 0.326679,…
-------------------------------
Replace reference with actual result? [y/n]
n
Test Failed at /Users/funks/.julia/packages/ReferenceTests/ehVLM/src/test_reference.jl:158
  Expression: false
ERROR: There was an error during testing

However the error message could be improved, as it gets cut off after a couple of dictionary entries, making debugging harder.
In this example the strings look identical:

julia> A[end] = 42;

julia> @test_reference "myref.jld2" Dict("A" => A) by=(r,a)->isapprox(r["A"], a["A"], rtol=0.1)
┌ Info: Reference Test for "myref.jld2" failed.
│   reference = "/Users/funks/myref.jld2"
└   actual = "/var/folders/74/wcz8c9qs5dzc8wgkk7839k5c0000gn/T/jl_H4C9fc/myref.jld2"
- REFERENCE -------------------
Dict{String, Vector{Float64}} with 1 entry:
  "A" => [0.968553, 0.439909, 0.555604, 0.151386, 0.687479, 0.14697, 0.408348, …
-------------------------------
- ACTUAL ----------------------
Dict{String, Vector{Float64}} with 1 entry:
  "A" => [0.968553, 0.439909, 0.555604, 0.151386, 0.687479, 0.14697, 0.408348, …
-------------------------------
Replace reference with actual result? [y/n]