Sampling with evidence might produce slightly different probabilities due to random seed

[RenatoGeh]

Sampling with evidence might produce slightly different probabilities. I know this is most probably a result of chance, but might cause trouble with Travis or with someone else who might not go investigate the real cause of this test not passing and assume the code is faulty.

So to deal with this we should either set a fixed seed for this particular test (or all tests for that matter), increase our error tolerance or number of samples. This is such a trivial change that I don't think a PR is necessary, and this could be introduced in a later patch when real change is made.

Below is the error stracktrace.

9/14 Probabilistic/SamplingTest.jl
Sampling With Evidence: Test Failed at /home/renatogeh/.julia/packages/ProbabilisticCircuits/NTXbs/test/Probabilistic/SamplingTest.jl:77
  Expression: ≈(calc_prob_all[ind], hist[cur], atol = EPS)
   Evaluated: 0.7 ≈ 0.698978 (atol=0.001)
Stacktrace:
 [1] macro expansion at /home/renatogeh/.julia/packages/ProbabilisticCircuits/NTXbs/test/Probabilistic/SamplingTest.jl:77 [inlined]
 [2] macro expansion at /build/julia/src/julia-1.4.1/usr/share/julia/stdlib/v1.4/Test/src/Test.jl:1113 [inlined]
 [3] top-level scope at /home/renatogeh/.julia/packages/ProbabilisticCircuits/NTXbs/test/Probabilistic/SamplingTest.jl:40
    Fail 1
    Pass 19  (12.79 seconds)
ERROR: LoadError: Some tests did not pass: 19 passed, 1 failed, 0 errored, 0 broken.
in expression starting at /home/renatogeh/.julia/packages/ProbabilisticCircuits/NTXbs/test/runtests.jl:14
ERROR: Package ProbabilisticCircuits errored during testing
Stacktrace:
 [1] pkgerror(::String, ::Vararg{String,N} where N) at /build/julia/src/julia-1.4.1/usr/share/julia/stdlib/v1.4/Pkg/src/Types.jl:53
 [2] test(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; coverage::Bool, julia_args::Cmd, test_args::Cmd, test_fn::Nothing) at /build/julia/src/julia-1.4.1/usr/share/julia/stdlib/v1.4/Pkg/src/Operations.jl:1503
 [3] test(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; coverage::Bool, test_fn::Nothing, julia_args::Cmd, test_args::Cmd, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /build/julia/src/julia-1.4
.1/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:316
 [4] test(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}) at /build/julia/src/julia-1.4.1/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:303
 [5] #test#68 at /build/julia/src/julia-1.4.1/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:297 [inlined]
 [6] test at /build/julia/src/julia-1.4.1/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:297 [inlined]
 [7] #test#67 at /build/julia/src/julia-1.4.1/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:296 [inlined]
 [8] test at /build/julia/src/julia-1.4.1/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:296 [inlined]
 [9] test(::String; kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /build/julia/src/julia-1.4.1/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:295
 [10] test(::String) at /build/julia/src/julia-1.4.1/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:295
 [11] top-level scope at none:1

[guyvdbroeck]

Thanks @RenatoGeh that is a great point. We should add a keyword argument that can fix the random seed.

[guyvdbroeck]

Sampling with evidence was recently reimplemented and now has a random seed that can be fixed.