Error passing results from loo to compare

[sethaxen]

Originally reported by user uferc on Julia Discourse: https://discourse.julialang.org/t/bayesian-model-comparison-selection/80420

julia> using ArviZ

julia> idata1 = load_arviz_data("centered_eight");

julia> idata2 = load_arviz_data("non_centered_eight");

julia> model_dict = Dict("model1" => idata1, "model2" => idata2)
Dict{String, InferenceData} with 2 entries:
  "model1" => InferenceData with groups:…
  "model2" => InferenceData with groups:…

julia> compare(model_dict)  # passing InferenceData is fine
2×10 DataFrame
 Row │ name    rank   loo       p_loo     d_loo     weight   se       dse        warning  loo_scale 
     │ String  Int64  Float64   Float64   Float64   Float64  Float64  Float64    Bool     String    
─────┼──────────────────────────────────────────────────────────────────────────────────────────────
   1 │ model2      0  -30.6876  0.842156  0.0           1.0  1.3653   0.0          false  log
   2 │ model1      1  -30.8105  0.954222  0.122985      0.0  1.42713  0.0859593    false  log

julia> loo1 = loo(idata1)
1×9 DataFrame
 Row │ loo       loo_se   p_loo     n_samples  n_data_points  warning  loo_i                              pareto_k ⋯
     │ Float64   Float64  Float64   Int64      Int64          Bool     PyObject                           PyObject ⋯
─────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1 │ -30.8105  1.42713  0.954222       2000              8    false  PyObject <xarray.DataArray 'loo_…  PyObject ⋯
                                                                                                   2 columns omitted

julia> loo2 = loo(idata2)
1×9 DataFrame
 Row │ loo       loo_se   p_loo     n_samples  n_data_points  warning  loo_i                              pareto_k ⋯
     │ Float64   Float64  Float64   Int64      Int64          Bool     PyObject                           PyObject ⋯
─────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1 │ -30.6876   1.3653  0.842156       2000              8    false  PyObject <xarray.DataArray 'loo_…  PyObject ⋯
                                                                                                   2 columns omitted

julia> loo_dict = Dict("model1" => loo1, "model2" => loo2)  #  Passing loo results is not 
Dict{String, DataFrames.DataFrame} with 2 entries:
  "model1" => 1×9 DataFrame…
  "model2" => 1×9 DataFrame…

julia> compare(loo_dict)
ERROR: PyError ($(Expr(:escape, :(ccall(#= /home/sethaxen/.julia/packages/PyCall/7a7w0/src/pyfncall.jl:43 =# @pysym(:PyObject_Call), PyPtr, (PyPtr, PyPtr, PyPtr), o, pyargsptr, kw))))) <class 'TypeError'>
TypeError('Encountered error in ic computation of compare.')
  File "/home/sethaxen/.julia/conda/3/lib/python3.9/site-packages/arviz/stats/stats.py", line 173, in compare
    raise e.__class__("Encountered error in ic computation of compare.") from e

The issue is that we convert the arviz.ELPDData on the Julia side to be a DataFrame, but we don't currently convert it back when calling compare, so it encounters a type it doesn't know. We can currently work around this by calling ArviZ.arviz.loo directly, which bypasses the conversion:

julia> loopy1 = ArviZ.arviz.loo(idata1);

julia> loopy2 = ArviZ.arviz.loo(idata2);

julia> loopy_dict = Dict("model1" => loopy1, "model2" => loopy2)
Dict{String, PyCall.PyObject} with 2 entries:
  "model1" => PyObject Computed from 2000 by 8 log-likelihood matrix…
  "model2" => PyObject Computed from 2000 by 8 log-likelihood matrix…

julia> compare(loopy_dict)
2×10 DataFrame
 Row │ name    rank   loo       p_loo     d_loo     weight   se       dse        warning  loo_scale 
     │ String  Int64  Float64   Float64   Float64   Float64  Float64  Float64    Bool     String    
─────┼──────────────────────────────────────────────────────────────────────────────────────────────
   1 │ model2      0  -30.6876  0.842156  0.0           1.0  1.3653   0.0          false  log
   2 │ model1      1  -30.8105  0.954222  0.122985      0.0  1.42713  0.0859593    false  log

[ahartikainen]

Is there a reason we can not accept a more broader range of types in arviz python side? Would this work if we would accept dataframes?

I think any mapping should work assuming we can access all the correct methods / variables?

[sethaxen]

I think either way we will need to convert the object to one Python ArviZ recognizes before sending it over. Python will expect at least some object methods to be available, and this is not a syntax Julia objects use (because Julia is functional). Also, a DataFrames.DataFrame is not automatically converted to a Pandas.Dataframe when passed to Python.

Actually, I may have already solved this for plot_elpd, in which case this is a really easy fix:

ArviZ.jl/src/plots.jl

Lines 33 to 40 in 71aebbe

	function convert_arguments(::typeof(plot_elpd), data, args...; kwargs...)
	dict = Dict(k => try
	topandas(Val(:ELPDData), v)
	catch
	convert_to_inference_data(v)
	end for (k, v) in pairs(data))
	return tuple(dict, args...), kwargs
	end