stored type DataFrames.Index does not match currently loaded type

[floswald]

hi!
i'm not sure how to get around this error. I save a dict with several dataframes in one session from within Module mig. then in another session i want to load the object. Am I doing something wrong here?

using mig
julia> d=mig.load("/Users/florianoswald/git/migration/data/exp_shockp.JLD")
ERROR: stored type DataFrames.Index does not match currently loaded type

mig is

using GLM, MOpt.transpose, PDMats, Distributions, DataFrames, DataFramesMeta, ApproXD, Lazy, Optim, HDF5, JLD

[joshjob42]

I got the same message when I saved a BitArray. However, I can still load the array (saved as variable "arr") if I say "load(filename,"arr")", it just doesn't work when I simply run "load(filename)".

[timholy]

When you reload the object, are you within the same module?

If you can reproduce with an explicit, short testcase, that would be tremendously helpful.

[floswald]

Not sure what you mean. At the moment my module imports hdf5 via using, so
then I do

mymod.load(object)

With the error. Should I rather define a function inside the module

function loaddata()
x = load(file)
Return x
End

?

On Sunday, 5 October 2014, Tim Holy notifications@github.com wrote:

When you reload the object, are you within the same module?

If you can reproduce with an explicit, short testcase, that would be
tremendously helpful.

—
Reply to this email directly or view it on GitHub
#158 (comment).

[timholy]

Maybe. I'm not quite sure what you're trying. My main point is that it may be a scoping problem. (The converse question is: if you save the same data from the REPL, does loading it from the REPL work?)

[floswald]

did verify that the below works. as far as I can see that's more or less what I'm doing, so I'm a bit stuck. currently waiting for the computation to finish so I can try and save from the REPL. I really just have a dict like the one in the example below.

module mymod

using DataFrames, HDF5, JLD, DataFramesMeta

function makedat()
    df = DataFrame(g=["a","a","b","b","c","c"],v=rand(6))
    bydf = @by(df,:g,v=mean(:v))
    dict = ["df" => df,"bydf"=>bydf]
    save("/Users/florianoswald/test.JLD",dict)
end

end

julia> include("src/migsrc/test.jl")
Warning: replacing module mymod

julia> mymod.makedat()

julia> mymod.load("/Users/florianoswald/test.JLD")
Dict{Union(UTF8String,ASCIIString),Any} with 2 entries:
  "bydf" => 3x2 DataFrame…
  "df"   => 6x2 DataFrame…

[floswald]

so here is what I have got in my real world problem. I tried saving and loading from the REPL, same result. mig is a module that uses HDF5,JLD.

julia> p
Dict{ASCIIString,DataFrame} with 8 entries:
  "fromj"      => 46x12 DataFrame…
  "toj"        => 46x12 DataFrame…
  "Baseline"   => 1067671x34 DataFrame…
  "Policy"     => 1067671x34 DataFrame…
  "fromj_rent" => 46x4 DataFrame…
  "fromj_own"  => 46x4 DataFrame…
  "toj_own"    => 46x4 DataFrame…
  "toj_rent"   => 46x4 DataFrame…

julia> mig.save("/home/florian_o/test.JLD",p)

julia> x = mig.load("/home/florian_o/test.JLD")
ERROR: stored type DataFrames.Index does not match currently loaded type
 in jldatatype at /home/florian_o/.julia/v0.3/HDF5/src/jld_types.jl:597
 in read at /home/florian_o/.julia/v0.3/HDF5/src/JLD.jl:299
 in read_ref at /home/florian_o/.julia/v0.3/HDF5/src/JLD.jl:441
 in jlconvert at /home/florian_o/.julia/v0.3/HDF5/src/jld_types.jl:357
 in read_scalar at /home/florian_o/.julia/v0.3/HDF5/src/JLD.jl:327
 in read at /home/florian_o/.julia/v0.3/HDF5/src/JLD.jl:299
 in read at /home/florian_o/.julia/v0.3/HDF5/src/JLD.jl:284
 in anonymous at /home/florian_o/.julia/v0.3/HDF5/src/JLD.jl:915
 in jldopen at /home/florian_o/.julia/v0.3/HDF5/src/JLD.jl:205
 in load at /home/florian_o/.julia/v0.3/HDF5/src/JLD.jl:914

[simonster]

If you can come up with a test case that reproduces this or send a file to simon@simonster.com, I'll take a look.

[floswald]

thanks for that, i'll see what i can do. it's odd because if I don't include the two very big objects "Baseline" and "Policy", everything works fine. must be something i'm doing to those dataframes? the fact they are so big makes it pretty much impossible to send as an email though i think. anyway, do you think it's useful to look at those objects? maybe we can use some filesharing thing?

[simonster]

Can you give the output of DataFrames.showcols for those objects? If you save just the first 10 or so rows of each of those DataFrames, can you reproduce the issue?

[simonster]

This should have been fixed for the JLD 0.1+ format along with #198. Ideally we should make the same change in the JLD 0.0 reader (#205).