File saved with JLD2 can't be read, from identical OS and JLD2 version

[Datseris]

Hello, I am facing a serious problem, namely that a file saved with JLD2 version 0.0.5 and FileIO cannot be read from the same OS with the same version of JLD2.

It is really weird for me because I have a bunch of files (computed with different parameters of a model) and most of them work perfectly.

Notice that for all of my files being computed, STDERR was empty, and STDOUT was exactly as expected (reporting some timings of computations and whatever).

I know paste my error messages:

which does not work. However, another file, produced by the identical code with one parameter different (positive instead of negative energy):

I have no reason to believe that there is something wrong with the data of the first case, as I have done the computation for one sub-case and it work.

I am more than willing to share the 2 files presented here if they can help (I really need some help since I cannot load some long-running cluster results).

[Datseris]

I have noticed that all my data that have negative energy cannot be loaded.

Can there be an issue with having - in the file name?

[ararslan]

Can there be an issue with having - in the file name?

If you save it as something like test.jld2 and try to load that, does it work?

[Datseris]

Unfortunately to save it with a different name, I would have to re-create the data and thus submit a lot of hours of cluster work.

Simply renaming the file did not fix the issue however. I will try to produce a similar dataset with a much smaller system on my laptop in the mean-time.

[ararslan]

If renaming the file didn't work then I wouldn't bother generating a new file just to test this. I think we can rule out an issue with the file name.

[Datseris]

I have re-run all my simulations, made extra care to be sure that JLD2 version is everywhere 0.0.5 and now I can load the data.

I wonder whether this was an issue on my end or something weird happened somehow. I am closing this assuming it was an issue on my end. If you think ottherwise you can reopen.

[Datseris]

@ararslan Hello, with many tears I return to this issue once more.

It happened to me again, and once again I can't load my data... :(

Versions:

 - FileIO                        0.6.1
 - JLD2                          0.0.5

I am 100% certain that my data has been produced on the same versions of FileIO and JLD2. I can also tell you with certainty that the data were saved on a linux.

The problem is the same for both linux or windows, when I try to load the file:

file = "husimis_B=10.0_Dk=0.2_wid=80_leng=100_w=30.0_V0=0.4_E=-0.1.jld2"
"husimis_B=10.0_Dk=0.2_wid=80_leng=100_w=30.0_V0=0.4_E=-0.1.jld2"

julia> a = load(file)
Error encountered while loading "/scratch05/datseris/KleinGraphene/data/husimis/husimis_B=10.0_Dk=0.2_wid=80_leng=100_w=30.0_V0=0.4_E=-0.1.jld2".
Fatal error:
ERROR: JLD2.InvalidDataException()
Stacktrace:
 [1] read_obj_start(::JLD2.MmapIO) at /home/datseris/.julia/v0.6/JLD2/src/object_headers.jl:47
 [2] isgroup(::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.RelOffset) at /home/datseris/.julia/v0.6/JLD2/src/object_headers.jl:74
 [3] getindex(::JLD2.Group{JLD2.JLDFile{JLD2.MmapIO}}, ::String) at /home/datseris/.julia/v0.6/JLD2/src/groups.jl:103
 [4] loadtodict!(::Dict{String,Any}, ::JLD2.JLDFile{JLD2.MmapIO}, ::String) at /home/datseris/.julia/v0.6/JLD2/src/loadsave.jl:114
 [5] #jldopen#31(::Array{Any,1}, ::Function, ::JLD2.##44#45, ::String, ::Vararg{String,N} where N) at /home/datseris/.julia/v0.6/JLD2/src/loadsave.jl:4
 [6] #load#43(::Array{Any,1}, ::Function, ::FileIO.File{FileIO.DataFormat{:JLD2}}) at /home/datseris/.julia/v0.6/JLD2/src/loadsave.jl:126
 [7] load(::FileIO.File{FileIO.DataFormat{:JLD2}}) at /home/datseris/.julia/v0.6/JLD2/src/loadsave.jl:126
 [8] eval(::Module, ::Any) at ./boot.jl:235
 [9] #load#20(::Array{Any,1}, ::Function, ::FileIO.File{FileIO.DataFormat{:JLD2}}) at /home/datseris/.julia/v0.6/FileIO/src/loadsave.jl:98
 [10] load(::FileIO.File{FileIO.DataFormat{:JLD2}}) at /home/datseris/.julia/v0.6/FileIO/src/loadsave.jl:86
 [11] #load#13(::Array{Any,1}, ::Function, ::String) at /home/datseris/.julia/v0.6/FileIO/src/loadsave.jl:52
 [12] load(::String) at /home/datseris/.julia/v0.6/FileIO/src/loadsave.jl:52

I can tell you something new, something that really might help.

This specific file contains a lot of Arrays where one of the dimensions is 1, like a vector with 1 element, or a 1x1000 matrix or a 1000x1000x1x6 array, etc.

Could it be that somehow this messes things up?

Once again I am willing to share my data if this will help you.

[Datseris]

Another info: My STDOUT is perfectly fine while my program runs, i.e. I get no warnings even during the saving of the file. My STDERR is complitely empty, I get no errors while the program runs and finishes.

[ararslan]

Hm, I'm really not sure and I don't have time to investigate deeply right now. @simonster, do you have any ideas?

[tscode]

I can confirm this (or at least a closely related) issue. I use JLD2 to save trained convolutional network models (using Knet.jl). When I save a freshly initialized model in the REPL, everything works fine, and I can load the model.

When I train the network (updating data in Float32 arrays) on a cluster-submission system, save it (A.jld2) and then try to load it, I get the same error as @Datseris. Interestingly, when I also save the model during the same run with julia's serialization functions, and then load it with deserialize in the REPL afterwards, I can again save the loaded model using JLD2 (B.jld2). This file, then, can be loaded successfully. So it is not the data itself that causes the corruption in A.jld2 [edit: this statement is wrong - see my answer below].

I looked at the hex-differences between the binaries A.jld2 and B.jld2. They are mostly identical - up to small regions, where A.jld2 contains a sequence of 0 entries. These regions (there were three of them in my case) always started where B.jld2 would read 4f4844520201360. Not sure if this helps, though. Unfortunately, I don't know anything about HDF5 files. [edit: this observation seems to miss the core issue - see below]

Currently, I have the feeling that this only happens when I submit my simulation through the submission system (condor), and not when I start the simulation on the respective nodes by hand [edit: this is also wrong - see below]. I am, however, not sure about that yet. Still, this feels quite puzzling.

The cluster nodes (regular PCs in my university's network) run Julia 0.6.2 and JLD 0.2.0 on linux.

[Datseris]

Thanks @tscode for giving much more details here, since I lack knowledge of saving formats etc.

I can also confirm that this happens for me when I use our submission system (called qsub) so maybe it can be somehow related.

You can also confirm that the STDERR is empty when the simulation is finished, even though the file cannot actually be loaded?

[tscode]

Yes, I can confirm that. JLD2 does not complain, and everything proceeds without errors.

However, my guess that starting the simulation by hand would make a difference is false. The problem is that different PCs on the cluster seem to create different versions of .jld2 files that are incompatible. I can now reliably create .jld2 files on node A, such that the file can be loaded from node A but not from other nodes B and C, and vice versa.

Note that all tested nodes (A, B, and C) in this example make use of the very same julia binaries and packages, since julia is only installed locally (not system-wide) in a shared folder that all systems have access to. They also have the same CPU and OS (Fedora 25).

On my laptop (with Arch linux, but same Julia and JLD versions), I can open none of the files created by A, B, or C. The crashes always happen due to os.signature = 0x00000000 in the function read_obj_start in the file object_headers.jl after the function has been called several times sucessfully.

I will try to create a minimal example that exhibits this behavior, but this could take a while.

Update: Okay, I have cornered the issue in my case. It seems that certain array sizes trigger the described behavior. If I run the code

using JLD2
weights = zeros(4, 4, 64, 64)
@save "test.jld2" weights

on one of the cluster-nodes, then I can load the file test.jld2 from the very same node without error. However, I cannot load it from other nodes or from my laptop. If the array size is changed, like weights = zeros(4, 4, 64, 8) or weights = zeros(1, 1, 64, 64), everything seems to work fine. I didn't do an exhaustive check for sizes, though, so I have no idea by which logic sizes / dimensions are affected.

[simonster]

My guess is that your cluster uses a network file system that is not handling mmap properly. If you're okay with using the jldopen interface, you could try:

f = jldopen(fn, true, true, true, IOStream)

which directs JLD2 not to use mmap. Unfortunately, this mode is many times slower, and it has never been tested in the wild (although it is tested in the package tests), so you may run into other issues.

[tscode]

You seem to be right. I cannot reproduce the error if I use jldopen in the way you suggested. That is fine for my use case.