AstroIO required when it shouldn't be

[junglegobs]

I load a lot (~10,000) JLD2 files using collect_results from DrWatson.jl. For a couple of files it's fine, but for the ~10,000 that I just mentioned I get the error that AstrioIO is required. This strikes me as very strange, since it's a rather obscure package which I definitely don't need to load a JLD2 file.

Stacktrace:

Error encountered while load FileIO.File{FileIO.DataFormat{:Gadget2}, String}("/home/u0128861/Desktop/storage-and-LOLE/results/monte_carlo/default_nP=1/mc_year_1016.jld2").

Fatal error:
ERROR: LoadError: ArgumentError: Package AstroIO [c85a633c-0c3f-44a2-bffe-7f9d0681b3e7] is required but does not seem to be installed:
 - Run `Pkg.instantiate()` to install all recorded dependencies.

Stacktrace:
  [1] _require(pkg::Base.PkgId)
    @ Base ./loading.jl:1012
  [2] require(uuidkey::Base.PkgId)
    @ Base ./loading.jl:936
  [3] #34
    @ ~/.julia/packages/FileIO/DNKwN/src/loadsave.jl:203 [inlined]
  [4] lock(f::FileIO.var"#34#35"{Base.PkgId}, l::ReentrantLock)
    @ Base ./lock.jl:187
  [5] action(::Symbol, ::Vector{Union{Base.PkgId, Module}}, ::FileIO.Formatted; options::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ FileIO ~/.julia/packages/FileIO/DNKwN/src/loadsave.jl:203
  [6] action
    @ ~/.julia/packages/FileIO/DNKwN/src/loadsave.jl:197 [inlined]
  [7] action(::Symbol, ::Vector{Union{Base.PkgId, Module}}, ::Symbol, ::String; options::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ FileIO ~/.julia/packages/FileIO/DNKwN/src/loadsave.jl:185
  [8] action
    @ ~/.julia/packages/FileIO/DNKwN/src/loadsave.jl:185 [inlined]
  [9] load(::String; options::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ FileIO ~/.julia/packages/FileIO/DNKwN/src/loadsave.jl:113
 [10] load
    @ ~/.julia/packages/FileIO/DNKwN/src/loadsave.jl:110 [inlined]
 [11] #wload#26
    @ ~/.julia/packages/DrWatson/OgRKj/src/DrWatson.jl:38 [inlined]
 [12] wload(data::String)
    @ DrWatson ~/.julia/packages/DrWatson/OgRKj/src/DrWatson.jl:38
 [13] collect_results!(filename::String, folder::String; valid_filetypes::Vector{String}, subfolders::Bool, rpath::Nothing, verbose::Bool, update::Bool, newfile::Bool, kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ DrWatson ~/.julia/packages/DrWatson/OgRKj/src/result_collection.jl:151
 [14] #collect_results#83
    @ ~/.julia/packages/DrWatson/OgRKj/src/result_collection.jl:249 [inlined]
 [15] collect_results(folder::String)
    @ DrWatson ~/.julia/packages/DrWatson/OgRKj/src/result_collection.jl:249
 [16] top-level scope
    @ ~/Desktop/storage-and-LOLE/scripts/storage_operation_comparison.jl:33
Stacktrace:
  [1] handle_error(e::ArgumentError, q::Base.PkgId, bt::Vector{Union{Ptr{Nothing}, Base.InterpreterIP}})
    @ FileIO ~/.julia/packages/FileIO/DNKwN/src/error_handling.jl:61
  [2] handle_exceptions(exceptions::Vector{Tuple{Any, Union{Base.PkgId, Module}, Vector{T} where T}}, action::String)
    @ FileIO ~/.julia/packages/FileIO/DNKwN/src/error_handling.jl:56
  [3] action(::Symbol, ::Vector{Union{Base.PkgId, Module}}, ::FileIO.Formatted; options::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ FileIO ~/.julia/packages/FileIO/DNKwN/src/loadsave.jl:228
  [4] action
    @ ~/.julia/packages/FileIO/DNKwN/src/loadsave.jl:197 [inlined]
  [5] action(::Symbol, ::Vector{Union{Base.PkgId, Module}}, ::Symbol, ::String; options::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ FileIO ~/.julia/packages/FileIO/DNKwN/src/loadsave.jl:185
  [6] action
    @ ~/.julia/packages/FileIO/DNKwN/src/loadsave.jl:185 [inlined]
  [7] load(::String; options::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ FileIO ~/.julia/packages/FileIO/DNKwN/src/loadsave.jl:113
  [8] load
    @ ~/.julia/packages/FileIO/DNKwN/src/loadsave.jl:110 [inlined]
  [9] #wload#26
    @ ~/.julia/packages/DrWatson/OgRKj/src/DrWatson.jl:38 [inlined]
 [10] wload(data::String)
    @ DrWatson ~/.julia/packages/DrWatson/OgRKj/src/DrWatson.jl:38
 [11] collect_results!(filename::String, folder::String; valid_filetypes::Vector{String}, subfolders::Bool, rpath::Nothing, verbose::Bool, update::Bool, newfile::Bool, kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ DrWatson ~/.julia/packages/DrWatson/OgRKj/src/result_collection.jl:151
 [12] #collect_results#83
    @ ~/.julia/packages/DrWatson/OgRKj/src/result_collection.jl:249 [inlined]
 [13] collect_results(folder::String)
    @ DrWatson ~/.julia/packages/DrWatson/OgRKj/src/result_collection.jl:249
 [14] top-level scope
    @ ~/Desktop/storage-and-LOLE/scripts/storage_operation_comparison.jl:33
in expression starting at /home/u0128861/Desktop/storage-and-LOLE/scripts/storage_operation_comparison.jl:32

[timholy]

We'd need a reproducible test case to debug this. Does it work when you just load that specific file? If so, can you share it?

[junglegobs]

Yes, I can: https://www87.zippyshare.com/v/NS6wRS5q/file.html

When I use JLD2.@load I get the following error:

@load "/home/u0128861/Desktop/storage-and-LOLE/results/monte_carlo/default_nP=1/mc_year_1016.jld2"
ERROR: LoadError: ArgumentError: "/home/u0128861/Desktop/storage-and-LOLE/results/monte_carlo/default_nP=1/mc_year_2000.jld2" is not a JLD2 file

If I do FileIO.load I get the previous error:

Error encountered while load FileIO.File{FileIO.DataFormat{:Gadget2}, String}("/home/u0128861/Desktop/storage-and-LOLE/results/monte_carlo/default_nP=1/mc_year_2000.jld2").

Fatal error:
ERROR: ArgumentError: Package AstroIO [c85a633c-0c3f-44a2-bffe-7f9d0681b3e7] is required but does not seem to be installed:
 - Run `Pkg.instantiate()` to install all recorded dependencies.

I presume my file is corrupted but the error from FileIO is miseleading.

[timholy]

@islent, this appears to be an issue for you to tackle (see #264). The problem is that the Gadget2 detection is very "loose"; in

FileIO.jl/src/registry.jl

Line 476 in e04d83d

return temp1 == temp2

both temp1 and temp2 are zero, and this particular file starts with 196612 UInt8(0)s (wow!).

[timholy]

It's definitely not a JLD file:

julia> f = File{format"JLD2"}("mc_year_1016.jld2")
File{DataFormat{:JLD2}, String}("mc_year_1016.jld2")

julia> load(f)
[ Info: Precompiling JLD2 [033835bb-8acc-5ee8-8aae-3f567f8a3819]
Error encountered while load File{DataFormat{:JLD2}, String}("mc_year_1016.jld2").

Fatal error:
ERROR: ArgumentError: "/tmp/fileio/mc_year_1016.jld2" is not a JLD2 file
Stacktrace:
  [1] verify_file_header(f::JLD2.JLDFile{JLD2.MmapIO})
    @ JLD2 ~/.julia/packages/JLD2/15Kr7/src/file_header.jl:26
  [2] jldopen(fname::String, wr::Bool, create::Bool, truncate::Bool, iotype::Type{JLD2.MmapIO}; fallback::Type{IOStream}, compress::Bool, mmaparrays::Bool)
    @ JLD2 ~/.julia/packages/JLD2/15Kr7/src/JLD2.jl:301
[...]

Clearly it's corrupted so there is no hope of reading it that way you intended. But it would be good if it simply said "file not recognized" rather than picking something essentially random.

[junglegobs]

Thanks! I'm curious as to how it got so corrupted, also since I was getting the same error with all the files which were created on our cluster (which sounds very annoying to debug).