DRAFT: env + binary blobs

[JonasIsensee]

This is a demonstration implementation of two things inspired by JLSO.jl

It adds definition write_env(f), has_env(f), read_env(f) and Pkg.activate(f::JLDFile) to write, check, read, and activate an environment stored in a file.

It also adds a proof of concept custom serialization Blob. Blob has a single field of type Any that gets serialized using Serialization.serialize.

julia> f = jldopen("test2.jld2", "w")
JLDFile /home/jonas/.julia/dev/JLD2/test2.jld2 (read/write)
  (no datasets)

julia> JLD2.write_blob(f, "h", ()->rand(2,2))

julia> close(f)

## new session

julia> d = load("test2.jld2")
Dict{String, Any} with 1 entry:
  "h" => Blob(#7)

julia> d["h"].obj()
2×2 Matrix{Float64}:
 0.983637  0.0874438
 0.641339  0.81356

alternatively write jldsave("test.jld2"; a=JLD2.Blob(()->rand(3,3)))

a proper API for wrapping and unwrapping in Blobs could of course be built.

EDIT: it might be better to store the enviroment metadata as attributes.

[codecov]

Codecov Report

Base: 88.57% // Head: 83.26% // Decreases project coverage by -5.30% ⚠️

Coverage data is based on head (3abc785) compared to base (35baacc).
Patch coverage: 0.00% of modified lines in pull request are covered.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #377      +/-   ##
==========================================
- Coverage   88.57%   83.26%   -5.31%     
==========================================
  Files          29       32       +3     
  Lines        2896     4136    +1240     
==========================================
+ Hits         2565     3444     +879     
- Misses        331      692     +361     

Impacted Files	Coverage Δ
src/JLD2.jl	89.65% <0.00%> (-2.90%)	⬇️
src/metadata.jl	0.00% <0.00%> (ø)
src/write_blob.jl	0.00% <0.00%> (ø)
src/object_headers.jl	38.37% <0.00%> (-43.11%)	⬇️
src/data/number_types.jl	73.68% <0.00%> (-20.44%)	⬇️
src/backwards_compatibility.jl	37.00% <0.00%> (-18.56%)	⬇️
src/misc.jl	76.82% <0.00%> (-16.51%)	⬇️
src/attributes.jl	89.65% <0.00%> (-10.35%)	⬇️
src/data/specialcased_types.jl	90.20% <0.00%> (-5.83%)	⬇️
src/global_heaps.jl	93.90% <0.00%> (-3.51%)	⬇️
... and 14 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

[JonasIsensee]

Hey @oxinabox,
your ridiculously fast response to me closing issue #57 inspired me to give this some thought again.
I'm still not sure what the best option is or whether there's a good way to integrate JLD2/HDF5 as an "object" format in JLSO.

However, I noticed that I can actually add some of the nice features of JLSO to JLD2 with fairly little effort.
Just pinging for your thoughts / opinion

[oxinabox]

I think this is a nice idea