Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

function support on 0.5+ #137

Closed
wants to merge 1 commit into from
Closed

function support on 0.5+ #137

wants to merge 1 commit into from

Conversation

MikeInnes
Copy link
Contributor

@MikeInnes MikeInnes commented Feb 17, 2017 •
edited

fixes #57. Due to the broken type serialisation I've had to find a way to escape non-identifier symbols such that they are parseable.

I kept 0.4 compatibility; the old function storage type is still readable so this should be fully backwards-compatible.

@ChrisRackauckas
Copy link

This is huge! #57 was probably one of the biggest issues of JLD since v0.5.

https://github.com/JuliaIO/JLD.jl/blob/master/src/JLD.jl#L853

It's not immediately apparent to me why v0.4 would end up with more generic functions than v0.5, or at least, why this would be an issue Travis would have on v0.4 but not v0.5.

@ChrisRackauckas
Copy link

https://travis-ci.org/JuliaIO/JLD.jl/jobs/202743914#L1519

sub needs to be view, but that looks unrelated.

@ChrisRackauckas
Copy link

Could this get a review?

@ChrisRackauckas
Copy link

I tried this out and I got an error:

using JLD
JLD.save("out.jld","sol",sol2)

cannot write a pointer to JLD file
 in h5type(::JLD.JldFile, ::Type{Ptr{Void}}, ::Bool) at jld_types.jl:287
 in h5fieldtype(::JLD.JldFile, ::Any, ::Bool) at jld_types.jl:351
 in JLD.JldTypeInfo(::JLD.JldFile, ::SimpleVector, ::Bool) at jld_types.jl:41
 in h5type(::JLD.JldFile, ::Any, ::Bool) at jld_types.jl:367
 in #write_compound#21(::Array{Any,1}, ::Function, ::JLD.JldGroup, ::String, ::SymEngine.Basic, ::JLD.JldWriteSession) at JLD.jl:705
 in _write(::JLD.JldGroup, ::String, ::SymEngine.Basic, ::JLD.JldWriteSession) at JLD.jl:698
 in write_ref(::JLD.JldFile, ::SymEngine.Basic, ::JLD.JldWriteSession) at JLD.jl:667
 in h5convert_array(::JLD.JldFile, ::Array{SymEngine.Basic,1}, ::JLD.JldDatatype, ::JLD.JldWriteSession) at JLD.jl:611
 in #_write#17(::Array{Any,1}, ::Function, ::JLD.JldGroup, ::String, ::Array{SymEngine.Basic,1}, ::JLD.JldWriteSession) at JLD.jl:574
 in _write(::JLD.JldGroup, ::String, ::Array{SymEngine.Basic,1}, ::JLD.JldWriteSession) at JLD.jl:572
 in write_ref(::JLD.JldFile, ::Array{SymEngine.Basic,1}, ::JLD.JldWriteSession) at JLD.jl:667
 in macro expansion at jld_types.jl:669 [inlined]
 in h5convert!(::Ptr{UInt8}, ::JLD.JldFile, ::LotkaVolterra, ::JLD.JldWriteSession) at jld_types.jl:685
 in #write_compound#21(::Array{Any,1}, ::Function, ::JLD.JldGroup, ::String, ::Function, ::JLD.JldWriteSession) at JLD.jl:709
 in _write(::JLD.JldGroup, ::String, ::Function, ::JLD.JldWriteSession) at JLD.jl:698
 in write_ref(::JLD.JldFile, ::Function, ::JLD.JldWriteSession) at JLD.jl:667
 in macro expansion at jld_types.jl:669 [inlined]
 in h5convert!(::Ptr{UInt8}, ::JLD.JldFile, ::DiffEqBase.ODEProblem{Array{Float64,1},Float64,true,LotkaVolterra,Void,UniformScaling{Int64}}, ::JLD.JldWriteSession) at jld_types.jl:685
 in #write_compound#21(::Array{Any,1}, ::Function, ::JLD.JldGroup, ::String, ::DiffEqBase.ODEProblem{Array{Float64,1},Float64,true,LotkaVolterra,Void,UniformScaling{Int64}}, ::JLD.JldWriteSession) at JLD.jl:709
 in _write(::JLD.JldGroup, ::String, ::DiffEqBase.ODEProblem{Array{Float64,1},Float64,true,LotkaVolterra,Void,UniformScaling{Int64}}, ::JLD.JldWriteSession) at JLD.jl:698
 in write_ref(::JLD.JldFile, ::DiffEqBase.ODEProblem{Array{Float64,1},Float64,true,LotkaVolterra,Void,UniformScaling{Int64}}, ::JLD.JldWriteSession) at JLD.jl:667
 in macro expansion at jld_types.jl:669 [inlined]
 in h5convert!(::Ptr{UInt8}, ::JLD.JldFile, ::DiffEqBase.ODESolution{Array{Array{Float64,1},1},Void,Void,Array{Float64,1},Array{Array{Array{Float64,1},1},1},DiffEqBase.ODEProblem{Array{Float64,1},Float64,true,LotkaVolterra,Void,UniformScaling{Int64}},OrdinaryDiffEq.Tsit5,OrdinaryDiffEq.InterpolationData{LotkaVolterra,Array{Array{Float64,1},1},Array{Float64,1},Array{Array{Array{Float64,1},1},1},OrdinaryDiffEq.Tsit5Cache{Array{Float64,1},Array{Float64,1},Array{Float64,1},Array{Float64,1},OrdinaryDiffEq.Tsit5ConstantCache{Float64,Float64}}}}, ::JLD.JldWriteSession) at jld_types.jl:685
 in #write_compound#21(::Array{Any,1}, ::Function, ::JLD.JldFile, ::String, ::DiffEqBase.ODESolution{Array{Array{Float64,1},1},Void,Void,Array{Float64,1},Array{Array{Array{Float64,1},1},1},DiffEqBase.ODEProblem{Array{Float64,1},Float64,true,LotkaVolterra,Void,UniformScaling{Int64}},OrdinaryDiffEq.Tsit5,OrdinaryDiffEq.InterpolationData{LotkaVolterra,Array{Array{Float64,1},1},Array{Float64,1},Array{Array{Array{Float64,1},1},1},OrdinaryDiffEq.Tsit5Cache{Array{Float64,1},Array{Float64,1},Array{Float64,1},Array{Float64,1},OrdinaryDiffEq.Tsit5ConstantCache{Float64,Float64}}}}, ::JLD.JldWriteSession) at JLD.jl:709
 in #write#14(::Array{Any,1}, ::Function, ::JLD.JldFile, ::String, ::DiffEqBase.ODESolution{Array{Array{Float64,1},1},Void,Void,Array{Float64,1},Array{Array{Array{Float64,1},1},1},DiffEqBase.ODEProblem{Array{Float64,1},Float64,true,LotkaVolterra,Void,UniformScaling{Int64}},OrdinaryDiffEq.Tsit5,OrdinaryDiffEq.InterpolationData{LotkaVolterra,Array{Array{Float64,1},1},Array{Float64,1},Array{Array{Array{Float64,1},1},1},OrdinaryDiffEq.Tsit5Cache{Array{Float64,1},Array{Float64,1},Array{Float64,1},Array{Float64,1},OrdinaryDiffEq.Tsit5ConstantCache{Float64,Float64}}}}, ::JLD.JldWriteSession) at JLD.jl:524
 in (::JLD.##35#36{String,DiffEqBase.ODESolution{Array{Array{Float64,1},1},Void,Void,Array{Float64,1},Array{Array{Array{Float64,1},1},1},DiffEqBase.ODEProblem{Array{Float64,1},Float64,true,LotkaVolterra,Void,UniformScaling{Int64}},OrdinaryDiffEq.Tsit5,OrdinaryDiffEq.InterpolationData{LotkaVolterra,Array{Array{Float64,1},1},Array{Float64,1},Array{Array{Array{Float64,1},1},1},OrdinaryDiffEq.Tsit5Cache{Array{Float64,1},Array{Float64,1},Array{Float64,1},Array{Float64,1},OrdinaryDiffEq.Tsit5ConstantCache{Float64,Float64}}}},Tuple{}})(::JLD.JldFile) at JLD.jl:1279
 in #jldopen#11(::Array{Any,1}, ::Function, ::JLD.##35#36{String,DiffEqBase.ODESolution{Array{Array{Float64,1},1},Void,Void,Array{Float64,1},Array{Array{Array{Float64,1},1},1},DiffEqBase.ODEProblem{Array{Float64,1},Float64,true,LotkaVolterra,Void,UniformScaling{Int64}},OrdinaryD...

This was for a DiffEq solution, which is an overloaded type which does not <:Function. That's better than before (before it would say it can't write it because it's a generic function), but I am wondering if there could be a mechanism for making call overloaded types work.

@mbauman
Copy link
Member

mbauman commented May 10, 2017

Ok, so if I understand this correctly, this patch essentially saves an empty object of the given function's type. It saves no behavior. I suppose that's a nice analog to how it behaves for types; no methods get written alongside the data.

My concern is with anonymous functions and accurately communicating to the user what actually is being saved — it's essentially just the name of the function. So this breaks badly with anonymous functions:

julia> using JLD
INFO: Recompiling stale cache file /Users/mbauman/.julia/lib/v0.5/JLD.ji for module JLD.

julia> f1 = x -> x+2
       f2 = x -> x-2
(::#7) (generic function with 1 method)

julia> @save "test.jld" f1 f2
#################################
$ julia -q
julia> using JLD

julia> @load "test.jld"
WARNING: type :"##5#6" not present in workspace; reconstructing
WARNING: type :"##7#8" not present in workspace; reconstructing
2-element Array{Symbol,1}:
 :f1
 :f2

julia> f1()
ERROR: MethodError: objects of type JLD.##:"##5#6"#270 are not callable
#################################
$ julia -q
julia> using JLD

julia> _ = x -> x+20
       _ = x -> x-20
(::#7) (generic function with 1 method)

julia> @load "test.jld"
2-element Array{Symbol,1}:
 :f1
 :f2

julia> f1(1)
21

@MikeInnes
Copy link
Contributor Author

Yes, that's tricky. I think it would be OK, at least as a first pass, to simply say that anonymous functions aren't supported. Unfortunately I don't know of a way to tell whether a function is anonymous, for the sake of throwing an early error.

Serialising functions for parallelism has essentially the same problem; I think in that case it just serialises functions from Main, but that's kind of a nasty solution.

@mbauman
Copy link
Member

mbauman commented May 13, 2017 •
edited

I suppose we could use the name of the function or function type. Seems super hacky, but any names beginning with # or ## are likely going to have trouble resolving across new sessions. Are there any other cases where saving functions by name alone could silently do the wrong thing?

@MikeInnes
Copy link
Contributor Author

It could cause issues with gensym functions, though only in the slightly pathological case where you use a number as the id:

julia> gensym("1")
Symbol("##1#656")

julia> typeof(() -> ())
##1#2

@MikeInnes
Copy link
Contributor Author

MikeInnes commented Feb 23, 2018 •
edited

https://github.com/MikeInnes/BSON.jl

See also here for the anonymous functions issue (I haven't addressed this yet in BSON.jl, but plan to).

@MikeInnes MikeInnes closed this Feb 23, 2018
@mauro3
Copy link

mauro3 commented Feb 23, 2018 •
edited

BSON looks cool! Maybe you could add a line or two to the README about its status and plans.

So, saving functions which are defined works (editing Matt's example from above #137 (comment)) :

/tmp >> julia -q                                                                                                                                                         
julia> using BSON                                                                                                                                                        
                                                                                                                                                                         
julia> bson("test.bson", Dict(:b => sin))                                                                                                                                
                                                                                                                                                                         
julia>                                                                                                                                                                   
/tmp >> julia -q                                                                                                                                                         
julia> using BSON                                                                                                                                                        
                                                                                                                                                                         
julia> BSON.parse("test.bson")                                                                                                                                           
Dict{Symbol,Any} with 1 entry:                                                                                                                                           
  :b => sin                                                                                                                                                              

julia> ans[:b]===sin                                                                                                                                                     
true

but undefined ones don't work:

/tmp >> julia -q                                                                                                                                                         
julia> using BSON                                                                                                                                                        
                                                                                                                                                                         
julia> f(x) = x                                                                                                                                                          
f (generic function with 1 method)                                                                                                                                       

julia> bson("test.bson", Dict(:b => f))                                                                                                                                  
                                                                                                                                                                         
julia>                                                                                                                                                                   
/tmp >> julia -q                                                                                                                                                         
julia> using BSON                                                                                                                                                        
^[[A                                                                                                                                                                     
julia> BSON.parse("test.bson")                                                                                                                                           
ERROR: UndefVarError: #f not defined                                                                                                                                     
Stacktrace:                                                                                                                                                              
 [1] mapfoldl_impl(::Base.#identity, ::BSON.##29#30, ::Module, ::Array{Any,1}, ::Int64) at ./reduce.jl:43                                                                
 [2] (::BSON.##33#34)(::Dict{Symbol,Any}) at /home/mauro/.julia/v0.6/BSON/src/extensions.jl:36                                                                           
 [3] _raise_recursive(::Dict{Symbol,Any}, ::ObjectIdDict) at /home/mauro/.julia/v0.6/BSON/src/read.jl:71                                                                 
 [4] raise_recursive(::Dict{Symbol,Any}, ::ObjectIdDict) at /home/mauro/.julia/v0.6/BSON/src/read.jl:81                                                                  
 [5] (::BSON.##43#45)(::Dict{Symbol,Any}, ::ObjectIdDict) at /home/mauro/.julia/v0.6/BSON/src/extensions.jl:86                                                           
 [6] raise_recursive(::Dict{Symbol,Any}, ::ObjectIdDict) at /home/mauro/.julia/v0.6/BSON/src/read.jl:80                                                                  
 [7] (::BSON.##22#24{ObjectIdDict})(::Dict{Symbol,Any}) at /home/mauro/.julia/v0.6/BSON/src/read.jl:74                                                                   
 [8] applychildren!(::BSON.##22#24{ObjectIdDict}, ::Dict{Symbol,Any}) at /home/mauro/.julia/v0.6/BSON/src/BSON.jl:18                                                     
 [9] _raise_recursive(::Dict{Symbol,Any}, ::ObjectIdDict) at /home/mauro/.julia/v0.6/BSON/src/read.jl:74                                                                 
 [10] raise_recursive(::Dict{Symbol,Any}, ::ObjectIdDict) at /home/mauro/.julia/v0.6/BSON/src/read.jl:81                                                                 
 [11] open(::BSON.#parse, ::String) at ./iostream.jl:152                                                                                                                 
 [12] parse(::String) at /home/mauro/.julia/v0.6/BSON/src/read.jl:94

but you mean to address this, correct?

@MikeInnes
Copy link
Contributor Author

Sort of, although I'm not planning to support named functions, only anonymous ones (i.e. f = x -> x). For named ones you'll have to make sure you set up the environment before hand, as with any other non-Base type.

add a line or two to the README about its status and plans.

Any particular requests? It more or less is what it says on the tin at this stage. I'll let you know if I come up with any BSON-based plans for world domination :)

@mauro3
Copy link

mauro3 commented Feb 23, 2018

Yes, anonymous function support would be awesome!

It more or less is what it says on the tin at this stage.

I meant whether it's ready for use, alpha, beta, etc.

@MikeInnes
Copy link
Contributor Author

Ah I see. It should be ready to use for things it works with; more advanced types (e.g. Julia internal objects) might break it but it works for some fairly complex objects I've used it with (Flux models).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Broken function support in julia-0.5 (all functions are generic)
4 participants
@MikeInnes @ChrisRackauckas @mbauman @mauro3