Saving in produce_or_load does not work for producing anonymous functions

[racinmat]

I want to use produce_or_load in my code, and I am using some anonymous functions for preprocessing data before feeding them to model which I want to persist to file too, as an output from my function along with the trained model.
But then the saving fails, because it is not able to save it.

function train_model(settings)
        # ... some training
	# model is Flux model
	# edge_descriptor is anonymous function
	@dict(model, edge_descriptor)
end

things = produce_or_load(datadir("sims"), settings, train_model, prefix="model", tag=false)
# this does not save thing, thows warning that the saving failed

I noticed serialization is not used here, it would be nice if I could specified e.g. jls suffix and then serialization would be used, so it would work for this usecase.

[Datseris]

Hi @racinmat ,

I am sorry but I don't understand your problem... :(

Please provide the error message.

The title of your message says that produce_or_load doesn't work with anonymous functions. I assume that what you mean is that you cannot save an anonymous function. Another problem could be that if the function f that is an argument to produce_or_load is an anonymous function, then produce_or_load doesn't work. To be clear, which of the two is the problem you have?

DrWatson does not do any saving. All save commands call FileIo. Maybe it already has a backend which saves via serialization?

[sebastianpech]

With suffix="jld" you can at least save a function to the result file. BSON which is the default can't do that. However, this is not reflected in the savename, as Function is not considered to be a valid type for generation.

Something like this:

using DrWatson

function g(conf)
    # Compute something
    Dict(:y=>conf[:f](conf[:x]),:f=>conf[:f])
end

file = produce_or_load(
    "data",
    Dict(:x=>-13,:f=>x->x^2),
    g,
    suffix="jld")

Alternatively you could map your functions to a Symbol maybe:

using DrWatson
const my_functions = Dict(
    :square => x->x^2,
    :times2 => x->2*x)

function g(conf)
    # Compute something
    Dict(:y=>my_functions[conf[:f]](conf[:x]))
end

file = produce_or_load(
    "data",
    Dict(:x=>-13,:f=>:square),
    g,
    suffix="jld")

[racinmat]

@Datseris yes, I want to save an anonymous function.
Yes, it's the backend, the FileIO.

@sebastianpech oh, thanks, I didn't know I can use the jld extension.

Unfortunately the code snipped you provided did not work for me either, causing

Warning: Could not save file, got error ErrorException("Keys must be strings (the names of variables), got y").

but when I used @strdict(model, edge_descriptor) instead of @dict(model, edge_descriptor) the code started working, and with "jld" extension and @strdict macro it works.
Maybe the documentation should state this behaviour, because nowhere in documentation of DrWatson I found any information about suffixes (or the need to use strdict).

[Datseris]

The need to use strdict doesn't come from DrWatson, please check the error stack traces. It comes from JLD which demands that all dictionaries have string type as key.

Information about suffixes is found in savename : https://juliadynamics.github.io/DrWatson.jl/dev/name/#DrWatson.savename

while produce_or_load states that it uses savename (and thus one should look there).

Where else do you think this should be mentioned? Please keep in mind that it is not DrWatson's duty to inform you about every package hooked to FileIO. The responsibility to know what savetypes e.g. the package JLD needs is yours, since it is impossible for us to be informed for all hundreds of packages that FileIO can use... You are welcome to make a Pull Request to the documentation and clarify this in a place you see fit.

[Datseris]

43fc068