-
Notifications
You must be signed in to change notification settings - Fork 25
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Reproducible configuration files/CLI args like ArgParse.jl and jsonargparse #225
Comments
Thanks for the feature request (I assume this should be FR rather than bug report?) I have to admit, I'm a bit confused after reading this proposal, what's the difference between this and the option types support-based SimpleConfigs? #222 IIUC, the only thing you want is to save your CLI inputs? Why not just do something like I think @avik-pal might have something to say about this, I'm quite confused on this design. Tbh I do tweak hyperparameters too, but I don't really use this workflow. I have a private implementation that fuses a different range of hyperparameters into multiple tasks and distributes them on the cluster in parallel. So I mainly just read in multiple config files that contain the range I want to tweak and also keep those config files as logs.
The option type feature is much more flexible than this?
This does not seem to require a compiler/codegen support you can different do it in the function that Comonicon mycmd --config=config1.toml --config=config2.toml --config=config3.toml is equivalent to d1 = from_toml(MyOptionType, "config1.toml)
merge!(d1, from_toml(MyOptionType, "config2.toml"))
merge!(d1, from_toml(MyOptionType, "config3.toml")) I haven't added a test for something like this, but I think this should work in Comonicon now, otherwise, it should be considered a bug.
See #223
Why not just have a you can definitely have @cast function mycmd(;config::MyOptionType, save::Bool=false)
if save
to_toml(a_file_path_to_save, config)
end
end but I can see, here we will be missing a file path information if the file path is fed as struct ConfigType{T}
schema::T
meta::Dict{String, Any}
end the parser actually saves the information for conversion while parsing https://github.com/Roger-luo/Configurations.jl/blob/master/src/from_toml.jl#L13 it's just not exposed to the users. Also, I'm a bit concerned about mutating a local config file for numerical experiments - shouldn't we consider all the different configurations as immutable and save all the configs so we can trace back previous experiments by time/tags etc.? |
@Roger-luo Thanks for the response. Sorry if it didn't come across the right way, but I had said I think the big issue here seems to me that I didn't understand the role of https://github.com/avik-pal/SimpleConfig.jl from @avik-pal and your configuraitons stuff. The example in https://comonicon.org/dev/conventions/#Working-with-option-types-from-Configurations had all sorts of hidden goodness behind it. Let me make a few responses and then if you feel that there already are sufficient features for this workflow we should close this issue.
For sure, no mutation. Had just stripped down the example too far and was just trying to point out there needs to be a way to save it, and that config should be directly includable at some other time. No encourgement of overwriting of course.
Now that I am starting to understand the tight connection to the config files and types, I think you may have everything you need right now. If we add in a custom @option struct ExperimentConfig
global_seed::Int64 = 0
lr::Float64 = 1e-5
save_config_path::string = ""
end
@cast function mycmd(;config::ExperimentConfig)
if config.save_config_path != ""
to_toml(config.save_config_path config)
end
# code
@show config.global_seed
end Is that right? Then I should be able do things like the following:
It that all works, then I think this workflow is fulfilled completley. Might be worth a doc addition at some point perhaps if this is useful for others. |
Yes, exactly. The only thing that might be missing is pass through the actually file path into the function, which I don't have a good idea for the interface yet.
Yeah, I was hoping to rewrite and polish the docs at some point... some part of the docs are out of date and was written in 0.10.0. Any help on doc improvement is definitely welcome. |
Amazing! Sorry I missed this workflow in the docs. I think this covers everything. I may come back to this during a rewrite in a few months on the CLI stuff for a project now that ArgParse can be exorcized completely.
For sure. I also think that it can have a few manual steps required if you had an example to follow. I am going to close this since the short answer is that you already have this functionality. |
Comonicon has support for an impressive number of features, but is missing a crucial feature for reproducibility: support for config options setup in files (with overwriting on commandline).
The usecases for this is pretty simple: lots of CLIs (especially those related to machine learning) have a lot of different configuration options like hyperparameter settings, various parameters for priors, etc. Often dozens. There is often no natural set of "default" values because sensible defaults may be interrelated, and you may be tweaking those parameters as you go through - making one or two changes in parameters at a time. Furthermore, if you use a bunch of defaults you may have no idea in various versions of your experiments which options you used at that time. Not only that, even if you knew the exact set of defaults for a particular commit, you can't reproduce the results of a CLI call easily because you need to go back and reconstruct the set of defaults and parameters.
The way they did this in python ecosystems (and which we can emulate crudely in Argpase.jl) is the following:
Argparse.jl
this is handled with the special@my_file_path/my_filename.txt
approaches you can see in https://github.com/carlobaldassi/ArgParse.jl/blob/master/test/argparse_test08.jlmy_config_args.txt
withjulia my_script.jl @my_config_args.txt --first 3.0
then it is just like it had called it with--first 3.0 --second 8
as the passed on CLI args overrule the first ones.Hopefully that gives a sense of the workflow. This is not just for minimal CLI terminal apps where I think your current feature set is more than sufficient, but rather for things related to reproducible experiments. Let me give a really simple strawman for how it could work with comonicon (not suggesting this is a good interface, but it seems to capture the minimum software requirements using a variation on the ArgParse.jl approach).
comonicon_argfile_default
reserved kwarg for the@main
function. This establishes a default file to load up prior to applying any CLI args. Maybe you specify a function like@argfile_save
CLI argument. This gives a filename to save the end results of the arguments that would be passed directly to the julia function. The key is that whatever format this exports as, it would be compatible with passing in argiles.@argfile mypath/somefile.txt
or whatever which the user can pass in to pass in pointers to these files.The logic would be as follows:
comonicon_argfile_default
is not empty, then push the results of that into an@argfile
argument on that stack of parameters , overwriting those before it.@argfile
arguments are passed in, then apply them overtop in order. Or just support a single one if that is easier.@argfile_save
then save out the exact CLI arguments as an@argfile
compatible file which would replicate calling that function.With this, something like the following might work
The text was updated successfully, but these errors were encountered: