Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Config file formats for trials and experiments #29

Closed
wessle opened this issue Nov 9, 2020 · 2 comments
Closed

Config file formats for trials and experiments #29

wessle opened this issue Nov 9, 2020 · 2 comments
Assignees
Labels
enhancement New feature or request

Comments

@wessle
Copy link
Owner

wessle commented Nov 9, 2020

Issue

We need to come up with standard config file formats for specifying both trials and experiments (which are just collections of trials). These are important because we need to have an easy way of saving the (hyper-)parameters we used to generate data alongside the data itself. This will make it easy to identify and replicate experiments, if necessary.

@wessle wessle created this issue from a note in Cost-aware project (To Do) Nov 9, 2020
@wessle
Copy link
Owner Author

wessle commented Nov 9, 2020

Suggestion

These config files could be used by a ConfigManager that can translate in both directions between a YAML (or JSON?) file and the actual ConfigTuple used to create a TrialRunner at runtime. Experiment config files could just be concatenations of trial configs with some additional logic to reduce redundancy (e.g. specifying repeated trials with different seeds, experiment-wide CPU/GPU allocations, etc.). The ConfigManager could also cooperate with individual IOManagers to save config files alongside data generated during experiments.

@wessle wessle added the enhancement New feature or request label Nov 9, 2020
@wessle wessle moved this from To Do to In progress in Cost-aware project Nov 11, 2020
@wessle wessle self-assigned this Nov 11, 2020
@wessle
Copy link
Owner Author

wessle commented Nov 16, 2020

Created YAML formats for specifying trials as well as more general experiments, which are collections of potentially distinct trial types. Created examples scripts demonstrating that they work and showing how to use them.

@wessle wessle closed this as completed Nov 16, 2020
@wessle wessle moved this from In progress to Done in Cost-aware project Nov 16, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Development

No branches or pull requests

1 participant