New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Robust handling of inconsistent TabularInput keys #45
Comments
IntroductionDowel is a tool that the garage Team uses for logging results from our various Reinforcement learning experiments. Dowel can be used to log different types of data such as floats or strings. The logs can be logged to stdout (the console), CSV files, and Tensorboard. You can check out an example of how Dowel is used here. In fact, almost all parts of the Dowel API are used in this example. The problemAfter statistics such as loss have been logged, and a call to Your goal is to solve the problem as well as introduce tests into our testing framework in order to verify your solution. Some General Instructions
If you have any questions, open an issue in your fork, and tag @avnishn and @zequnyu. Our preferred mode of communication on any questions that you have is through github issues and pull requests, as this is how the Garage team communicates generally. For this reason, we won’t respond to any direct emails with regards to help with your project. We will however respond to any other questions that you have via email (interview scheduling, etc). Best of luck, and let us know if there are any issues as early on as possible |
Before this commit, adding a new key in the tabular type after the first call would lead to a warning. Now, if a new key is added after the initial call CSV_output will rewrite the CSV file to include additional columns corresponding to the new keys. This allows for dynamic keys when using Tabulars. Resolves rlworkgroup#45
This commit adds robust handling of inconsistent TabularInput keys, with two implementations. The current behavior of ignoring new keys is kept as the default, but the users can now optionally specify how to record new keys and the corresponding values. (They must consider the trade-off between the two implementations.)
Currently,
CsvOutput
emits a warning if the keys of aTabularInput
change after the first call tologger.log(TabularInput)
. A new key not seen before will be ignored and an old key not presented will be left blank. In other words,CsvOutput
conservatively handles dynamic fieldnames.This behaviour of
CsvOutput
makes it tricky to log performance of Multi- and Meta- ML algorithms, where there are usually per-task fields but not every task is presented in every iteration, resulting in missing of logs for some tasks.The desired behaviour to handle inconsistent keys should be
The text was updated successfully, but these errors were encountered: