-
Notifications
You must be signed in to change notification settings - Fork 11
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add batch-evaluate mode #47
Conversation
Pull Request Test Coverage Report for Build 2961163548Warning: This coverage report may be inaccurate.This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.
Details
💛 - Coveralls |
src/batch_eval.rs
Outdated
impl BatchEvaluateOpt { | ||
/// Evaluates the given parameters. | ||
pub fn run(&self) -> Result<()> { | ||
let mut rx: MessageReceiver<BatchEvaluateMessage, _> = MessageReceiver::new(io::stdin()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think that we can use kurobako_core::json::load()
function (https://github.com/optuna/kurobako/blob/master/kurobako_core/src/json.rs#L19) to read parameters from the stdin until it reaches EOS.
IMO, the load function is simpler than MessageReceiver
for this purpose and, by using the load function, we could remove BatchEvaluateMessage
enum as it would be sufficient to define a minimum struct like the following:
#[derive(Deug, Deserialize)]
struct EvalCall {
params: Params,
step: Option<u64>
}
impl BatchEvaluateOpt {
pub fn run(&self) -> Result<()> {
let calls: Vec<EvalCall> = track!(kurobako_core::json::load(io::stdin()))?;
...
}
}
Thanks for pointing that out! Co-authored-by: Takeru Ohta <phjgt308@gmail.com>
FYI: I just created #48 to fix CI (lint) failure. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM.
Let me merge this PR with my one approval since we need this feature. |
Kurobako has
evaluate
subcommand that evaluates a problem at a given set of params. However, when we want to repeatedly evaluate different sets of params and the problem needs to load a large dataset, this is inefficient because the problem reloads the dataset every time. We addbatch-evaluate
subcommand that uses stdio to repeatedly evaluate sets of params without reloading the process.