Join GitHub today
GitHub is home to over 36 million developers working together to host and review code, manage projects, and build software together.Sign up
Timing of Pruning #276
Thank you for your interest in Optuna. I'll try to answer your questions, but I'm not sure that I can fully understand them. If you have further questions, please feel free to contact us.
If you satisfy the requirement, you can set the different intervals of
def objective(trial): iris = sklearn.datasets.load_iris() classes = list(set(iris.target)) train_x, test_x, train_y, test_y = \ sklearn.model_selection.train_test_split(iris.data, iris.target, test_size=0.25) alpha = trial.suggest_loguniform('alpha', 1e-5, 1e-1) clf = sklearn.linear_model.SGDClassifier(alpha=alpha) for step in range(100): clf.partial_fit(train_x, train_y, classes=classes) # Report intermediate objective value. if step % 5 == 0: intermediate_value = 1.0 - clf.score(test_x, test_y) trial.report(intermediate_value, step) # Handle pruning based on the intermediate value. if step % 10 == 0 and trial.should_prune(step): raise optuna.structs.TrialPruned() return 1.0 - clf.score(test_x, test_y)
The above example code based on the
If you use integration modules for xgboost and lightgbm, they invoke
If you use chainer integration, you can change the interval of
I think your question is about parallel execution of trials. Optuna's pruner is called in a trial individually, and the other trials running in other process does not affect the timing of pruning condition checks.
Let me confirm our understanding of your questions. I think your question is about the relationship between Optuna's pruning mechanism and over-fitting of the models trained in your objective functions.
If so, the pruning mechanism does not detect the over-fitting because the pruners do not compare current values with previous values in the same trial. So, I think we need to use early stopping mechanism provided by ML libraries like chainer.training.triggers.EarlyStoppingTrigger for that purpose.
Thank you for the details.
I'm sorry I didn't make the 2nd question clear enough.
I executed parallel trials at the same time.
Is there a way to execute more efficiently?
The difference may come from the implemetation of MedianPruner. Currently, MedianPruner only takes care of completed trials and ignores running trials when it calculate the threshold of pruning (more specifically, the median value of past trials). So, some trials may not be targets of pruning when we use parallel optimization.
Let me explain it using a simple example.
In the case of (a), the pruner is activated after 11 trials. So, 90 trials are the targets of the pruner.
If it is not matched with your case, something may be wrong with the pruning mechanism. If you give us further information such as sample code to reproduce such phenomena and error logs, it will be a great help of us.
I have good news for you. We plan to add a new pruner which significantly accelerates deep-learning tasks in a parallel computing environment.