Inconsistent Hyperparameter Tutorial Results

[glenn-jocher]

I've run the hyperparameter tuning tutorial in Google Colab:
https://colab.research.google.com/drive/1P6TvA9UZDtLf9dMFTcYWm_RUBW0wpsiV#scrollTo=mzPpAbbyGSBf

I wasn't able to plot my results due to #83, but I am able to see the optimization results printed in the notebook.

best_parameters
{'lr': 0.00018387879800690676, 'momentum': 0.8395379415413641}
means, covariances = values
means, covariances
({'accuracy': 0.9633318647366059},
 {'accuracy': {'accuracy': 1.5112844861703536e-08}})

The 'optimal' momentum above is wildly different from the results shown on the Ax website. I understand there exist 2D local minima in this parameter space, but its a bit surprising to see such enormous differences.

best_parameters
{'lr': 0.0029176399675537317, 'momentum': 3.0347402313065844e-16}
means, covariances = values
means, covariances
({'accuracy': 0.968833362542745},
 {'accuracy': {'accuracy': 1.3653840299223108e-08}})

[lena-kashtelyan]

Now that you are able to plot the outcomes, do the results look like they make more sense, @glenn-jocher?

[glenn-jocher]

@lena-kashtelyan ah, unfortunately the plotting problem #83 persists (it seems to not be numpy related, which was solved in #82). You can reproduce the plotting problem directly by running the tutorial notebook, which I have in a Google Colab:
https://colab.research.google.com/drive/1P6TvA9UZDtLf9dMFTcYWm_RUBW0wpsiV#scrollTo=DMsfBROgGSBK

[lena-kashtelyan]

@glenn-jocher, let's wait until we have the plots of this? I think it's just fine that the results are different, since optimization is stochastic and there is likely more than one optimal hyperparameter setting in your setup.

[glenn-jocher]

@lena-kashtelyan yes, sure I agree. Hyperparameter tuning will always be a bit mystifying I'm afraid, at least for the foreseeable future.