notebook test; spark warning message; reproducibility bug; sequential tuning stop condition

[sonichi]

Why are these changes needed?

• Add tests for some notebooks.
• Removed warning message about Spark which is unnecessary for non-Spark users.
• Fixed a reproducibility bug of global search.
• In sequential tuning, stop tuning if the returned result from the evaluation function is None. (in future we can allow users to specify max number of None results before stopping)

Related issue number

#851

Checks

• I've used pre-commit to lint the changes in this PR, or I've made sure lint with flake8 output is two 0s.
• I've included any doc changes needed for https://microsoft.github.io/FLAML/. See https://microsoft.github.io/FLAML/docs/Contribute#documentation to build and test documentation locally.
• I've added tests (if relevant) corresponding to the changes introduced in this PR.
• I've made sure all auto checks have passed.

[sonichi]

@thinkall have you run into this error in your notebook test? https://github.com/microsoft/FLAML/actions/runs/3809341315/jobs/6480639426#step:12:178
This passed locally but failed on github. I suppose your test would fail on github too. But your code catches all the exceptions, so it might have failed silently.

[thinkall]

@thinkall have you run into this error in your notebook test? https://github.com/microsoft/FLAML/actions/runs/3809341315/jobs/6480639426#step:12:178 This passed locally but failed on github. I suppose your test would fail on github too. But your code catches all the exceptions, so it might have failed silently.

Ah, let me check.

[thinkall]

Need to install ipykernel in test env to enable default kernel python3.

Please add one line in setup.py.

            "nbformat",
            "ipykernel",

[sonichi]

"ipykernel",

Thanks. Shall I also remove the catch for all exceptions in spark/test_notebook.py?

[sonichi]

@thinkall the spark notebook fails after I removed the except clause for all exceptions. https://github.com/microsoft/FLAML/actions/runs/3819302422/jobs/6496741544#step:12:178

[thinkall]

@thinkall the spark notebook fails after I removed the except clause for all exceptions. https://github.com/microsoft/FLAML/actions/runs/3819302422/jobs/6496741544#step:12:178

>           raise CellExecutionError.from_cell_and_msg(cell, exec_reply_content)
E           nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:
E           ------------------
E           ''' retrieve best config'''
E           print('Best hyperparmeter config:', automl.best_config)
E           print('Best r2 on validation data: {0:.4g}'.format(1-automl.best_loss))
E           print('Training duration of best run: {0:.4g} s'.format(automl.best_config_train_time))
E           ------------------
E           
E           ---------------------------------------------------------------------------
E           KeyError                                  Traceback (most recent call last)
E           Cell In[6], line 4
E                 2 print('Best hyperparmeter config:', automl.best_config)
E                 3 print('Best r2 on validation data: {0:.4g}'.format(1-automl.best_loss))
E           ----> 4 print('Training duration of best run: {0:.4g} s'.format(automl.best_config_train_time))
E           
E           File ~/work/FLAML/FLAML/flaml/automl/automl.py:896, in AutoML.best_config_train_time(self)
E               892 @property
E               893 def best_config_train_time(self):
E               894     """A float of the seconds taken by training the best config."""
E               895     return getattr(
E           --> 896         self._search_states[self._best_estimator], "best_config_train_time", None
E               897     )
E           
E           KeyError: None
E           KeyError: None

It seems that no model returned from the trial. I re-ran the job, it passed. And the same test in other envs all passed. So maybe we can ignore this error or pass a starting point to make sure it can return a model?