[Finalizing] NAS/Extending Tutorials and Scheduler Fix

[zhanghang1989]

• Removed script/ folder, due to duplicated information with tutorials.
• Removed not working examples/
• Tutorial for user-defned class (network)
• Tutorial for user-defined training script
• EfficientNet training command, pretrained models
• ImageNet dataset for Image Classification
• ENAS training and tutorial
• Scheduler Enhancement (graceful exit)
• ag.space.Real(lower, upper, default, log=True/False)
• ag.done() for shutting down

fixes #19
fixes #17
fixes #26
fixes #46
fixes #28

[zhanghang1989]

@jwmueller Thanks for reviewing and commenting the code. Sorry that this PR is still work in progress (WIP).

I will ping you guys when it is ready to review.

[jwmueller]

I have a number of concerns about grid_searcher.

Will this work for conditional spaces? Doesn't self._configs contain many invalid options in this case?
I think get_config is overly simplified here (also doesn't seem like it will necessary work correctly in asynchronous settings). Perhaps it should look like this:

def get_config(self):
    if len(self._results) == 0: # no hyperparams have been tried yet, first try default config
        config_cs = self.configspace.get_default_configuration()
        return config_cs.get_dictionary()
    
    # otherwise return next valid configuration in the grid:
    while len(self._configs) > 0:
        config_dict = self._configs.pop()
        config_cs = convert_dict_to_configpaceConfig(config_dict)
        try:
            config_cs.is_valid_configuration() # check that it is valid
            config_dict = config_cs.get_dictionary()
            if (pickle.dumps(config_dict) not in self._results.keys()): # have not yet encountered this config
                 self._results[pickle.dumps(config_dict)] = 0
                 return config_dict
        except ValueError:
            pass 
    
    return None # there are no many valid configurations if you reached this point

[zhanghang1989]

This grid searcher only works for unconditial categorical space. Currently, conditional space is handled in different way in NestedSpace, we use different key names to handle that. Therefore, there is 'no' conditional space in AutoGluon.

[jwmueller]

more clarity would be good here as well. Something like:

"How to perform the neural architecture search procedure to discover EfficientNet from scratch?"

[cgraywang]

The space.* refactor is important for other tasks as discussed in #19

[cgraywang]

Change to categorical?

[zhanghang1989]

Categorical is implemented as we discussed. List is the original Sequence, which returns a python list:

ag.space.List(ag.Real(1e-3, 1e-1), ag.Int(0, 5))

will return a list for example:

[0.01, 3]

[cgraywang]

We might delete this based on the discussion.

[zhanghang1989]

Did we want to remove bool? I thought ag.space.Bool() is simpler.
@jwmueller @mseeger

[jwmueller]

I'm fine leaving ag.space.Bool

[cgraywang]

I am fine as well, the original discussion is to move it though it is more general

[cgraywang]

It generally looks good, thanks. Based on our discussion, could you please do three things 1) change ag.* -ag.space.* in all the notebooks and internal usage; 2) delete controller_type == 'atten' since it is only experimental; 3) move efficientnet/mobilenet to contrib

[zhanghang1989]

It generally looks good, thanks. Based on our discussion, could you please do three things 1) change ag.* -ag.space.* in all the notebooks and internal usage; 2) delete controller_type == 'atten' since it is only experimental; 3) move efficientnet/mobilenet to contrib

1. changed to ag.space
2. 'atten' is working, doesn't break anything and give better result than lstm. If you insist, I can remove it
   3). I have move efficientnet/mobilenet to nas/models because model_zoo should not import contrib stuff. All contrib stuff is not imported by default.

[cgraywang]

for 1) just grep ag.LogLinear, ag.Categorical, ag.Int you would see... for 2) I think it would be better to remove experimental feature, especially when it is under user API (ag.searcher) not under contrib, for 3), I am fine

[zhanghang1989]

for 1) just grep ag.LogLinear, ag.Categorical, ag.Int you would see... for 2) I think it would be better to remove experimental feature, especially when it is under user API (ag.searcher) not under contrib, for 3), I am fine

addressed 1).
2) 'atten' version works the same as 'lstm' from the user side. There is no difference.

[jwmueller]

for 1) just grep ag.LogLinear, ag.Categorical, ag.Int you would see... for 2) I think it would be better to remove experimental feature, especially when it is under user API (ag.searcher) not under contrib, for 3), I am fine

addressed 1).
2) 'atten' version works the same as 'lstm' from the user side. There is no difference.

I vote with Hang that we should keep the controller_type == 'atten' option and not move it to contrib/. It is a functional feature, we just need to be clear in the documentation what exactly it does.

[cgraywang]

Generally it looks good and provides important features. I merge this PR to move faster, as discussed, Iet's try to make a single PR focusing on limited functionalities. The original thought is to make the first release feature limited as well. Thanks