Recursive Defaults List support

[omry]

Closes #171
Closes #326
Closes #1080
Closes #1089

• Add _self_ support (Ability to specify composition order of primary config file #326)
• Support recursive defaults (with all the existing features, deletion, package rename, overriding)
• Clean new TODOs in the code
• Deprecated - group: null deletion form from defaults list in favor of ~group and ~group=value.
• Ensure no performance regression (will likely require caching of loaded configs because the new system does two passes.
• Include information about actually used defaults list in the generated Hydra config node.
• Consider changing default for unspecified _self_ to be last in the list and not first as it is now
  • Decided against. While this is arguably more intuitive for config files it's less intuitive for dataclasses with defaults list.
• Document recursive defaults (probably warrants a dedicated page just for the defaults list).
• Document _self_
• Deprecated old way of using interpolation in the defaults list. specifically the pattern in specializing configs will be removed in the future in favor of the simpler but more magical new style. Old style is still supported in 1.1 but will be removed in 1.2.

This:

defaults:
  - dataset: imagenet
  - model: alexnet
  - dataset_model: ${defaults.0.dataset}_${defaults.1.model}
    optional: true

Will become:

defaults:
  - dataset: imagenet
  - model: alexnet
  - dataset_model: ${dataset}_${model}`
    optional: true

[lgtm-com]

This pull request introduces 1 alert when merging 01379f3 into 6374fa2 - view on LGTM.com

new alerts:

• 1 for Unused local variable

[lgtm-com]

This pull request introduces 1 alert when merging 5a1f9f2 into 1503d1d - view on LGTM.com

new alerts:

• 1 for Unused local variable

[lgtm-com]

This pull request introduces 1 alert when merging d5b494a into 8adcee0 - view on LGTM.com

new alerts:

• 1 for Unreachable code

[lgtm-com]

This pull request introduces 1 alert when merging 878a458 into 8adcee0 - view on LGTM.com

new alerts:

• 1 for Unreachable code

[lgtm-com]

This pull request introduces 1 alert when merging a708283 into 3775a29 - view on LGTM.com

new alerts:

• 1 for Unreachable code

[lgtm-com]

This pull request introduces 1 alert when merging dd5caaa into 3775a29 - view on LGTM.com

new alerts:

• 1 for Unreachable code

[nmichlo]

Does this mean that the new style will also support nested variables for specialization? Or rather would the following work if I were to extend your last example?

defaults:
  - dataset: imagenet
  - model: alexnet
  - data_wrapper: ${dataset.data_type}_${model.data_input_mode}

This would be an extremely useful pattern, I am currently lacking this feature in an existing project, but just wanted to confirm before submitting a feature request.

[omry]

No. the defaults list will never be able to use config variables in interpolation.
The defaults list is used to compose final config - it obviously cannot access things in the config that is not there yet to determine how to compose the same config.

[omry]

FYI: #1170 re-implements this and will be merged soon.
There are some major changes relative to this implementation there.