Enable variable interpolation in the catalog with OmegaConfigLoader

[merelcht]

Description

In the new OmegaConfigLoader templating/variable interpolation works out of the box for parameters, but not for catalog files. This is because the catalog goes through a validation check which checks if all entries in it are valid datasets. When using the ConfigLoader or TemplatedConfigLoader, users can have entries with "yaml anchors", those need to be preceded by an underscore _ so they're skipped from the catalog validation.

Context

Variable interpolation is a core feature of omegaconf and allows users to make their configuration files "smarter" and more reusable. Even with the addition of parsing syntax to the catalog to minimise the number of catalog entries (#2423), it is desirable to allow users to fully leverage omegaconf variable interpolation in the catalog as well as parameters.

Possible Implementation

Require template values to be preceded by a special character, e.g. _ so they're not read as a dataset in the catalog validation.

You could then have a catalog like this:

_pandas:
  type: pandas.CSVDataSet

example_iris_data:
  type: ${_pandas.type}
  filepath: data/01_raw/iris.csv

This idea was already discussed as part of: #2175
A concern with this implementation is that it introduces special Kedro syntax to enable core omegaconf functionality. Users would then not be able to just use the omegaconf docs to find out how variable interpolation works.

Possible Alternatives

• Another way to skip the catalog validation
• Resolve variables at an earlier time. Currently, resolution happens at access.

[datajoely]

Just confirming that this syntax should work for more complex objects too:

_driver_properties:
  properties:
      driver: org.postgresql.Driver
weather:
  type: spark.SparkJDBCDataSet
  table: weather_table
  url: jdbc:postgresql://localhost/test
  credentials: db_credentials
  load_args: ${_driver_properties}
  save_args: ${_driver_properties}

which should render as:

weather:
  type: spark.SparkJDBCDataSet
  table: weather_table
  url: jdbc:postgresql://localhost/test
  credentials: db_credentials
  load_args:
    properties:
      driver: org.postgresql.Driver
  save_args:
    properties:
      driver: org.postgresql.Driver

[merelcht]

As discussed in #2516 we will be going ahead with the _ character syntax to enable variable interpolation in the catalog.