Compose parameters for DEBUG mode

[noklam]

Description

I found it due to the config file, I cannot do dynamic stuff like

if DEBUG:
  epochs=1
else:
  epochs=100

One way of doing so is leverage the environment and create a debug environment. However, it is only composable for additional key, if identical keys exist, the debug enviornment will overwrite completely.

Context

This is very common usage to have a debug mode running fast development

Possible Implementation

Instead of overwrite the key, it should merge them, the provided environment should have higher priority. It also avoids duplicate the parameters, especially for ML project the # of parameters can be large. We should only specify the parameters that is changed.

What I wanted:

base

model:
   layer: 32
   epochs: 100

debug

model:
   epochs: 1

result

model:
  layer: 32 
  epochs: 1

Current behavior

base

model:
   layer: 32
   epochs: 100

debug

model:
   epochs: 1

result

model:
  epochs: 1

Possible Alternatives

A library like hydra provides the ability to compose two configuration file. We just need to merge two dictionary instead of overwriting it.

[noklam]

Seems that currently it only supports top-level key. Is there a reason to restrict it at the top-level only?

https://github.com/quantumblacklabs/kedro/blob/25fd242faafd1740f37e5094cd2e51f2120a0313/kedro/config/config.py#L182

[lorenabalan]

Hi @noklam
I think you can achieve this currently using the TemplatedConfigLoader. Assuming you have a base config

model:
    layer: 32
    epochs: ${model_epochs}

Then you can dedicate globals.yml in each environment:

debug/globals.yml

model_epochs: 1

base/globals.yml

model_epochs: 100

You can find an example in the documentation.

[noklam]

Thx @lorenabalan, this would be a valida workaround.

I am not sure if TemplatedConfig is designed for this purpose, I feel it is designed for string templat. Being able to compose configuration directly is still useful, as now we have 4 configuration files.
i.e.
local/parameters -> local/globals -> debug/parameters -> debug/paramters, it will become harder to lookup for the parameters since it is possible defined in any of these 4 files.

[lorenabalan]

Yeah I agree with you, I had made a note of this for a future team discussion. :)
There might've been an "explicit is better than implicit" motivation back when this was implemented, though.

[lorenabalan]

Closing this as resolved, with configuration environments to be looked at holistically for a future release.